SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2205.07802 · ICML · 2022

The Primacy Bias in Deep Reinforcement Learning

Pierluca D'oro, Aaron Courville, Pierre-Luc Bacon, Max Schwarzer, Evgenii Nikishin, -Isaac Asimov

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance. "Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won't come in.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2205.07802")
get_code_for_paper("2205.07802")
have("2205.07802")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2205.07802.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2205.07802.svg)](https://syntology.ai/paper/2205.07802)

Connect an agent — have() is free.