SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1707.01495 · 2017

Hindsight Experience Replay

Pieter Abbeel, Marcin Andrychowicz, Jonas Schneider, Alex Ray, Bob Mcgrew, Peter Welinder, Wojciech Zaremba, Filip Wolski, Josh Tobin, Rachel Fong

arXiv · PDF · Open in the Atlas

Code that ran

16 of 16 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (10 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 10 have no contract verdict; 5 failures of our instrument; 0 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/1707.01495.json · badge: /api/ran/1707.01495.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
YangRui2015/Modular_HER community
Papers with Code archive
3 ran · 0 no recorded run
bacdavid/HER community
Papers with Code archive
2 ran · 0 no recorded run
JunkyByte/HER_DQN community
Papers with Code archive
1 ran · 0 no recorded run
sumitsk/HER community
Papers with Code archive
1 ran · 0 no recorded run
opendilab/DI-engine community
Papers with Code archive
1 ran · 0 no recorded run
hemilpanchiwala/Hindsight-Experience-Replay community
Papers with Code archive
1 ran · 0 no recorded run
hietalajulius/dynamic-cloth-folding community
Papers with Code archive
1 ran · 0 no recorded run
tenvinc/CS4246-Project community
Papers with Code archive
1 ran · 0 no recorded run
sdicastro/simandrealbettertogether community
Papers with Code archive
1 ran · 0 no recorded run
Medabid1/RL_Project community
Papers with Code archive
1 ran · 0 no recorded run
copy not recorded community
not in the paper's declared repositories
1 ran · 0 no recorded run
Baichenjia/BHER community
Papers with Code archive
1 ran · 0 no recorded run
flowersteam/curious community
Papers with Code archive
1 ran · 0 no recorded run
FunctionStatusWhere it lives
HER Ran
ran on a synthesized input · constructed an object; no result was computed
bacdavid/HER/hindsight_experience.py
pointer only (licence: NONE) · get_code("a2d77ddd5e9cb248")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, bacdavid/HER when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

HER Ran
ran on a synthesized input · constructed an object; no result was computed
JunkyByte/HER_DQN/src/her_module/her.py
pointer only (licence: NONE) · get_code("9d20e8525bae5d5c")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, JunkyByte/HER_DQN when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

HER_Sampler Ran
ran on a synthesized input · constructed an object; no result was computed
YangRui2015/Modular_HER/mher/samplers/her_sampler.py
code served (permissive licence) · get_code("e959db3969fc9191")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, HER_Sampler: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def reward_fun(ag_2, g, info):
    """Stand-in reward function that returns rewards based on goal and achieved goal."""
    # Simple reward: negative distance between achieved goal and goal
    distance = np.linalg.norm(ag_2 - g, axis=-1)
    return -distance

# Create episode_batch with required keys
T = 10
batch_size = 4
state_dim = 5
action_dim = 3
goal_dim = 4

# Build episode_batch with all required keys
episode_batch = {
    'u': np.random.randn(8, T, action_dim).astype(np.float32),  # actions
    'ag': np.random.randn(8, T, goal_dim).astype(np.float32),   # achieved goals
    'ag_2': np.random.randn(8, T, goal_dim).astype(np.float32), # next achieved goals
    'g': np.random.randn(8, T, goal_dim).astype(np.float32),    # goals
    'r': np.random.randn(8, T).astype(np.float32),              # rewards
    'o': np.random.randn(8, T, state_dim).astype(np.float32),   # observations
    'info_test': np.random.randn(8, T, 2).astype(np.float32),   # info field for testing
}

ARGS = {
    'T': T,
    'reward_fun': reward_fun,
    'batch_size': batch_size,
    'relabel_p': 0.8,
    'strategy': 'future',
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type HER_Sampler — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a HER_Sampler output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
HER_sampler Ran
ran on a synthesized input · constructed an object; no result was computed
sumitsk/HER/her.py
code served (permissive licence) · get_code("f0fd2810588c0760")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, HER_sampler: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def simple_reward_fun(ag_2, g, info):
    """
    Simple reward function that computes rewards based on achieved goal and desired goal.
    ag_2 shape: (batch_size, goal_dim)
    g shape: (batch_size, goal_dim)
    """
    # Simple L2 distance-based reward
    distance = np.linalg.norm(ag_2 - g, axis=1)
    reward = -distance
    return reward

# Set random seed for reproducibility
np.random.seed(0)

# Define arguments for HER_sampler.__init__
ARGS = {
    'replay_k': 4,
    'reward_fun': simple_reward_fun,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type HER_sampler — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a HER_sampler output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
HerRewardModel Ran
ran on a synthesized input · constructed an object; no result was computed
opendilab/DI-engine/ding/reward_model/her_reward_model.py
code served (permissive licence) · get_code("06e4f2c26ad752ba")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, HerRewardModel: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import List, Dict, Any, Optional, Callable, Tuple

torch.manual_seed(0)
np.random.seed(0)

# Create a minimal config class with required attributes
class Config:
    def __init__(self):
        self.her_strategy = 'final'
        self.her_replay_k = 2
        self.episode_size = 4
        self.sample_per_episode = 2
    
    def get(self, key, default=None):
        return getattr(self, key, default)

# Create sample episode data
# Each transition has 'obs', 'next_obs', 'reward', and other fields
# obs and next_obs are split into (obs, desired_goal, achieved_goal) by split_func
def create_sample_episode():
    episode = []
    for i in range(5):
        # obs: concatenation of [observation(4,), desired_goal(3,), achieved_goal(3,)]
        obs = torch.randn(10)  # 4 + 3 + 3
        next_obs = torch.randn(10)
        
        transition = {
            'obs': obs,
            'next_obs': next_obs,
            'reward': torch.tensor(0.0),
            'action': torch.tensor([0.1, 0.2]),
            'done': False,
        }
        episode.append(transition)
    return episode

# Custom split function that extracts obs, desired_goal, achieved_goal
def custom_split_func(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """Split concatenated tensor into obs, desired_goal, achieved_goal"""
    obs = x[:4]
    desired_goal = x[4:7]
    achieved_goal = x[7:10]
    return obs, desired_goal, achieved_goal

# Custom merge function
def custom_merge_func(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
    """Merge obs and goal"""
    return torch.cat([x, y], dim=0)

# Custom goal reward function
def custom_goal_reward_func(achieved_goal: torch.Tensor, desired_goal: torch.Tensor) -> torch.Tensor:
    """Compute reward based on goal achievement"""
    distance = torch.norm(achieved_goal - desired_goal)
    reward = torch.tensor(-distance.item())
    return reward

# Build arguments
cfg = Config()
episode = create_sample_episode()

ARGS = {
    'cfg': cfg,
    'cuda': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type HerRewardModel — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a HerRewardModel output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
HindsightExperienceReplayMemory Ran
ran on a synthesized input · constructed an object; no result was computed
hemilpanchiwala/Hindsight-Experience-Replay/dqn_with_her/HERMemory.py
pointer only (licence: NONE) · get_code("cee4c46cd21980ff")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hemilpanchiwala/Hindsight-Experience-Replay when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

RelabelSampler Ran
ran on a synthesized input · constructed an object; no result was computed
YangRui2015/Modular_HER/mher/samplers/her_sampler.py
code served (permissive licence) · get_code("5b95feb61a547c1b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, RelabelSampler: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def dummy_reward_fun(ag_2, g, info):
    """
    Dummy reward function that takes ag_2, g, and info.
    Returns a reward array matching batch size.
    """
    batch_size = ag_2.shape[0]
    return np.ones(batch_size, dtype=np.float32)

# Create small episode batch with required keys
batch_size = 2
T = 5
state_dim = 4
action_dim = 3
goal_dim = 4

episode_batch = {
    'u': np.random.randn(batch_size, T, action_dim).astype(np.float32),  # actions
    'ag_2': np.random.randn(batch_size, T, goal_dim).astype(np.float32),  # achieved goals
    'g': np.random.randn(batch_size, T, goal_dim).astype(np.float32),  # goals
    'r': np.random.randn(batch_size, T).astype(np.float32),  # rewards
    'info_extra': np.random.randn(batch_size, T, 2).astype(np.float32),  # extra info
}

ARGS = {
    'T': T,
    'reward_fun': dummy_reward_fun,
    'batch_size': batch_size,
    'relabel_p': 0.5,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type RelabelSampler — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a RelabelSampler output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Sampler Ran
ran on a synthesized input · constructed an object; no result was computed
YangRui2015/Modular_HER/mher/samplers/her_sampler.py
code served (permissive licence) · get_code("8c4664b53d0576fa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, Sampler: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def dummy_reward_fun(ag_2, g, info):
    """Simple reward function that returns a scalar reward per transition."""
    # ag_2 shape: (batch_size, goal_dim)
    # g shape: (batch_size, goal_dim)
    # Returns rewards of shape (batch_size,)
    return np.linalg.norm(ag_2 - g, axis=1)

# Construct episode_batch with required keys
batch_size = 2
T = 5
state_dim = 4
goal_dim = 3
action_dim = 2

episode_batch = {
    'u': np.random.randn(batch_size, T, action_dim),  # actions
    'ag_2': np.random.randn(batch_size, T, goal_dim),  # achieved goals
    'g': np.random.randn(batch_size, T, goal_dim),  # goals
    'info_extra': np.random.randn(batch_size, T, 2),  # example info field
}

ARGS = {
    'T': T,
    'reward_fun': dummy_reward_fun,
    'batch_size': batch_size,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Sampler — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a Sampler output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
concat Ran
ran, honoured the contract we drafted
bacdavid/HER/hindsight_experience.py
pointer only (licence: NONE) · get_code("f616d3cbede3329d")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, bacdavid/HER when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_replay_buffer_kwargs Ran
ran; our contract draft was wrong, not the code
hietalajulius/dynamic-cloth-folding/utils/general_utils.py
code served (permissive licence) · get_code("40295510a8fb1d25")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

class Args:
    def __init__(self):
        self.buffer_size = 10000
        self.her_percent = 0.2

ARGS = {
    'args': Args()
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
her_sample Ran
ran; our contract draft was wrong, not the code
tenvinc/CS4246-Project/agent/core.py
pointer only (licence: NONE) · get_code("18eb3a56918d44a6")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, tenvinc/CS4246-Project when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

her_sampler Ran
ran on a synthesized input · constructed an object; no result was computed
sdicastro/simandrealbettertogether/her_modules/her.py
pointer only (licence: NONE) · get_code("6001bd605e7c8a38")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, sdicastro/simandrealbettertogether when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

her_sampler Ran
ran on a synthesized input · constructed an object; no result was computed
Medabid1/RL_Project/utils/her.py
pointer only (licence: NONE) · get_code("732ee2071c58239b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, Medabid1/RL_Project when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

make_sample_her_transitions Ran
ran; our contract draft was wrong, not the code
this paper's copy was not recorded; identical code first harvested from jangirrishabh/Overcoming-exploration-from-demos
pointer only · get_code("e0785843af667cdd")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

make_sample_her_transitions Ran
ran; our contract draft was wrong, not the code
Baichenjia/BHER/BHER/bher/her.py
pointer only (licence: NONE) · get_code("14bbcdb9ee8b58d7")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, Baichenjia/BHER when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

make_sample_her_transitions Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
flowersteam/curious/baselines/her/her.py
code served (permissive licence) · get_code("dc602be819c160dc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def reward_function(ag_2, g, task_descr=None, info=None):
    """Mock reward function that computes rewards based on achieved and goal states."""
    # ag_2 and g should be arrays of shape (batch_size, goal_dim)
    # Return rewards of shape (batch_size,)
    batch_size = ag_2.shape[0]
    # Simple L2 distance-based reward
    distances = np.linalg.norm(ag_2 - g, axis=1)
    rewards = -distances
    return rewards

# Create mock episode batch data
T = 5  # trajectory length
rollout_batch_size = 3  # number of episodes in buffer
goal_dim = 4
action_dim = 3
state_dim = 4

episode_batch = {
    'u': np.random.randn(rollout_batch_size, T, action_dim),  # actions
    'ag': np.random.randn(rollout_batch_size, T, goal_dim),   # achieved goals
    'g': np.random.randn(rollout_batch_size, T, goal_dim),    # goals
    'ag_2': np.random.randn(rollout_batch_size, T, goal_dim), # next achieved goals
    'info_extra': np.random.randn(rollout_batch_size, T, 2),  # extra info
}

# Mock task configuration
tasks_ag_id = [
    [0, 1],      # task 0 uses goal dimensions 0, 1
    [2, 3],      # task 1 uses goal dimensions 2, 3
]
tasks_g_id = [
    [0, 1],      # task 0 goal indices
    [2, 3],      # task 1 goal indices
]

ARGS = {
    'goal_replay': 'her',
    'her_replay_k': 4,
    'reward_fun': reward_function,
    'task_replay': '',
    'tasks_ag_id': tasks_ag_id,
    'tasks_g_id': tasks_g_id,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 16 samples lifted from this paper's linked repositories, 16 carry a function identity and 0 do not. Of the 16 distinct functions with an identity, 2 are also present in other papers whose code we hold (1 with at least one paper dated earlier, 0 the same month, 1 later) and 14 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
HindsightExperienceReplayMemory
brick of this paper's assembly: entry
1 other paper
1 dated earlier · 1 only as an assembly brick
1509.02971 (2015-09)
Continuous control with deep reinforcement learning
make_sample_her_transitions
brick of this paper's assembly: entry
1 other paper
1 later
1709.10089 (2017-09)
Overcoming Exploration in Reinforcement Learning with Demonstrations

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task. The video presenting our experiments is available at https://goo.gl/SMrQnI.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1707.01495")
get_code_for_paper("1707.01495")
have("1707.01495")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1707.01495.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1707.01495.svg)](https://syntology.ai/paper/1707.01495)

Connect an agent — have() is free.