Learning Synthetic Environments and Reward Networks for Reinforcement Learning

Code that ran

We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repository	Role	Ran
automl/learning_environments	canonical	2 of 2

Function	Status	Where it lives
build_nn_from_config	Ran	automl/learning_environments/models/model_utils.py code served (permissive licence) · get_code("8d761786c13867be")
score_transform	Ran	automl/learning_environments/experiments/demo_score_transform.py code served (permissive licence) · get_code("10c5ef0d687bb482")

Repositories linked to this paper

automl/learning_environments — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We introduce Synthetic Environments (SEs) and Reward Networks (RNs), represented by neural networks, as proxy environment models for training Reinforcement Learning (RL) agents. We show that an agent, after being trained exclusively on the SE, is able to solve the corresponding real environment. While an SE acts as a full proxy to a real environment by learning about its state dynamics and rewards, an RN is a partial proxy that learns to augment or replace rewards. We use bi-level optimization to evolve SEs and RNs: the inner loop trains the RL agent, and the outer loop trains the parameters of the SE / RN via an evolution strategy. We evaluate our proposed new concept on a broad range of RL algorithms and classic control environments. In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment. However, once such an SE has been learned, we do not need any interactions with the real environment to train new agents. Moreover, the learned SE proxies allow us to train agents with fewer interactions while maintaining the original task performance. Our empirical results suggest that SEs achieve this result by learning informed representations that bias the agents towards relevant states. Moreover, we find that these proxies are robust against hyperparameter variation and can also transfer to unseen agents.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2202.02790")
get_code_for_paper("2202.02790")
have("2202.02790")

Connect an agent — have() is free.