Arthur Guez, Doina Precup, Nicolas Heess, Kaist, Daniel Mankowitz, Cosmin Paduraru, Kee-Eung Kim, Jongmin Lee
We lifted 7 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| deepmind/constrained_optidice | canonical | 4 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| domain_cost_fn | Ran | deepmind/constrained_optidice/neural/cost.py code served (permissive licence) · get_code("021d008f035670a2") |
| get_f_divergence_fn | Ran | deepmind/constrained_optidice/neural/learning.py code served (permissive licence) · get_code("b0a316d25b4cad24") |
| policy_evaluation | Ran | deepmind/constrained_optidice/tabular/mdp_util.py code served (permissive licence) · get_code("77a7c8ddd9479170") |
| policy_evaluation_mdp | Ran | deepmind/constrained_optidice/tabular/mdp_util.py code served (permissive licence) · get_code("3bb97e9b8f8bef6d") |
| conditional_update | Not yet run | deepmind/constrained_optidice/neural/learning.py code served (permissive licence) · get_code("68bce47fe9bdd627") |
| generate_random_cmdp | Not yet run | deepmind/constrained_optidice/tabular/mdp_util.py code served (permissive licence) · get_code("06a80085f44b4edd") |
| periodic_update | Not yet run | deepmind/constrained_optidice/neural/learning.py code served (permissive licence) · get_code("ba29d1b163a7d24d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. This problem setting is appealing in many real-world scenarios, where direct interaction with the environment is costly or risky, and where the resulting policy should comply with safety constraints. However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the offline RL setting, since the offpolicy evaluation inherently has an estimation error. In this paper, we present an offline constrained RL algorithm that optimizes the policy in the space of the stationary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constraint satisfaction. Experimental results show that COptiDICE attains better policies in terms of constraint satisfaction and return-maximization, outperforming baseline algorithms.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2204.08957")
get_code_for_paper("2204.08957")
have("2204.08957")
Connect an agent — have() is free.