Jun Wang, Bo Liu, Rui Zhu, Yaodong Yang, Jie Ren, Xidong Feng, Luo Mai, Haifeng Zhang
We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Benjamin-eecs/Theoretical-GMRL | canonical | 2 of 2 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| magic_box | Ran | this paper's copy was not recorded; identical code first harvested from alexis-jacq/LOLA_DICE pointer only · get_code("2d97875446464dc6") |
| phi | Ran | Benjamin-eecs/Theoretical-GMRL/lola/lola_dice_original.py code served (permissive licence) · get_code("1bfb38a7f9b8eda2") |
| safemean | Ran | Benjamin-eecs/Theoretical-GMRL/mgrl/main_meta_condition_kl.py code served (permissive licence) · get_code("a7b323241cd70612") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of O 𝐾𝛼 𝐾 σIn |𝜏| -0.5 w.r.t. inner-loop update step 𝐾, learning rate 𝛼, estimate variance σ2 In and sample size |𝜏|, and 2) the multi-step Hessian estimation bias Δ𝐻 due to the use of autodiff, which has a polynomial impact O (𝐾 -1) ( Δ𝐻 ) 𝐾 -1 on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general. * Equal contribution, the order is determined by flipping a coin. See Appendix J for more details. † Corresponding author. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2112.15400")
get_code_for_paper("2112.15400")
have("2112.15400")
Connect an agent — have() is free.