We lifted 4 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| tinkoff-ai/cnf | canonical | 2 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| make_optimizers | Ran | tinkoff-ai/cnf/cnf/train_rl.py code served (permissive licence) · get_code("8ac3758971b9694b") |
| valid_batch | Ran | tinkoff-ai/cnf/cnf/train_flow.py code served (permissive licence) · get_code("632f8f7c106515fc") |
| optimize_flow | Not yet run | tinkoff-ai/cnf/cnf/train_flow.py code served (permissive licence) · get_code("a56fcb19dade7c8e") |
| train_epoch | Not yet run | tinkoff-ai/cnf/cnf/train_flow.py code served (permissive licence) · get_code("1310d5462c8d6dba") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. There are two major challenges in this setting: (1) extrapolation error caused by approximating the value of state-action pairs not well-covered by the training data and (2) distributional shift between behavior and inference policies. One way to tackle these problems is to induce conservatism - i.e., keeping the learned policies closer to the behavioral ones. To achieve this, we build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model, which we use as a conservative action encoder. This Normalizing Flows action encoder is pre-trained in a supervised manner on the offline dataset, and then an additional policy model - controller in the latent space - is trained via reinforcement learning. This approach avoids querying actions outside of the training dataset and therefore does not require additional regularization for out-of-dataset actions. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms with generative action models on a large portion of datasets.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2211.11096")
get_code_for_paper("2211.11096")
have("2211.11096")
Connect an agent — have() is free.