Sen Lin, Junshan Zhang, Sheng Yue, Xingyuan Hua, Ju Ren, Yaoxue Zhang
We lifted 13 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| secury/optidice | canonical | 2 of 6 |
| hansenhua/ollie-offline-to-online-imitation-learning | alias | 4 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| FullyConnectedQFunction | Ran | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("60820261a045e8c6") |
| ReparameterizedTanhGaussian | Ran | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("90b115e018d31278") |
| ReplayBuffer | Ran | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("edb756571764039b") |
| TanhGaussianPolicy | Ran | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("e4002bdf5440538b") |
| dice_dataset | Ran | secury/optidice/util.py pointer only (licence: NONE) · get_code("d19706eb97928131") |
| generate_random_mdp | Ran | secury/optidice/mdp.py pointer only (licence: NONE) · get_code("6047e3c9383e4164") |
| OTOF | Not yet run | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("0a4fc95528324a12") |
| Trainer | Not yet run | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("c3ae101599b97fa4") |
| boolean | Not yet run | secury/optidice/default_config.py pointer only (licence: NONE) · get_code("3b81875ee24223a6") |
| init_module_weights | Not yet run | hansenhua/ollie-offline-to-online-imitation-learning/offline_to_online.py pointer only (licence: NONE) · get_code("5c12e4fde6f862c0") |
| move | Not yet run | secury/optidice/example_fourrooms.py pointer only (licence: NONE) · get_code("4a2bd7bd68b877ce") |
| policy_evaluation | Not yet run | secury/optidice/mdp.py pointer only (licence: NONE) · get_code("f9ceec323249d2b9") |
| solve_MDP | Not yet run | secury/optidice/mdp.py pointer only (licence: NONE) · get_code("a85083311a121189") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this paper, we study offline-to-online Imitation Learning (IL) that pretrains an imitation policy from static demonstration data, followed by fast finetuning with minimal environmental interaction. We find the naïve combination of existing offline IL and online IL methods tends to behave poorly in this context, because the initial discriminator (often used in online IL) operates randomly and discordantly against the policy initialization, leading to misguided policy optimization and unlearning of pretraining knowledge. To overcome this challenge, we propose a principled offline-toonline IL method, named OLLIE, that simultaneously learns a near-expert policy initialization along with an aligned discriminator initialization, which can be seamlessly integrated into online IL, achieving smooth and fast finetuning. Empirically, OLLIE consistently and significantly outperforms the baseline methods in 20 challenging tasks, from continuous control to vision-based domains, in terms of performance, demonstration efficiency, and convergence speed. This work may serve as a foundation for further exploration of pretraining and finetuning in the context of IL.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.17477")
get_code_for_paper("2405.17477")
have("2405.17477")
Connect an agent — have() is free.