Sen Lin, Honghao Wei, Zhigang Deng, Keru Chen
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
High costs and risks involved in extensive environmental interactions hinder the practical application of current online safe reinforcement learning (RL) methods. Inspired by recent successes in offline-to-online (O2O) RL, it is crucial to explore whether offline safe RL can be leveraged to facilitate faster and safer online learning, a direction that has yet to be fully investigated. To fill this gap, we first show that naively applying existing O2O algorithms from standard RL would not work well in safe RL due to two unique challenges: erroneous Q-estimations, resulted from offline-online objective mismatch and offline cost sparsity, and Lagrangian mismatch, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce Marvel, the first policy-finetuning based framework for O2O safe RL, comprising two key components that work in concert: Value Pre-Alignment to align the learned Q-functions with the online objective before finetuning, and Adaptive PID Control to effectively adjust the Lagrange multipliers during finetuning. Extensive experiments demonstrate the superior performance of Marvel over related baselines.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2412.04426")
get_code_for_paper("2412.04426")
have("2412.04426")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2412.04426.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2412.04426)
Connect an agent — have() is free.