We lifted 2 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| shuoli90/CAN | canonical | 0 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| get_PKU | Not yet run | shuoli90/CAN/safe_rlhf/algorithms/cdpo/dpo.py code served (permissive licence) · get_code("1c1b24cc3c61dac4") |
| str2bool | Not yet run | shuoli90/CAN/safe_rlhf/utils.py code served (permissive licence) · get_code("9b9e20fb02209913") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The growing safety concerns surrounding large language models raise an urgent need to align them with diverse human preferences to simultaneously enhance their helpfulness and safety. A promising approach is to enforce safety constraints through Reinforcement Learning from Human Feedback (RLHF). For such constrained RLHF, typical Lagrangian-based primal-dual policy optimization methods are computationally expensive and often unstable. This paper presents a perspective of dualization that reduces constrained alignment to an equivalent unconstrained alignment problem. We do so by pre-optimizing a smooth and convex dual function that has a closed form. This shortcut eliminates the need for cumbersome primal-dual policy iterations, greatly reducing the computational burden and improving training stability. Our strategy leads to two practical algorithms in model-based and preference-based settings (MoCAN and PeCAN, respectively). A broad range of experiments demonstrate the effectiveness and merits of our algorithms.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.19544")
get_code_for_paper("2405.19544")
have("2405.19544")
Connect an agent — have() is free.