SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2602.11523 · 2026

Unifying Stable Optimization and Reference Regularization in RLHF

Lina Yao, He Zhao, Tongliang Liu, Li He, Stephen Wan, Qiang Qu, Dadong Wang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: reward hacking and stable optimization. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against a supervised fine-tuned model (π 0 ) to mitigate reward hacking, and policy ratio clipping towards the current policy (π t ) to promote stable alignment. However, the implicit trade-off arising from simultaneously regularizing towards both π 0 and π t remains under-explored. In this paper, we introduce a unified regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RLHF and online preference learning methods, achieving enhanced alignment performance and stability. Our implementation is available at github.com/tmllab/2026 ICLR DAR.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.11523")
get_code_for_paper("2602.11523")
have("2602.11523")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2602.11523.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2602.11523.svg)](https://syntology.ai/paper/2602.11523)

Connect an agent — have() is free.