Pengcheng Xu, Tong Zheng, Zhan Cheng, Shuo Zhou, Youling Huang, Skylar Zhai, Xuwei Ding, Shaotong Qi, Jingcheng Liang, TianMing Sha
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Multi-reward reinforcement learning trains large language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, we show that this energy is proportional to active-group density: the fraction of rollout groups in which the reward provides nonzero relative advantages. This reveals a residual batch-level signal imbalance and provides a basis for calibrating reward contributions. Based on this relation, we propose Density-Aware Reward Aggregation (DARA). We derive an inverse-square-root density correction that gives greater weight to signals from less frequently active rewards. DARA computes its weights from each rollout batch, adapting to changes in reward activity throughout training without modifying the underlying policy optimization objective. Experiments on tool calling and mathematical reasoning show that DARA learns the targeted behaviors faster than GDPO, reaching high format compliance in up to 26% fewer training steps on tool calling and near-saturated length compliance in up to 65% fewer steps on mathematical reasoning, while remaining competitive in final performance. Our code is available at https://github.com/zhaihaotian/DARA.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2610.00574")
get_code_for_paper("2610.00574")
have("2610.00574")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2610.00574.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2610.00574)
Connect an agent — have() is free.