Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning

Ran He, Jian Liang, Qianlong Xie, Xingxing Wang, Meng Wang, Kuangpu Guo, Lingxiao He, Yongcan Yu

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 10 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• yuyongcan/DDRL

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Test-time reinforcement learning (TTRL) always adapts models at inference time via pseudo-labeling, leaving it vulnerable to spurious optimization signals from label noise. Through an empirical study, we observe that responses with medium consistency form an ambiguity region and constitute the primary source of reward noise. Crucially, we find that such spurious signals can be even amplified through group-relative advantage estimation. Motivated by these findings, we propose a unified framework, Debiased and Denoised test-time Reinforcement Learning (DDRL), to mitigate spurious signals. Concretely, DDRL first applies a frequency-based sampling strategy to exclude ambiguous samples while maintaining a balanced set of positive and negative examples. It then adopts a debiased advantage estimation with fixed advantages, removing the bias introduced by grouprelative policy optimization. Finally, DDRL incorporates a consensus-based off-policy refinement stage, which leverages the rejectionsampled dataset to enable efficient and stable model updates. Experiments on three large language models across multiple mathematical reasoning benchmarks demonstrate that DDRL consistently outperforms existing TTRL baselines. The code will soon be released at https://github.com/yuyongcan/DDRL.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2604.21327")
get_code_for_paper("2604.21327")
have("2604.21327")

Connect an agent — have() is free.