Ziyi Yang, Ming Yan, Xiaojun Quan, Fanqi Wan, Weizhou Shen, Longguang Zhong, Tianyuan Shi, Canbin Huang
We lifted 7 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Stycoo/Mutual-Taught | canonical | 1 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| maybe_torch | Ran | Stycoo/Mutual-Taught/reward_model_merge/rm_merge_custom.py code served (permissive licence) · get_code("10ff293597eab810") |
| build_dpo_dataset | Not yet run | Stycoo/Mutual-Taught/completion_inference/build_dataset.py code served (permissive licence) · get_code("f7d5662f1399084a") |
| create_node | Not yet run | Stycoo/Mutual-Taught/completion_inference/completion_infer.py code served (permissive licence) · get_code("11ae53d561cabab0") |
| lerp | Not yet run | Stycoo/Mutual-Taught/reward_model_merge/rm_merge_custom.py code served (permissive licence) · get_code("ed054b3a2f7dde35") |
| read_json_file | Not yet run | Stycoo/Mutual-Taught/completion_inference/build_dataset.py code served (permissive licence) · get_code("1acc67cef16f4df5") |
| slerp | Not yet run | Stycoo/Mutual-Taught/reward_model_merge/rm_merge_custom.py code served (permissive licence) · get_code("343787d4eb1da770") |
| split_by_newlines | Not yet run | Stycoo/Mutual-Taught/completion_inference/completion_infer.py code served (permissive licence) · get_code("615a32d374e639ea") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model (RM). This shift reduces the efficacy of the RM, which in turn negatively impacts the performance of the policy model (PM). To address this challenge, we propose Mutual-Taught, a self-training method that iteratively improves both the PM and RM without requiring additional human annotation. Our approach mirrors the expectation-maximization (EM) algorithm. In the E-step, the PM is updated using feedback from the current RM, guiding the PM toward a better approximation of the latent optimal preference distribution. In the M-step, we update the RM by constructing training data from the outputs of the PM before and after the E-step update. This process ensures that the RM adapts to the evolving policy distribution. Experimental results demonstrate that this iterative approach leads to consistent improvements in both models. Specifically, our 8B policy model, Llama-3-8B-Instruct-MT, achieves a length-controlled win rate of 54.1% on AlpacaEval-2, while our 8B reward model, FsfairX-Llama3-RM-MT, performs on par with GPT-4o-2024-08-06 on Re-wardBench. Our code is available at https: //github.com/Stycoo/Mutual-Taught.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2506.06292")
get_code_for_paper("2506.06292")
have("2506.06292")
Connect an agent — have() is free.