Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Yejin Choi, Hannaneh Hajishirzi, Noah Smith, Yizhong Wang, Jiacheng Liu, Valentina Pyatkin, Zeqiu Wu, Hamish Ivison, Nathan Lambert

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 19 functions out of this paper's own repositories and ran 15 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• allenai/open-instruct — marked official in Papers with Code; named in the paper
• hamishivi/easylm — marked official in Papers with Code; named in the paper
• hamishivi/EasyLM

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training 1 and evaluating 2 our models, along with the models and datasets themselves 3 . 1 https://github.com/hamishivi/EasyLM 2 https://github.com/allenai/open-instruct 3 https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618 4 Sometimes this stage is called reinforcement learning from human feedback (RLHF). However, the human and reinforcement learning aspects are not always present, while the learning from preferences aspect is always present. We will use the terms 'learning from preferences' and 'preference-based learning' interchangeably. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.09279")
get_code_for_paper("2406.09279")
have("2406.09279")

Connect an agent — have() is free.