SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2410.01735 · NeurIPS · 2024

LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits

Mohit Bansal, Elias Stengel-Eskin, Duy Nguyen, Archiki Prasad

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Reward Models (RMs) are crucial to aligning large language models (LLMs), but the degree to which an RM specialized to one task (e.g. writing) generalizes to new tasks (e.g. math) is often not known a priori, often making using only one fixed RM to train LLMs suboptimal. However, optimizing LLMs with multiple RMs simultaneously can incur a prohibitively high computational cost and lead to conflicting signals from different RMs that may degrade performance. To address these challenges, we introduce LASER (Learning to Adaptively Select Rewards), which frames reward model selection as a multi-armed bandit problem, efficiently and iteratively training LLMs using multiple RMs by selecting the most wellsuited RM for each instance. On commonsense and math reasoning tasks, we show that LASER boosts iterative LLM training, improving the absolute average accuracy of Llama-3-8B over three datasets by 2.67% over an ensemble of RM scores while also showing superior efficiency (e.g., a 2× speedup). Moreover, on WildChat (open-ended instruction-following tasks), LASER leads to a 72.69% AlpacaEval win rate over the RM score ensemble baseline. Extending to longcontext generation, LASER improves by 2.96 F1 points (avg.) on single-document QA tasks and 2.97 F1 points on few-shot learning over the RM score ensemble baseline with best-of-n sampling. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2410.01735")
get_code_for_paper("2410.01735")
have("2410.01735")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2410.01735.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2410.01735.svg)](https://syntology.ai/paper/2410.01735)

Connect an agent — have() is free.