WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

Yuxiao Dong, Jie Tang, Wei Xu, Xiao Liu, Yu Yang, Hanyu Lai, Zehan Qi, Iat Iong, Xueqiao Sun, Wenyi Zhao, Xinyue Yang, Jiadai Sun, and 2 more

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 3 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• THUDM/WebRL — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents heavily rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WEBRL, a selfevolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. WEBRL addresses three key challenges in building LLM web agents, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. Specifically, WEBRL incorporates 1) a self-evolving curriculum that generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure consistent improvements. We apply WEBRL to transform open Llama-3.1 and GLM-4 models into proficient web agents. On WebArena-Lite, WEBRL improves the success rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B. These open models significantly surpass the performance of GPT-4-Turbo (17.6%) and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WE-BRL's effectiveness in bridging the gap between open and proprietary LLM-based web agents, paving the way for more accessible and powerful autonomous web interaction systems. The code, model, and data are made publicly available at https://github.com/THUDM/WebRL.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2411.02337")
get_code_for_paper("2411.02337")
have("2411.02337")

Connect an agent — have() is free.