SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2501.15034 · NeurIPS · 2025

Divergence-Augmented Policy Optimization

Tong Zhang, Qing Wang, Yingru Li, Jiechao Xiong

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In deep reinforcement learning, policy optimization methods need to deal with issues such as function approximation and the reuse of off-policy data. Standard policy gradient methods do not handle off-policy data well, leading to premature convergence and instability. This paper introduces a method to stabilize policy optimization when off-policy data are reused. The idea is to include a Bregman divergence between the behavior policy that generates the data and the current policy to ensure small and safe policy updates with off-policy data. The Bregman divergence is calculated between the state distributions of two policies, instead of only on the action probabilities, leading to a divergence augmentation formulation. Empirical experiments on Atari games show that in the data-scarce scenario where the reuse of off-policy data becomes necessary, our method can achieve better performance than other state-of-the-art deep reinforcement learning algorithms. * The work was done when Qing Wang was at Tencent AI Lab. † Equal contribution.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2501.15034")
get_code_for_paper("2501.15034")
have("2501.15034")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2501.15034.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2501.15034.svg)](https://syntology.ai/paper/2501.15034)

Connect an agent — have() is free.