SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2001.11113 · ICML · 2020

GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values

Shangtong Zhang, Bo Liu, Shimon Whiteson

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We present GradientDICE for estimating the density ratio between the state distribution of the target policy and the sampling distribution in offpolicy reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang et al., 2020a), the state-of-the-art for estimating such density ratios. Namely, the optimization problem in GenDICE is not a convex-concave saddlepoint problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so any primal-dual algorithm is not guaranteed to converge or find the desired solution. However, such nonlinearity is essential to ensure the consistency of GenDICE even with a tabular representation. This is a fundamental contradiction, resulting from GenDICE's original formulation of the optimization problem. In Gra-dientDICE, we optimize a different objective from GenDICE by using the Perron-Frobenius theorem and eliminating GenDICE's use of divergence, such that nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2001.11113")
get_code_for_paper("2001.11113")
have("2001.11113")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2001.11113.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2001.11113.svg)](https://syntology.ai/paper/2001.11113)

Connect an agent — have() is free.