Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

Tao Liu, Ruida Zhou, Dileep Kalathil, P Kumar, Chao Tian, Min Cheng

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 14 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• tliu1997/rnac — marked official in Papers with Code; named in the paper
• tliu1997/RNAC

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We study robust reinforcement learning (RL) with the goal of determining a wellperforming policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2307.08875")
get_code_for_paper("2307.08875")
have("2307.08875")

Connect an agent — have() is free.