Skip to yearly menu bar Skip to main content


A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning

Kihyuk Hong · Yuhang Li · Ambuj Tewari

MR1 & MR2 - Number 12
[ ]
Thu 2 May 8 a.m. PDT — 8:30 a.m. PDT


Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose Primal-Dual-Critic Algorithm (PDCA), a novel algorithm for offline constrained RL with general function approximation. PDCA runs a primal-dual algorithm on the Lagrangian function estimated by critics. The primal player employs a no-regret policy optimization oracle to maximize the Lagrangian estimate and the dual player acts greedily to minimize the Lagrangian estimate. We show that PDCA finds a near saddle point of the Lagrangian, which is nearly optimal for the constrained RL problem. Unlike previous work that requires concentrability and a strong Bellman completeness assumption, PDCA only requires concentrability and realizability assumptions for sample-efficient learning.

Live content is unavailable. Log in and register to view live content