Skip to yearly menu bar Skip to main content


Poster

Robust Offline Reinforcement Learning with Heavy-Tailed Rewards

Jin Zhu · Runzhe Wan · Zhengling Qi · Shikai Luo · Chengchun Shi

MR1 & MR2 - Number 16
[ ] [ Project Page ]
Thu 2 May 8 a.m. PDT — 8:30 a.m. PDT

Abstract:

This paper endeavors to augment the robustness of offline reinforcement learning (RL) in scenarios laden with heavy-tailed rewards, a prevalent circumstance in real-world applications. We propose two algorithmic frameworks, ROAM and ROOM, for robust off-policy evaluation and offline policy optimization (OPO), respectively. Central to our frameworks is the strategic incorporation of the median-of-means method with offline RL, enabling straightforward uncertainty estimation for the value function estimator. This not only adheres to the principle of pessimism in OPO but also adeptly manages heavy-tailed rewards. Theoretical results and extensive experiments demonstrate that our two frameworks outperform existing methods on the logged dataset exhibits heavy-tailed reward distributions. The implementation of the proposal is available at \url{https://github.com/Mamba413/ROOM}.

Live content is unavailable. Log in and register to view live content