Skip to yearly menu bar Skip to main content


Poster

SDEs for Minimax Optimization

Enea Monzio Compagnoni · Antonio Orvieto · Hans Kersting · Frank Proske · Aurelien Lucchi

MR1 & MR2 - Number 141
[ ] [ Project Page ]
[ Poster
Sat 4 May 6 a.m. PDT — 8:30 a.m. PDT

Abstract:

Minimax optimization problems have attracted a lot of attention over the past few years, with applications ranging from economics to machine learning. While advanced optimization methods exist for such problems, characterizing their dynamics in stochastic scenarios remains notably challenging. In this paper, we pioneer the use of stochastic differential equations (SDEs) to analyze and compare Minimax optimizers. Our SDE models for Stochastic Gradient Descent-Ascent, Stochastic Extragradient, and Stochastic Hamiltonian Gradient Descent are provable approximations of their algorithmic counterparts, clearly showcasing the interplay between hyperparameters, implicit regularization, and implicit curvature-induced noise. This perspective also allows for a unified and simplified analysis strategy based on the principles of Itô calculus. Finally, our approach facilitates the derivation of convergence conditions and closed-form solutions for the dynamics in simplified settings, unveiling further insights into the behavior of different optimizers.

Chat is not available.