Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent

Suriya Gunasekar · Blake Woodworth · Nathan Srebro

Keywords: [ Learning Theory and Statistics ] [ Gradient-Based Optimization ]

Abstract: We present a direct (primal only) derivation of Mirror Descent as a partial'' discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a full'' forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in $\mathbb{R}^d$, even when the metric tensor is {\em not} a Hessian, and thus there is no dual.''