Poster

Complex Momentum for Optimization in Games

Jonathan Lorraine · David Acuna · Paul Vicol · David Duvenaud

[ Abstract ]
Tue 29 Mar 1 a.m. PDT — 2:30 a.m. PDT

Abstract:

We generalize gradient descent with momentum for optimization in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a drop-in replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in realistic adversarial games—like generative adversarial networks—by showing we can find better solutions with an almost identical computational cost. We also show a practical complex-valued Adam variant, which we use to train BigGAN to improve inception scores on CIFAR-10.

Chat is not available.