Poster

Implicit Regularization via Neural Feature Alignment

Aristide Baratin · Thomas George · C├ęsar Laurent · R Devon Hjelm · Guillaume Lajoie · Pascal Vincent · Simon Lacoste-Julien

Keywords: [ Deep Learning ] [ Theory ]

Abstract:

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment ofthe neural tangent features introduced by Jacot et al. (2018), along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths. The code for our experiments is available as https://github.com/tfjgeorge/ntk_alignment.

Chat is not available.