Poster

Stable ResNet

Soufiane Hayou · Eugenio Clerico · Bobby He · George Deligiannidis · Arnaud Doucet · Judith Rousseau

Keywords: [ Deep Learning ] [ Theory ]

[ Abstract ]
Thu 15 Apr 7:30 a.m. PDT — 9:30 a.m. PDT
 
Oral presentation: Theory of Statistical and Deep Learning Methods
Tue 13 Apr 10:30 a.m. PDT — 11:30 a.m. PDT

Abstract:

Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as the depth becomes large (Yang et al. 2017). Moreover, recent results have shown that ResNet might lose expressivity as the depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these issues, we introduce a new class of ResNet architectures, calledStable ResNet, that have the property of stabilizing the gradient while ensuring expressivity in the infinite depth limit.

Chat is not available.