Poster

Contrasting the landscape of contrastive and non-contrastive learning

Ashwini Pokle · Jinjin Tian · Yuchen Li · Andrej Risteski

[ Abstract ]
Wed 30 Mar 8:30 a.m. PDT — 10 a.m. PDT

Abstract:

A lot of recent advances in unsupervised feature learning are based on designing features which are invariant under semantic data augmentations. A common way to do this is contrastive learning, which uses positive and negative samples. Some recent works however have shown promising results for non-contrastive learning, which does not require negative samples. However, the non-contrastive losses have obvious ``collapsed'' minima, in which the encoders output a constant feature embedding, independent of the input. A folk conjecture is that so long as these collapsed solutions are avoided, the produced feature representations should be good. In our paper, we cast doubt on this story: we show through theoretical results and controlled experiments that even on simple data models, non-contrastive losses have a preponderance of non-collapsed bad minima. Moreover, we show that the training process does not avoid these minima. Code for this work can be found at https://github.com/ashwinipokle/contrastive_landscape.

Chat is not available.