Supervised Contrastive Learning

By Prannay Khosla, Piotr Teterwak, Chen Wang et al (Google Research), 2020

The authors use contrastive loss, which has recently been shown to be very effective at learning deep neural network representations in the self-supervised setting, for supervised learning, and achieve better results than those obtained with cross entropy loss for ResNet-50 and ResNet-200.

What can we learn from this paper?

That using the standard cross-entropy loss may not be optimal when training deep neural networks for image classification tasks.

Prerequisites (to understand the paper, what does one need to be familiar with?)

  • Neural networks
  • Loss functions


The main idea of the contrastive loss is that members of the same class in a classification task are expected to produce similar representations at the output of the neural network (before the final softmax layer), while members of different classes will result in fairly different representations. Thus, members of each class form a separate cluster in the representation space.

The contrastive loss is calculated by maximizing the dot products of normalized representations from images of the same class while minimizing the dot products of representations of different classes. The exact formula for the loss, adjusted for the supervised learning case from its unsupervised formulation and allowing for multiple instances of the same class, can be found in the paper.

At the first stage of training, the chosen network architecture (ResNet-50 and ResNet-200 were used in the paper) was trained on ImageNet using the contrastive loss. For its calculation, an extra fully-connected projection network with one hidden layer was added at the end of each ResNet to reduce the representation size from 2048 to 128. Two separate augmentations of each image were used as inputs.

After training with the contrastive loss, the projection network was discarded and a new randomly initialized fully connected layer was trained (with all other weights fixed) using the standard cross entropy loss.

The resulting trained network showed a significant improvement in top-1 accuracy on ImageNet (about 2% higher than the authors’ implementation of cross entropy loss training and better than other SOTA results with these architectures), while at the same time being noticeably more robust to image corruption as shown by Mean Corruption Error (mCE) results, and less sensitive to changes in training hyperparameters (optimizers, data augmentation techniques, etc.)

Many details of the suggested implementation are discussed in detail, and a Github repository is available. This paper has been generating a lot of interest among researchers since it was published a few weeks ago, and it definitely looks like a very promising new approach.

Original paper link

Supervised Contrastive Learning by P. Khosla, P. Teterwak, C. Wang et al, 2020

Github repository

Suggested reading

A Simple Framework for Contrastive Learning of Visual Representations by T. Chen, S. Kornblith, M. Norouzi, G. Hinton, 2020

Contrastive Representation Distillation by Y. Tian, D. Krishnan, P. Isola, 2019

Improving Generalization via Scalable Neighborhood Component Analysis by Z. Wu, A.A. Efros, S.X. Yu, 2018

Leave a Reply