Learning Transferable Visual Models From Natural Language Supervision

Abstract

We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn state-of-the-art image representations from scratch on a dataset of 400 million image-text pairs. CLIP models learn to connect images and text in a shared embedding space, enabling zero-shot transfer to downstream tasks.

Key Findings

• 1Learned visual representations from natural language supervision at scale
• 2Achieved competitive zero-shot image classification without task-specific training
• 3Created a shared embedding space for images and text enabling cross-modal retrieval
• 4Trained on 400 million image-text pairs from the internet
• 5Demonstrated remarkable robustness to distribution shift

Impact & Significance