What Is Self-Supervised Learning?
Self-supervised learning is a machine learning approach where models generate their own training labels from the structure of unlabeled data, enabling them to learn rich representations from massive datasets without human annotation.
How Self-Supervised Learning Works
Self-supervised learning bridges the gap between supervised and unsupervised learning. The model creates a supervisory signal from the data itself — for example, masking a word in a sentence and predicting it (as in BERT), or predicting the next token in a sequence (as in GPT). This allows models to learn from enormous amounts of unlabeled data, which is far more abundant than labeled data. Self-supervised learning is the foundation of modern pre-training for both language models and vision systems. It enables models to develop a deep understanding of language structure, visual features, and other patterns before being fine-tuned for specific tasks.
Real-World Examples
GPT learning language by predicting the next word in billions of sentences from the internet
BERT being pre-trained by masking 15% of words in sentences and learning to predict the missing words
A vision model learning visual features by predicting how rotated or cropped versions of images relate to the originals