Self-Supervised Learning (SSL) is a family of techniques for converting an unsupervised learning problem into a supervised one by creating surrogate labels from the unlabeled dataset.

Core Paradigms of Self-Supervised Learning

Three primary paradigms dominate the SSL landscape: Joint Embedding, Masked Image Modeling, and a hybrid approach that combines elements of both.

Joint Embedding vs. MIM:

Joint EmbeddingMasked Image Modeling (MIM)
Pros:Pros:
✓ Produces highly semantic features, great for classification.✓ Conceptually simple, with no need for positive/negative pairs.
✓ Architecture agnostic.✓ Masking reduces pre-training time.
✓ Achieves competitive results in linear probing evaluations.✓ Achieves competitive results with fine-tuning.
✓ Stronger fit for low-level tasks (e.g., denoising, super-resolution).
Cons:Cons:
✗ May require very large batch sizes (e.g., SimCLR).✗ Requires a Vision Transformer (ViT) backbone.
✗ Requires careful tuning of data augmentations.✗ Weaker performance on abstract, high-level tasks like classification.
✗ Requires special mechanisms to handle negative samples or avoid collapse.
✗ Not well-suited for low-level tasks.

1. Joint Embedding Architectures

Central idea: Enforce invariance to data augmentations. A Siamese network architecture with shared parameters processes two different augmented “views” of the same image and trains the model to produce similar or identical embeddings for both.

Challenge: model collapse, where the network learns a trivial solution by mapping all inputs to the same constant vector.

Method CategoryCore IdeaKey CharacteristicsExamples
ContrastivePull positive pairs (views of the same image) close in the embedding space while pushing negative pairs (views of different images) apart.Requires negative samples, which can necessitate large batch sizes. Good for multimodal data.SimCLR, MoCo
ClusteringLearn embeddings by grouping similar samples into clusters without using explicit negative pairs.Jointly learns feature representations and cluster assignments.SwAV, Deep Cluster
DistillationA “student” network is trained to match the output distribution of a “teacher” network on different augmented views.Avoids collapse via an asymmetric architecture (student vs. teacher). The teacher is often updated via an Exponential Moving Average (EMA) of the student’s weights. Does not require negative samples.BYOL, DINO
RegularizationAvoids collapse by imposing regularization terms on the embeddings, such as decorrelating feature dimensions.Maximizes the information content of the embeddings by penalizing redundancy. No negative samples required.Barlow Twins, VICReg

2. Masked Image Modeling (MIM)

Central idea: Reconstruction. The input image is split into patches, a significant portion of which (often ~75%) are masked. The model is then trained to predict the content of the masked patches based on the visible ones.

Prediction Targets: The model can be trained to predict various targets for the masked regions:

  • Pixel Reconstruction: Reconstructing the raw pixel values (e.g., MAE, SimMIM).
  • Feature Regression: Predicting abstract feature representations (e.g., MaskFeat).
  • Token Prediction: Predicting discrete visual tokens (e.g., BEiT).

3. Hybrid Architectures

Central idea: Combine the principles of masking and joint embedding

Examples:

  • Image-based Joint Embedding Predictive Architecture (I-JEPA)

Latest Articles

Stochastic Optimization Explained: Simulated Annealing & Pincus Theorem

 Posted on February 2, 2026  |  2792 words  • Other languages: CH

When optimization problems are trapped in the maze of local optima, deterministic algorithms are often helpless. This article takes you into the world of stochastic optimization, exploring how to transform the problem of finding minimum energy into finding maximum probability. We will delve into the physical intuition and mathematical principles of the Simulated Annealing algorithm, demonstrate its elegant mechanism of ‘high-temperature exploration, low-temperature locking’ through dynamic visualization, and derive the Pincus Theorem in detail, mathematically proving why the annealing algorithm can find the global optimal solution. [Read More]
Stochastic Optimization  Simulated Annealing  Optimization Algorithms  Machine Learning  Pincus Theorem  Python Implementation 

Deterministic Optimization Explained: The Mathematical Essence of Gradient Descent

 Posted on February 1, 2026  |  5200 words  • Other languages: CH

Deterministic optimization is the cornerstone for understanding modern MCMC algorithms (like HMC, Langevin). This article delves into three classic deterministic optimization strategies: Newton’s Method (second-order perspective using curvature), Coordinate Descent (the divide-and-conquer predecessor to Gibbs), and Steepest Descent (greedy first-order exploration). Through mathematical derivation and Python visualization, we compare their behavioral patterns and convergence characteristics across different terrains (convex surfaces, narrow valleys, strong coupling). [Read More]
Gradient Descent  Optimization Algorithms  Machine Learning  Deep Learning  Convex Optimization  Python Implementation 

Gibbs Sampling Explained: The Wisdom of Divide and Conquer

 Posted on January 30, 2026  |  2103 words  • Other languages: CH

When high-dimensional spaces are overwhelming, Gibbs sampling adopts a ‘divide and conquer’ strategy. By utilizing full conditional distributions, it breaks down complex N-dimensional joint sampling into N simple 1-dimensional sampling steps. This article explains its intuition, mathematical proof (Brook’s Lemma), and Python implementation. [Read More]
Gibbs Sampling  MCMC  Conditional Distribution  Bayesian Inference  Dimensionality Reduction  Python 

The Metropolis-Hastings Algorithm: Breaking the Symmetry

 Posted on January 29, 2026  |  3439 words  • Other languages: CH

The original Metropolis is limited by symmetric proposals, often ‘hitting walls’ at boundaries or getting lost in high dimensions. The MH algorithm introduces the ‘Hastings Correction’, allowing asymmetric proposals (like Langevin dynamics) while maintaining detailed balance, significantly improving efficiency. [Read More]
MCMC  MH Algorithm  Hastings Correction  Detailed Balance  Python  Bayesian Statistics 

Metropolis Algorithm Explained: Implementation & Intuition

 Posted on January 24, 2026  |  2757 words  • Other languages: CH

The Metropolis algorithm is the cornerstone of MCMC. We delve into its strategy for handling unnormalized densities, from the random walk mechanism to sampling 2D correlated Gaussians, complete with Python implementation and visual diagnostics. [Read More]
MCMC  Metropolis Algorithm  Monte Carlo Simulation  Bayesian Statistics  Python Implementation  Random Walk