Contrastive Self-Supervised Learning As Neural Manifold Packing

• URL: http://arxiv.org/abs/2506.13717v1
• Date: Mon, 16 Jun 2025 17:24:31 GMT
• Title: Contrastive Self-Supervised Learning As Neural Manifold Packing
• Authors: Guanming Zhang, David J. Heeger, Stefano Martiniani,
• Abstract summary: We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem.<n>In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image.<n>Under the standard linear evaluation protocol, CLAMP achieves competitive performance with state-of-the-art self-supervised models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive self-supervised learning based on point-wise comparisons has been widely studied for vision tasks. In the visual cortex of the brain, neuronal responses to distinct stimulus classes are organized into geometric structures known as neural manifolds. Accurate classification of stimuli can be achieved by effectively separating these manifolds, akin to solving a packing problem. We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem. CLAMP introduces a loss function inspired by the potential energy of short-range repulsive particle systems, such as those encountered in the physics of simple liquids and jammed packings. In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image. The sizes and positions of the sub-manifolds are dynamically optimized by following the gradient of a packing loss. This approach yields interpretable dynamics in the embedding space that parallel jamming physics, and introduces geometrically meaningful hyperparameters within the loss function. Under the standard linear evaluation protocol, which freezes the backbone and trains only a linear classifier, CLAMP achieves competitive performance with state-of-the-art self-supervised models. Furthermore, our analysis reveals that neural manifolds corresponding to different categories emerge naturally and are effectively separated in the learned representation space, highlighting the potential of CLAMP to bridge insights from physics, neural science, and machine learning.

Related papers

• KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks [9.512147747894026]
  We show how a gradient flow can be decomposed into a product that involves two operators.<n>We show how their interplay gives rise to low-dimensional latent dynamics under GD.<n>For multi-task training, we show that the operators can be used to measure how objectives relevant to individual sub-tasks align.
  arXiv  Detail & Related papers  (2025-07-08T20:33:15Z)
• Concept-Guided Interpretability via Neural Chunking [54.73787666584143]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract these emerging entities, complementing each other based on label availability and dimensionality.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU [50.9588132578029]
  This paper investigates machine unlearning in hyperbolic contrastive learning.<n>We adapt Alignment to MERU, a model that embeds images and text in hyperbolic space to better capture semantic hierarchies.<n>Our approach introduces hyperbolic-specific components including entailment calibration and norm regularization that leverage the unique properties of hyperbolic space.
  arXiv  Detail & Related papers  (2025-03-19T12:47:37Z)
• CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up [64.38715211969516]
  We introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token.<n>Experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity.
  arXiv  Detail & Related papers  (2024-12-20T17:57:09Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• On Computational Modeling of Sleep-Wake Cycle [5.234742752529437]
  Neuroscience treats sleep and wake as default and perturbation modes of the brain. It is hypothesized that the brain self-organizes neural activities without environmental inputs. This paper presents a new computational model of the sleep-wake cycle for learning and memory.
  arXiv  Detail & Related papers  (2024-04-08T13:06:23Z)
• Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP) [0.0]
  CLOP is a novel semi-supervised loss function designed to prevent neural collapse by promoting the formation of linear subspaces among class embeddings. We show that CLOP enhances performance, providing greater stability across different learning rates and batch sizes.
  arXiv  Detail & Related papers  (2024-03-27T15:48:16Z)
• Pushing Boundaries: Mixup's Influence on Neural Collapse [3.6919724596215615]
  Mixup is a data augmentation strategy that employs convex combinations of training instances and their respective labels to augment the robustness and calibration of deep neural networks. This study investigates the last-layer activations of training data for deep networks subjected to mixup. We show that mixup's last-layer activations predominantly converge to a distinctive configuration different than one might expect.
  arXiv  Detail & Related papers  (2024-02-09T04:01:25Z)
• A Waddington landscape for prototype learning in generalized Hopfield networks [0.0]
  We study the learning dynamics of Generalized Hopfield networks. We observe a strong resemblance to the canalized, or low-dimensional, dynamics of cells as they differentiate.
  arXiv  Detail & Related papers  (2023-12-04T21:28:14Z)
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning [57.98014222570084]
  We show that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimize the representation space without relying on a trainable projector.
  arXiv  Detail & Related papers  (2021-10-18T14:22:19Z)
• GEM: Group Enhanced Model for Learning Dynamical Control Systems [78.56159072162103]
  We build effective dynamical models that are amenable to sample-based learning. We show that learning the dynamics on a Lie algebra vector space is more effective than learning a direct state transition model. This work sheds light on a connection between learning of dynamics and Lie group properties, which opens doors for new research directions.
  arXiv  Detail & Related papers  (2021-04-07T01:08:18Z)
• The Role of Isomorphism Classes in Multi-Relational Datasets [6.419762264544509]
  We show that isomorphism leakage overestimates performance in multi-relational inference. We propose isomorphism-aware synthetic benchmarks for model evaluation. We also demonstrate that isomorphism classes can be utilised through a simple prioritisation scheme.
  arXiv  Detail & Related papers  (2020-09-30T12:15:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.