Contrastive Self-Supervised Learning As Neural Manifold Packing

• URL: http://arxiv.org/abs/2506.13717v1
• Date: Mon, 16 Jun 2025 17:24:31 GMT
• Title: Contrastive Self-Supervised Learning As Neural Manifold Packing
• Abstract summary: We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem.<n>In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image.<n>Under the standard linear evaluation protocol, CLAMP achieves competitive performance with state-of-the-art self-supervised models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive self-supervised learning based on point-wise comparisons has been widely studied for vision tasks. In the visual cortex of the brain, neuronal responses to distinct stimulus classes are organized into geometric structures known as neural manifolds. Accurate classification of stimuli can be achieved by effectively separating these manifolds, akin to solving a packing problem. We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem. CLAMP introduces a loss function inspired by the potential energy of short-range repulsive particle systems, such as those encountered in the physics of simple liquids and jammed packings. In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image. The sizes and positions of the sub-manifolds are dynamically optimized by following the gradient of a packing loss. This approach yields interpretable dynamics in the embedding space that parallel jamming physics, and introduces geometrically meaningful hyperparameters within the loss function. Under the standard linear evaluation protocol, which freezes the backbone and trains only a linear classifier, CLAMP achieves competitive performance with state-of-the-art self-supervised models. Furthermore, our analysis reveals that neural manifolds corresponding to different categories emerge naturally and are effectively separated in the learned representation space, highlighting the potential of CLAMP to bridge insights from physics, neural science, and machine learning.

Related papers

• Toward a mechanistic understanding of inference in visual cortex and diffusion models [46.86106767256413]
  We describe a model of perceptual inference in primary visual cortex (V1)<n>We efficiently train these recurrent dynamics using a denoising score-matching objective and implicit differentiation.<n>This model exhibits exceptionally good denoising performance, restoring image features such as extended contours amid extreme visual ambiguity.
  arXiv  Detail & Related papers  (2026-07-17T07:11:57Z)
• Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm [4.838965998214037]
  Group-Contrastive Forward-Forward (GCFF) is a forward-forward training algorithm that combines class-specific routing with within-class contrastive objectives.<n>On CLIP representations, a single trained GCFF module recovers monosemantic neurons whose abstraction increases progressively with depth.
  arXiv  Detail & Related papers  (2026-07-13T09:54:35Z)
• Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy [0.8665758002017515]
  This study investigates the internal neural representations of cognitive complexity using Bloom's taxonomy as a hierarchical lens.<n>We probe whether different cognitive levels, ranging from basic recall (Remember) to abstract synthesis (Create) are linearly separable within the model's residual streams.
  arXiv  Detail & Related papers  (2026-02-19T10:19:04Z)
• Quantum LEGO Learning: A Modular Design Principle for Hybrid Artificial Intelligence [63.39968536637762]
  We introduce Quantum LEGO Learning, a learning framework that treats classical and quantum components as reusable, composable learning blocks.<n>Within this framework, a pre-trained classical neural network serves as a frozen feature block, while a VQC acts as a trainable adaptive module.<n>We develop a block-wise generalization theory that decomposes learning error into approximation and estimation components.
  arXiv  Detail & Related papers  (2026-01-29T14:29:21Z)
• Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning [17.299267108673277]
  Hebbian learning is a biological principle that intuitively describes how neurons adapt their connections through repeated stimuli.<n>We introduce the Structural Projection Hebbian Representation (SPHeRe), a novel unsupervised learning method.<n> Experimental results show that SPHeRe achieves SOTA performance among unsupervised synaptic plasticity approaches.
  arXiv  Detail & Related papers  (2025-10-16T15:47:29Z)
• What Drives Compositional Generalization in Visual Generative Models? [56.01574461407906]
  We conduct a systematic study of how various design choices influence compositional generalization in image and video generation.<n>We identify two key factors: (i) whether the training objective operates on a discrete or continuous distribution, and (ii) to what extent conditioning provides information about the constituent concepts during training.<n>Building on these insights, we show that relaxing the MaskGIT discrete loss with an auxiliary continuous JEPA-based objective can improve compositional performance in discrete models like MaskGIT.
  arXiv  Detail & Related papers  (2025-10-03T15:02:27Z)
• Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning [18.581721927380684]
  We propose a novel fine-tuning method, Manifold-Preserving and Sculpting Tuning (MPS-Tuning)<n>MPS-Tuning explicitly constrains the intrinsic geometry of this manifold while further sculpting it to enhance class separability.<n>Experiments demonstrate that MPS-Tuning significantly improves model performance while effectively preserving the structure of the semantic manifold.
  arXiv  Detail & Related papers  (2025-08-18T12:28:43Z)
• KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks [9.512147747894026]
  We show how a gradient flow can be decomposed into a product that involves two operators.<n>We show how their interplay gives rise to low-dimensional latent dynamics under GD.<n>For multi-task training, we show that the operators can be used to measure how objectives relevant to individual sub-tasks align.
  arXiv  Detail & Related papers  (2025-07-08T20:33:15Z)
• Concept-Guided Interpretability via Neural Chunking [54.73787666584143]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract these emerging entities, complementing each other based on label availability and dimensionality.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU [50.9588132578029]
  This paper investigates machine unlearning in hyperbolic contrastive learning.<n>We adapt Alignment to MERU, a model that embeds images and text in hyperbolic space to better capture semantic hierarchies.<n>Our approach introduces hyperbolic-specific components including entailment calibration and norm regularization that leverage the unique properties of hyperbolic space.
  arXiv  Detail & Related papers  (2025-03-19T12:47:37Z)
• CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up [64.38715211969516]
  We introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token.<n>Experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity.
  arXiv  Detail & Related papers  (2024-12-20T17:57:09Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• On Computational Modeling of Sleep-Wake Cycle [5.234742752529437]
  Neuroscience treats sleep and wake as default and perturbation modes of the brain. It is hypothesized that the brain self-organizes neural activities without environmental inputs. This paper presents a new computational model of the sleep-wake cycle for learning and memory.
  arXiv  Detail & Related papers  (2024-04-08T13:06:23Z)
• Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP) [0.0]
  CLOP is a novel semi-supervised loss function designed to prevent neural collapse by promoting the formation of linear subspaces among class embeddings. We show that CLOP enhances performance, providing greater stability across different learning rates and batch sizes.
  arXiv  Detail & Related papers  (2024-03-27T15:48:16Z)
• Pushing Boundaries: Mixup's Influence on Neural Collapse [3.6919724596215615]
  Mixup is a data augmentation strategy that employs convex combinations of training instances and their respective labels to augment the robustness and calibration of deep neural networks. This study investigates the last-layer activations of training data for deep networks subjected to mixup. We show that mixup's last-layer activations predominantly converge to a distinctive configuration different than one might expect.
  arXiv  Detail & Related papers  (2024-02-09T04:01:25Z)
• A Waddington landscape for prototype learning in generalized Hopfield networks [0.0]
  We study the learning dynamics of Generalized Hopfield networks. We observe a strong resemblance to the canalized, or low-dimensional, dynamics of cells as they differentiate.
  arXiv  Detail & Related papers  (2023-12-04T21:28:14Z)
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning [57.98014222570084]
  We show that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimize the representation space without relying on a trainable projector.
  arXiv  Detail & Related papers  (2021-10-18T14:22:19Z)
• GEM: Group Enhanced Model for Learning Dynamical Control Systems [78.56159072162103]
  We build effective dynamical models that are amenable to sample-based learning. We show that learning the dynamics on a Lie algebra vector space is more effective than learning a direct state transition model. This work sheds light on a connection between learning of dynamics and Lie group properties, which opens doors for new research directions.
  arXiv  Detail & Related papers  (2021-04-07T01:08:18Z)
• The Role of Isomorphism Classes in Multi-Relational Datasets [6.419762264544509]
  We show that isomorphism leakage overestimates performance in multi-relational inference. We propose isomorphism-aware synthetic benchmarks for model evaluation. We also demonstrate that isomorphism classes can be utilised through a simple prioritisation scheme.
  arXiv  Detail & Related papers  (2020-09-30T12:15:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.