A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics

• URL: http://arxiv.org/abs/2604.09979v1
• Date: Sat, 11 Apr 2026 01:27:15 GMT
• Title: A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics
• Abstract summary: Self-supervised representation learning is central to modern machine learning because it extracts structured latent features from unlabeled data.<n>It can suffer from representation collapse, a widely observed failure mode in which embeddings lose discriminative structure and distinct inputs become indistinguishable.<n>We introduce a minimal embedding-only model whose gradient-flow dynamics and fixed points can be analyzed in closed form.
• Score: 12.453886119376902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised representation learning is central to modern machine learning because it extracts structured latent features from unlabeled data and enables robust transfer across tasks and domains. However, it can suffer from representation collapse, a widely observed failure mode in which embeddings lose discriminative structure and distinct inputs become indistinguishable. To understand the mechanisms that drive collapse and the ingredients that prevent it, we introduce a minimal embedding-only model whose gradient-flow dynamics and fixed points can be analyzed in closed form, using a classification-representation setting as a concrete playground where collapse is directly quantified through the contraction of label-embedding geometry. We illustrate that the model does not collapse when the data are perfectly classifiable, while a small fraction of frustrated samples that cannot be classified consistently induces collapse through an additional slow time scale that follows the early performance gain. Within the same framework, we examine collapse prevention by adding a shared projection head and applying stop-gradient at the level of the training dynamics. We analyze the resulting fixed points and develop a dynamical mean-field style self-consistency description, showing that stop-gradient enables non-collapsed solutions and stabilizes finite class separation under frustration. We further verify empirically that the same qualitative dynamics and collapse-prevention effects appear in a linear teacher-student model, indicating that the minimal theory captures features that persist beyond the pure embedding setting.

Related papers

• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• On the Limits of Self-Improving in LLMs and Why AGI, ASI and the Singularity Are Not Near Without Symbolic Model Synthesis [0.01269104766024433]
  We formalise self-training in Large Language Models (LLMs) and Generative AI as a discrete-time dynamical system.<n>We derive two fundamental failure modes: (1) Entropy Decay, where finite sampling effects cause a monotonic loss of distributional diversity (mode collapse), and (2) Variance Amplification, where the loss of external grounding causes the model's representation of truth to drift as a random walk.
  arXiv  Detail & Related papers  (2026-01-05T19:50:49Z)
• Activation Function Design Sustains Plasticity in Continual Learning [1.618563064839635]
  In continual learning, models can progressively lose the ability to adapt.<n>We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss.
  arXiv  Detail & Related papers  (2025-09-26T16:41:47Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• On the Embedding Collapse when Scaling up Recommendation Models [53.66285358088788]
  We identify the embedding collapse phenomenon as the inhibition of scalability, wherein the embedding matrix tends to occupy a low-dimensional subspace. We propose a simple yet effective multi-embedding design incorporating embedding-set-specific interaction modules to learn embedding sets with large diversity.
  arXiv  Detail & Related papers  (2023-10-06T17:50:38Z)
• Perturbation Analysis of Neural Collapse [24.94449183555951]
  Training deep neural networks for classification often includes minimizing the training loss beyond the zero training error point. Recent works analyze this behavior via idealized unconstrained features models where all the minimizers exhibit exact collapse. We propose a richer model that can capture this phenomenon by forcing the features to stay in the vicinity of a predefined features matrix.
  arXiv  Detail & Related papers  (2022-10-29T17:46:03Z)
• Are Gradients on Graph Structure Reliable in Gray-box Attacks? [56.346504691615934]
  Previous gray-box attackers employ gradients from the surrogate model to locate the vulnerable edges to perturb the graph structure. In this paper, we discuss and analyze the errors caused by the unreliability of the structural gradients. We propose a novel attack model with methods to reduce the errors inside the structural gradients.
  arXiv  Detail & Related papers  (2022-08-07T06:43:32Z)
• Memorization-Dilation: Modeling Neural Collapse Under Label Noise [10.134749691813344]
  During the terminal phase of training a deep neural network, the feature embedding of all examples of the same class tend to collapse to a single representation. Empirical evidence suggests that the memorization of noisy data points leads to a degradation (dilation) of the neural collapse. Our proofs reveal why label smoothing, a modification of cross-entropy empirically observed to produce a regularization effect, leads to improved generalization in classification tasks.
  arXiv  Detail & Related papers  (2022-06-11T13:40:37Z)
• ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning [75.2748374360642]
  We propose ModeRNN, which introduces a novel method to learn hidden structured representations between recurrent states. Across the entire dataset, different modes result in different responses on the mixtures of slots, which enhances the ability of ModeRNN to build structured representations.
  arXiv  Detail & Related papers  (2021-10-08T03:47:54Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.