Neural Collapse Is Forbidden: Information Floors in Language Models

• URL: http://arxiv.org/abs/2607.09487v1
• Date: Fri, 10 Jul 2026 15:04:02 GMT
• Title: Neural Collapse Is Forbidden: Information Floors in Language Models
• Abstract summary: Within-class variance in language-model representations is commonly read as incomplete neural collapse.<n>We argue it is allocated information storage, and that the allocation obeys a law.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Within-class variance in language-model representations is commonly read as incomplete neural collapse. We argue it is allocated information storage, and that the allocation obeys a law. A one-line centering identity voids a family of simplex equiangular-tight-frame claims, including our own earlier ones; in dimensionless variance shares across 14 models, macro-category structure carries only 4-12% of representational variance and within-token context carries 79-91%, stable across a 100x parameter range. On the theory side, token-level weight decay penalizes a category in proportion to its type count, not its occurrence mass, reducing next-token prediction to an imbalanced K-class problem whose optimum orders category norms by type count. A converse floor, proved for binary categories, forces within-category dispersion to be at least proportional to the conditional mutual information I(token; context | category). The law holds: identity dispersion, not total variance, tracks this information across every tested model and partition, under a model-free estimate and even across models, where one model's information predicts another's dispersion; and over pretraining the category share overshoots, decays, and partially recovers, because the information it must carry never left.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• VAE-Inf: A statistically interpretable generative paradigm for imbalanced classification [8.677199689027772]
  We propose a two-stage framework to bridge the gap between generative modeling and discriminative classification.<n>For inference, we introduce a projection-based score that admits a natural hypothesis testing interpretation.<n>Experiments on diverse real-world benchmarks demonstrate that our framework achieves competitive performance against other approaches.
  arXiv  Detail & Related papers  (2026-04-28T07:50:56Z)
• A weighted-likelihood framework for class imbalance in Bayesian prediction models [0.0]
  Class imbalance is a pervasive problem in predictive toxicology, where the number of non-toxic compounds often exceeds the number of toxic ones.<n>We propose a simple and general Bayesian framework that addresses class imbalance by modifying the likelihood function.
  arXiv  Detail & Related papers  (2025-04-23T18:01:47Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• A statistical theory of overfitting for imbalanced classification [0.6144680854063939]
  We develop a statistical theory for high-dimensional imbalanced classification.<n>We find that dimensionality induces truncation or skewing effects on the logit distribution.<n>This phenomenon explains why the minority class is more severely affected by overfitting.
  arXiv  Detail & Related papers  (2025-02-17T00:21:33Z)
• FocalCount: Towards Class-Count Imbalance in Class-Agnostic Counting [34.25988613929425]
  In class-agnostic object counting, the goal is to estimate the total number of object instances in an image without distinguishing between specific categories.<n>Our approach significantly improves the model's ability to distinguish between specific classes and general counts.
  arXiv  Detail & Related papers  (2025-02-15T05:24:43Z)
• Rethinking generalization of classifiers in separable classes scenarios and over-parameterized regimes [0.0]
  We show that in separable classes scenarios the proportion of "bad" global minima diminishes exponentially with the number of training data n. We propose a model for the density distribution of the true error, yielding learning curves that align with experiments on MNIST and CIFAR-10.
  arXiv  Detail & Related papers  (2024-10-22T10:12:57Z)
• Deep Imbalanced Regression via Hierarchical Classification Adjustment [50.19438850112964]
  Regression tasks in computer vision are often formulated into classification by quantizing the target space into classes. The majority of training samples lie in a head range of target values, while a minority of samples span a usually larger tail range. We propose to construct hierarchical classifiers for solving imbalanced regression tasks. Our novel hierarchical classification adjustment (HCA) for imbalanced regression shows superior results on three diverse tasks.
  arXiv  Detail & Related papers  (2023-10-26T04:54:39Z)
• Non-Asymptotic Performance of Social Machine Learning Under Limited Data [45.48644055449902]
  This paper studies the probability of error associated with the social machine learning framework. It addresses the problem of classifying a stream of unlabeled data in a distributed manner.
  arXiv  Detail & Related papers  (2023-06-15T17:42:14Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• On how to avoid exacerbating spurious correlations when models are overparameterized [33.315813572333745]
  We show that VS-loss learns a model that is fair towards minorities even when spurious features are strong. Compared to previous works, our bounds hold for more general models, they are non-asymptotic, and, they apply even at scenarios of extreme imbalance.
  arXiv  Detail & Related papers  (2022-06-25T21:53:44Z)
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests [87.60900567941428]
  A spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character. We study stress testing using the tools of causal inference.
  arXiv  Detail & Related papers  (2021-05-31T14:39:38Z)
• Learning Invariances in Neural Networks [51.20867785006147]
  We show how to parameterize a distribution over augmentations and optimize the training loss simultaneously with respect to the network parameters and augmentation parameters. We can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations.
  arXiv  Detail & Related papers  (2020-10-22T17:18:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.