Escaping Mode Collapse in LLM Generation via Geometric Regulation

• URL: http://arxiv.org/abs/2605.00435v1
• Date: Fri, 01 May 2026 06:12:05 GMT
• Title: Escaping Mode Collapse in LLM Generation via Geometric Regulation
• Abstract summary: We take a dynamical-systems view and reinterpret mode collapse as reduced state-space accessibility.<n>We proposeReinforced Mode Regulation (RMR), a lightweight, online state-space intervention that regulates dominant self-reinforcing directions.<n>RMR substantially reduces mode collapse and enables stable, high-quality generation at extremely low entropy rates.
• Score: 11.183390901786659
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mode collapse is a persistent challenge in generative modeling and appears in autoregressive text generation as behaviors ranging from explicit looping to gradual loss of diversity and premature trajectory convergence. We take a dynamical-systems view and reinterpret mode collapse as reduced state-space accessibility caused by *geometric collapse*: during generation, the model's internal trajectory becomes confined to a low-dimensional region of its representation space. This implies mode collapse is not purely a token-level phenomenon and cannot be reliably solved by symbolic constraints or probability-only decoding heuristics. Guided by this perspective, we propose *Reinforced Mode Regulation* (RMR), a lightweight, online state-space intervention that regulates dominant self-reinforcing directions in the Transformer value cache (implemented as low-rank damping). Across multiple large language models, RMR substantially reduces mode collapse and enables stable, high-quality generation at extremely low entropy rates (down to 0.8 nats/step), whereas standard decoding typically collapses near 2.0 nats/step.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Adaptive Iterative Hard Thresholding for Online High-dimensional Quantile Regression [1.0067421338825542]
  We propose an online sparse-regression framework that alternates subgradient updates with adaptively scheduled hard-thresholding steps.<n>We develop the theory for high-dimensional online quantile regression, a challenging setting in which the loss is nonsmooth.
  arXiv  Detail & Related papers  (2026-06-26T23:44:10Z)
• Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation [50.139984798361375]
  We propose a training-free framework that removes redundancy via latent discrepancy.<n> LD-Pruning substantially reduces inference while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
  arXiv  Detail & Related papers  (2026-05-29T19:34:39Z)
• Local Truncation Error-Guided Neural ODEs for Large Scale Traffic Forecasting [26.116856080963966]
  Local Truncation Error-Guided Neural ODEs (LTE-ODE)<n>Our architecture gracefully preserves high-precision continuous ODE evolution in stable regions.<n>LTE-ODE achieves state-of-the-art performance on multiple large-scale benchmarks.
  arXiv  Detail & Related papers  (2026-05-05T05:46:00Z)
• Allo{SR}$^2$: Rectifying One-Step Super-Resolution to Stay Real via Allomorphic Generative Flows [32.29950358173077]
  Real-world image super-resolution (Real-SR) has been revolutionized by leveraging the powerful generative priors of large-scale diffusion and flow-based models.<n>AlloSR$2$ is a novel framework that rectifies one-step SR trajectories via allomorphic generative flows to maintain high-fidelity generative realism.
  arXiv  Detail & Related papers  (2026-04-21T08:44:40Z)
• LEGATO: Good Identity Unlearning Is Continuous [30.78585550012454]
  LEGATO is a machine unlearning method for learning to forget identity of generative models.<n>It achieves state-of-the-art forgetting performance, avoids catastrophic collapse and reduces fine-tuned parameters.<n>Our experiments show that LEGATO achieves state-of-the-art forgetting performance, avoids catastrophic collapse and reduces fine-tuned parameters.
  arXiv  Detail & Related papers  (2026-01-07T13:15:25Z)
• RRAEDy: Adaptive Latent Linearization of Nonlinear Dynamical Systems [2.4662459762262894]
  We introduce RRAEDy, a model for learning low-dimensional dynamics in the latent space.<n>We show that RRAEDy achieves accurate and robust predictions.<n>Our code is open-source and available at https://github.com/JadM133/RRAEDy.
  arXiv  Detail & Related papers  (2025-12-08T13:23:12Z)
• MambaX: Image Super-Resolution with State Predictive Control [48.76194230142064]
  Mamba has emerged as a promising approach that can represent the entire reconstruction process as a state sequence with multiple nodes, allowing for intermediate intervention.<n>We created a nonlinear state predictive control model textbfMambaX that maps consecutive spectral bands into a latent state space and generalizes the SR task by dynamically learning the nonlinear state parameters of control equations.<n>Our evaluation demonstrates the superior performance of the dynamic spectrum-state representation model in both single-image SR and multimodal fusion-based SR tasks.
  arXiv  Detail & Related papers  (2025-11-22T11:44:09Z)
• On the Joint Minimization of Regularization Loss Functions in Deep Variational Bayesian Methods for Attribute-Controlled Symbolic Music Generation [47.38557855930304]
  Explicit latent variable models provide a flexible yet powerful framework for data synthesis.<n>We show that existing approaches struggle to jointly minimize both regularization objectives.<n>We show that suitable attribute transformations can help achieve both controllability and regularization of the target latent dimensions.
  arXiv  Detail & Related papers  (2025-11-10T14:09:25Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.