The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path

• URL: http://arxiv.org/abs/2607.20484v1
• Date: Tue, 02 Jun 2026 09:11:50 GMT
• Title: The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path
• Abstract summary: The Topologically Regularized Side-Path (TRSP) is a non-invasive architectural intervention that achieves spectral balance.<n>TRSP employs a parameter-free Triangular Box mechanism, scaled by a lightweight, length-aware gate, to regularize the token interaction topology.<n>Experiments show significant improvements across general capabilities and long-context benchmarks.
• Score: 10.000577340093434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are fundamentally limited by representation collapse, a bottleneck that severely degrades long-context performance. We identify that existing approaches risk drifting into one of two pathological extremes: homogenization collapse (e.g., attention sinks causing rank deficiency) and isolation collapse (e.g., local attention causing context disconnection). Through spectral analysis of attention dynamics, we derive an intrinsic trade-off between mixing efficiency (spectral gap) and information capacity (effective rank) that standard mechanisms struggle to balance. To resolve this dilemma, we propose the Topologically Regularized Side-Path (TRSP), a non-invasive architectural intervention that achieves spectral balance. TRSP employs a parameter-free Triangular Box mechanism, scaled by a lightweight, length-aware gate, to regularize the token interaction topology. By integrating proximal coupling to preserve effective rank and distal propagation to support non-degenerate mixing, TRSP promotes a geometrically healthier transition operator without altering core attention. Experiments show significant improvements across general capabilities and long-context benchmarks. Notably, on NoLiMa at $8\times$ the training length, TRSP retains $83\%$ accuracy and surpasses the Differential Transformer and Gated Attention by approximately 30 and 50 percentage points, respectively. Code available at: https://github.com/Eziotao-tyd/TRSP.

Related papers

• From local weight selection to Zeno slowdown in an open Su-Schrieffer-Heeger chain with a single local loss [0.0]
  We show that the many-body fermionic Lindblad problem admits an exact reduction to a finite non-Hermitian one-body matrix with a rank-one imaginary impurity.<n>Its rapidities generate the complete Liouvillian spectrum and reveal three mechanisms governing the slowest relaxation.
  arXiv  Detail & Related papers  (2026-07-11T09:01:55Z)
• Trajectory-Anchor Optimization for Overconfident Thermal Visual Place Recognition: Zero-Leakage OOD Auditing and Kidnapped-Robot Recovery [3.9929096403817255]
  Trajectory-Anchor Optimization (TAO) is an efficient fail-safe filter at a macro-scale.<n>Within a 5m radius, hallucinations often possess a locally consistent geometry that deceives rigid alignment.<n>TAO establishes a distinct convergence basin (10m) where multi-view consistency reliably isolates catastrophic topological breaks.
  arXiv  Detail & Related papers  (2026-07-06T07:31:37Z)
• Self-Balancing Gradient Allocation for Heterogeneity-Aware Feature Generation in Click-Through Rate Prediction [9.542597285477683]
  HeteGenCTR resolves the generative difficulty imbalance through per-field learnable difficulty parameters jointly trained with the denoising network.<n> Experiments on five CTR benchmarks and a seven-day online A/B test demonstrate consistent, statistically significant improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-24T10:27:31Z)
• Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold [0.0]
  Generalization requires the textitanisotropic rectification unique to adaptives, which directs noise into the tangent space of the solution.<n>This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a Spectral Gating'' mechanism that regulates the transition from generalization to memorization.
  arXiv  Detail & Related papers  (2026-03-16T16:18:18Z)
• The Malignant Tail: Spectral Segregation of Label Noise in Over-Parameterized Networks [0.0]
  We experimentally isolate the Malignant Tail, a failure mode where networks functionally segregate signal and noise.<n>We show that untrained networks actively segregate noise, allowing post-hoc Explicit Spectral Truncation to surgically prune the noise-dominated subspace.<n>Our findings suggest that under label noise, excess spectral capacity is not harmless redundancy but a latent structural liability.
  arXiv  Detail & Related papers  (2026-03-02T16:39:42Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Parallel Complex Diffusion for Scalable Time Series Generation [50.01609741902786]
  PaCoDi is a spectral-native architecture that decouples generative modeling in the frequency domain.<n>We show that PaCoDi outperforms existing baselines in both generation quality and inference speed.
  arXiv  Detail & Related papers  (2026-02-10T14:31:53Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• MirrorLA: Reflecting Feature Map for Vision Linear Attention [49.41670925034762]
  Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance.<n>We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation.<n>MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
  arXiv  Detail & Related papers  (2026-02-04T09:14:09Z)
• Multi-Scale Negative Coupled Information Systems (MNCIS): A Unified Spectral Topology Framework for Stability in Turbulence, AI, and Biology [1.4213973379473657]
  This work generalizes the Multi-Scale Negative Coupled Information System (MNCIS) framework.<n>Global stability requires an active topological operator -- Adaptive Spectral Negative Coupling (ASNC) -- functioning as a state-dependent high-pass filter.<n>ASNC acts as a global-enstrophy adaptive sub-grid scale (SGS) model, stabilizing the inviscid limit and preserving the Kolmogorov $-5/3$ inertial range without artificial hyper-viscosity.<n>Our results suggest that the MNCIS framework provides a base-independent topological condition for distinguishing viable complex systems from those collapsing into thermal equilibrium
  arXiv  Detail & Related papers  (2026-01-06T21:11:33Z)
• CLAReSNet: When Convolution Meets Latent Attention for Hyperspectral Image Classification [20.37811669228711]
  CLAReSNet is a hybrid architecture that integrates multi-scale convolutional extraction with transformer-style attention via an adaptive latent bottleneck.<n> Experiments conducted on the Indian Pines and Salinas datasets show state-of-the-art performance, achieving overall accuracies of 99.71% and 99.96%.
  arXiv  Detail & Related papers  (2025-11-15T20:25:59Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• CWF: Consolidating Weak Features in High-quality Mesh Simplification [50.634070540791555]
  We propose a smooth functional that simultaneously considers all of these requirements. The functional comprises a normal anisotropy term and a Centroidal Voronoi Tessellation (CVT) energy term.
  arXiv  Detail & Related papers  (2024-04-24T05:37:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.