Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion

• URL: http://arxiv.org/abs/2605.22579v1
• Date: Thu, 21 May 2026 14:52:48 GMT
• Title: Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
• Abstract summary: We show that fine-tuning LLMs to near-zero training loss enhances open-ended generation quality and mitigates repetition in greedy decoding.<n>We introduce Late-Stage LoRA, a targeted fine-tuning strategy that updates only the final 5 layers, yielding robust generation with minimal parameter updates.
• Score: 1.7770888723114693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has identified a counterintuitive phenomenon termed "Hyperfitting", where fine-tuning Large Language Models (LLMs) to near-zero training loss on small datasets surprisingly enhances open-ended generation quality and mitigates repetition in greedy decoding. While effective, the underlying mechanism remains poorly understood, with the extremely low-entropy output distributions suggesting a potential equivalence to simple temperature scaling. In this work, we demonstrate that this phenomenon is fundamentally distinct from distribution sharpening; entropy-matched control experiments reveal that temperature scaling fails to replicate the diversity gains of hyperfitting. Furthermore, we falsify the hypothesis of static vocabulary reweighting, showing through ablation studies that hyperfitting relies on a dynamic, context-dependent rank reordering mechanism. Layer-wise analysis localizes this effect to a "Terminal Expansion" in the final transformer block, where a substantial geometric expansion of the feature space (Delta Dim approx +80.8) facilitates the promotion of deep-tail tokens. Additionally, we introduce Late-Stage LoRA, a targeted fine-tuning strategy that updates only the final 5 layers, yielding robust generation with minimal parameter updates

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Geometric Collapse: When Vision Models Fail to Verify Physical Causality [52.43819179934414]
  We propose Scrambled Edges, a controlled counterfactual that injects salient edge-like cues.<n>With energy-matched and structure-matched controls, we isolate the effect of unsupported edge evidence from high-frequency energy and edge sparsity.<n>Across CNN/ViT/SSL depth predictors, Scrambled Edges induce up to 3.2x larger deviation from clean predictions than energy-matched noise.
  arXiv  Detail & Related papers  (2026-07-08T00:04:02Z)
• Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development [0.13750624267664155]
  Confidence-based loss weighting is usually avoided in generative models because it accelerates errors when the model is confidently wrong.<n>We introduce the Eisbach log-barrier, a parameter-free weight derived from the entropy of the DiT output's spatial energy distribution.<n>It unexpectedly yields stronger thematic development, clearer acoustic differentiation, and higher textural diversity than unweighted training.
  arXiv  Detail & Related papers  (2026-06-05T12:19:00Z)
• Geometric and Dynamic Scaling in Deep Transformers [13.697614668609205]
  We argue that the collapse of deep Transformers is fundamentally a geometric problem.<n>We propose a unified geometric framework that addresses these failures through two principles.<n>Our analysis predicts that enforcing geometric validity while allowing dynamic erasure is essential for avoiding rank collapse in ultra-deep networks.
  arXiv  Detail & Related papers  (2026-01-03T00:41:46Z)
• PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction [87.33016661440202]
  Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality.<n>We propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions.<n> Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm.
  arXiv  Detail & Related papers  (2025-10-07T06:31:02Z)
• Tracing the Representation Geometry of Language Models from Pretraining to Post-training [22.18942718274405]
  We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training.<n>We uncover a consistent non-monotonic sequence of three geometric phases during autoregressive pretraining.<n>Post-training further transforms geometry: SFT and DPO drive "entropy-seeking" dynamics to integrate specific instructional or preferential data.
  arXiv  Detail & Related papers  (2025-09-27T00:46:29Z)
• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting.<n>In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)
• Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion [55.95767828747407]
  In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model.<n>We present a framework that reduces training variance and provides a provably lower-variance gradient estimator.<n>We also present a practical implementation of this estimator incorporating the loss and sampling procedure through a method we call Orbit Diffusion.
  arXiv  Detail & Related papers  (2025-02-14T03:26:57Z)
• Shrink the longest: improving latent space isotropy with symplicial geometry [0.0]
  We propose a novel regularization technique based on simplicial geometry to improve the isotropy of latent representations.<n>We demonstrate that the method leads to an increase in downstream performance while significantly lowering the anisotropy during fine-tuning.
  arXiv  Detail & Related papers  (2025-01-09T18:44:10Z)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression [53.15502562048627]
  Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator. This work delves into a statistical analysis of augmentation-based pretraining.
  arXiv  Detail & Related papers  (2023-06-01T15:18:55Z)
• Towards the Semantic Weak Generalization Problem in Generative Zero-Shot Learning: Ante-hoc and Post-hoc [89.68803484284408]
  We present a simple and effective strategy lowering the previously unexplored factors that limit the performance ceiling of generative Zero-Shot Learning (ZSL) We begin by formally defining semantic generalization, then look into approaches for reducing the semantic weak generalization problem. In the ante-hoc phase, we augment the generator's semantic input, as well as relax the fitting target of the generator.
  arXiv  Detail & Related papers  (2022-04-24T13:54:42Z)
• Eccentric Regularization: Minimizing Hyperspherical Energy without explicit projection [0.913755431537592]
  We introduce a novel regularizing loss function which simulates a pairwise repulsive force between items. We show that minimizing this loss function in isolation achieves a hyperspherical distribution. We apply this method of Eccentric Regularization to an autoencoder, and demonstrate its effectiveness in image generation, representation learning and downstream classification tasks.
  arXiv  Detail & Related papers  (2021-04-23T13:55:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.