Representational Curvature Modulates Behavioral Uncertainty in Large Language Models

• URL: http://arxiv.org/abs/2604.23985v1
• Date: Mon, 27 Apr 2026 03:00:47 GMT
• Title: Representational Curvature Modulates Behavioral Uncertainty in Large Language Models
• Abstract summary: temporal straightening offers an account of how the next-token prediction objective shapes representations.<n>In large language models (LLMs), temporal straightening offers an account of how the next-token prediction objective shapes representations.<n>We provide such a link by relating contextual curvature-a geometric measure of how sharply the representational trajectory bends over recent context-to next-token entropy.
• Score: 7.361948338002342
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In autoregressive large language models (LLMs), temporal straightening offers an account of how the next-token prediction objective shapes representations. Models learn to progressively straighten the representational trajectory of input sequences across layers, potentially facilitating next-token prediction via linear extrapolation. However, a direct link between this trajectory and token-level behavior has been missing. We provide such a link by relating contextual curvature-a geometric measure of how sharply the representational trajectory bends over recent context-to next-token entropy. Across two models (GPT-2 XL and Pythia-2.8B), contextual curvature is correlated with entropy, and this relationship emerges during training. Perturbation experiments reveal selective dependence: manipulating curvature through trajectory-aligned interventions reliably modulates entropy, while geometrically misaligned perturbations have no effect. Finally, regularizing representations to be straighter during training modestly reduces token-level entropy without degrading validation loss. These results identify trajectory curvature as a task-aligned representational feature that influences behavioral uncertainty in LLMs.

Related papers

• Curved Inference II: Sleeper Agent Geometry - Extending Interpretability Beyond Probes [0.0]
  This paper extends Anthropic's Sleeper Agents research, which showed artificial backdoors persist through safety training & can be detected by linear probes with >99% accuracy.<n>We introduce a naturalistic methodology using multi-turn context windows that simulates realistic deceptive reasoning without artificial triggers or supervised backdoor insertion.<n> Geometric structure reliably predicts semantic classification, with statistically significant differences in surface area across five prompt strategies & two model families.
  arXiv  Detail & Related papers  (2026-08-25T03:51:20Z)
• Probability-Conserving Flow Guidance [49.03107678763765]
  Adaptive Manifold Guidance (AdaMaG) improves realism, reduces hallucinations, and controlled desaturation in high-guidance.<n>Across image generation benchmarks, AdaMaG improves realism, reduces hallucinations, and controlled desaturation in high-guidance.
  arXiv  Detail & Related papers  (2026-05-19T16:34:01Z)
• Impact of Connectivity on Laplacian Representations in Reinforcement Learning [9.306521175972588]
  We show an upper bound on the approximation error of linear value function approximation under the learned spectral features.<n>We further bound the error introduced by the eigenvector estimation itself, leading to an end-to-end error decomposition.<n>Our results hold for general (non-uniform) policies without any assumptions on the symmetry of the induced transition kernel.
  arXiv  Detail & Related papers  (2026-03-09T16:20:31Z)
• Probing the Geometry of Diffusion Models with the String Method [13.771271465889432]
  We introduce a framework based on the string method that computes continuous paths between samples by evolving curves under the learned score function.<n> operating on pretrained models without retraining, our approach interpolates between three regimes.<n>For image diffusion models, MEPs contain high-likelihood but unrealistic ''cartoon'' images, confirming prior observations that likelihood maxima appear unrealistic.<n>For protein structure prediction, our method computes transition pathways between metastable conformers directly from models trained on static structures.
  arXiv  Detail & Related papers  (2026-02-25T17:10:59Z)
• Binary Flow Matching: Prediction-Loss Space Alignment for Robust Learning [23.616336786063552]
  Flow matching has emerged as a powerful framework for generative modeling.<n>We identify a latent structural mismatch that arises when it is coupled with velocity-based objectives.<n>We prove that re-aligning the objective to the signal space eliminates the singular weighting.
  arXiv  Detail & Related papers  (2026-02-11T02:02:30Z)
• The Curved Spacetime of Transformer Architectures [0.3670422696827525]
  We present a geometric framework for understanding Transformer-based language models, drawing an explicit analogy to General Relativity.<n>We show that token embeddings should not traverse straight paths in feature space; instead, their layer-wise steps should bend and reorient as interactions mediated by embedding space curvature.
  arXiv  Detail & Related papers  (2025-11-04T22:58:40Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [47.55215041326702]
  We discover an intriguing linear phenomenon in models that are from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL) We show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linearities of features in two finetuned models at each layer. We conjecture that in the pretraining-finetuning paradigm, neural networks approximately function as linear maps, mapping from the parameter space to the feature space.
  arXiv  Detail & Related papers  (2024-02-06T03:28:36Z)
• A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning [68.76846801719095]
  We show that double descent appears exactly when and where it occurs, and that its location is not inherently tied to the threshold p=n. This provides a resolution to tensions between double descent and statistical intuition.
  arXiv  Detail & Related papers  (2023-10-29T12:05:39Z)
• Gradient-Based Feature Learning under Structured Data [57.76552698981579]
  In the anisotropic setting, the commonly used spherical gradient dynamics may fail to recover the true direction. We show that appropriate weight normalization that is reminiscent of batch normalization can alleviate this issue. In particular, under the spiked model with a suitably large spike, the sample complexity of gradient-based training can be made independent of the information exponent.
  arXiv  Detail & Related papers  (2023-09-07T16:55:50Z)
• Score-based Causal Representation Learning with Interventions [54.735484409244386]
  This paper studies the causal representation learning problem when latent causal variables are observed indirectly. The objectives are: (i) recovering the unknown linear transformation (up to scaling) and (ii) determining the directed acyclic graph (DAG) underlying the latent variables.
  arXiv  Detail & Related papers  (2023-01-19T18:39:48Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.