How Alignment Shrinks the Generative Horizon

• URL: http://arxiv.org/abs/2506.17871v1
• Date: Sun, 22 Jun 2025 02:00:37 GMT
• Title: How Alignment Shrinks the Generative Horizon
• Authors: Chenghao Yang, Ari Holtzman,
• Abstract summary: Branching Factor (BF) is a token-invariant measure of the effective number of plausible next steps during generation.<n> alignment tuning substantially sharpens the model's output distribution from the outset.<n>Building on this insight, we find this stability has surprising implications for complex reasoning.
• Score: 20.243063721305116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite their impressive capabilities, aligned large language models (LLMs) often generate outputs that lack diversity. What drives this stability in the generation? We investigate this phenomenon through the lens of probability concentration in the model's output distribution. To quantify this concentration, we introduce the Branching Factor (BF) -- a token-invariant measure of the effective number of plausible next steps during generation. Our empirical analysis reveals two key findings: (1) BF often decreases as generation progresses, suggesting that LLMs become more predictable as they generate. (2) alignment tuning substantially sharpens the model's output distribution from the outset, reducing BF by nearly an order of magnitude (e.g., from 12 to 1.2) relative to base models. This stark reduction helps explain why aligned models often appear less sensitive to decoding strategies. Building on this insight, we find this stability has surprising implications for complex reasoning. Aligned Chain-of-Thought (CoT) models (e.g., DeepSeek-distilled models), for instance, leverage this effect; by generating longer reasoning chains, they push generation into later, more deterministic (lower BF) stages, resulting in more stable outputs. We hypothesize that alignment tuning does not fundamentally change a model's behavior, but instead steers it toward stylistic tokens (e.g., "Sure") that unlock low-entropy trajectories already present in the base model. This view is supported by nudging experiments, which show that prompting base models with such tokens can similarly reduce BF. Together, our findings establish BF as a powerful diagnostic for understanding and controlling LLM outputs - clarifying how alignment reduces variability, how CoT promotes stable generations, and how base models can be steered away from diversity.

Related papers

• A Convergence Theory for Diffusion Language Models: An Information-Theoretic Perspective [8.15094483029656]
  Diffusion models enable parallel token sampling, leading to faster generation and eliminating left-to-right generation constraints.<n>We develop convergence guarantees for diffusion language models from an information-theoretic perspective.<n>These results offer novel theoretical insights into the practical effectiveness of diffusion language models.
  arXiv  Detail & Related papers  (2025-05-27T16:24:20Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness.<n>We generalize a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes.<n>Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• On the Statistical Capacity of Deep Generative Models [10.288413514555861]
  We show that deep generative models can only generate concentrated samples that exhibit light tails.<n>These results shed light on the limited capacity of common deep generative models to handle heavy tails.
  arXiv  Detail & Related papers  (2025-01-14T00:39:46Z)
• Continuous Speculative Decoding for Autoregressive Image Generation [33.05392461723613]
  Continuous-valued Autoregressive (AR) image generation models have demonstrated notable superiority over their discrete-token counterparts. speculative decoding has proven effective in accelerating Large Language Models (LLMs) This work generalizes the speculative decoding algorithm from discrete tokens to continuous space.
  arXiv  Detail & Related papers  (2024-11-18T09:19:15Z)
• Language Models Resist Alignment: Evidence From Data Compression [11.208226196119895]
  Large language models (LLMs) may exhibit unintended or undesirable behaviors.<n>We show that fine-tuning disproportionately undermines alignment relative to pre-training, potentially by orders of magnitude.<n>Our findings underscore the need to address the inherent elasticity of LLMs to mitigate their resistance to alignment.
  arXiv  Detail & Related papers  (2024-06-10T10:03:16Z)
• Language Model Cascades: Token-level uncertainty and beyond [65.38515344964647]
  Recent advances in language models (LMs) have led to significant improvements in quality on complex NLP tasks. Cascading offers a simple strategy to achieve more favorable cost-quality tradeoffs. We show that incorporating token-level uncertainty through learned post-hoc deferral rules can significantly outperform simple aggregation strategies.
  arXiv  Detail & Related papers  (2024-04-15T21:02:48Z)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [67.9215891673174]
  We propose score entropy as a novel loss that naturally extends score matching to discrete spaces. We test our Score Entropy Discrete Diffusion models on standard language modeling tasks.
  arXiv  Detail & Related papers  (2023-10-25T17:59:12Z)
• Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models [49.81937966106691]
  We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
  arXiv  Detail & Related papers  (2023-06-15T16:30:08Z)
• Spontaneous Symmetry Breaking in Generative Diffusion Models [6.4322891559626125]
  Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. We show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides the generative dynamics into two distinct phases. We propose a new way to understand the generative dynamics of diffusion models that has the potential to bring about higher performance and less biased fast-samplers.
  arXiv  Detail & Related papers  (2023-05-31T09:36:34Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)
• Estimation of Bivariate Structural Causal Models by Variational Gaussian Process Regression Under Likelihoods Parametrised by Normalising Flows [74.85071867225533]
  Causal mechanisms can be described by structural causal models. One major drawback of state-of-the-art artificial intelligence is its lack of explainability.
  arXiv  Detail & Related papers  (2021-09-06T14:52:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.