Fine-Tuning Improves Information Conveyance in Language Models

• URL: http://arxiv.org/abs/2605.30844v1
• Date: Fri, 29 May 2026 05:05:01 GMT
• Title: Fine-Tuning Improves Information Conveyance in Language Models
• Abstract summary: We propose Canopy Entropy ($mathrmCEstar$), a measure that views language generation from a tree perspective.<n>$mathrmCEstar$ captures uncertainty in both the output length $N$ and the generated sequence $Y_1:N$.<n>We find that fine-tuning nearly triples the correlation strength between entropy rate and semantic diversity.
• Score: 34.13231597052981
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning is often believed to reduce uncertainty and diversity in large language models, but existing analyses overlook output length, a key confounder, and therefore fail to capture how uncertainty is distributed across an entire generation rollout. To address this, we propose Canopy Entropy ($\mathrm{CE}^\star$), a measure that views language generation from a tree perspective, where ``canopy'' represents the space of all possible rollouts, making $\mathrm{CE}^\star$ naturally quantify the effective size of the generation space. $\mathrm{CE}^\star$ jointly captures uncertainty in both the output length $N$ and the generated sequence $Y_{1:N}$ -- indeed, we show that it equals to total Shannon entropy $H(N, Y_{1:N}\mid X)$, where $X$ denotes the prompt. This formulation yields interpretable metrics, including a length-entropy correlation term $ρ(N, r_N)$, where $r_N$ is the entropy rate, quantifying information conveyance efficiency by indicating whether longer outputs are more or less informative per token. Empirically, across tasks and model families, we find that fine-tuned models consistently exhibit stronger positive correlation $ρ(N, r_N)$, even when total entropy decreases. Furthermore, after controlling for model family, task, prompt, and output-length effects, we find that fine-tuning nearly triples the correlation strength between entropy rate and semantic diversity, suggesting that aligned models convert token uncertainty into semantic diversity more efficiently. Overall, these results demonstrate that fine-tuning does not simply reduce uncertainty, but fundamentally reorganizes it into more informative and semantically meaningful generations. Our code is available at https://github.com/WeiyiTian/canopy-entropy.

Related papers

• Scaling limit of the Random Language Model [0.0]
  We develop a theory of the Random Language Model (RLM), an ensemble of context-free grammars.<n>We show that the RLM exhibits a condensation transition at a critical value $x_c=1/8$, below which rule usage concentrates and language statistics acquire a nontrivial dependence on corpus length.<n>Across these regimes, we derive explicit scaling laws for the number of distinct rules, entropy, and related observables.
  arXiv  Detail & Related papers  (2026-06-26T14:00:30Z)
• Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback [50.89125374999765]
  We provide the first convergence guarantee for Optimistic Multiplicative Weights Update ($mathtOMWU$) in NLHF.<n>Our analysis identifies a novel marginal convergence behavior, where the probability of rarely played actions grows exponentially from exponentially small values.
  arXiv  Detail & Related papers  (2025-12-31T12:08:29Z)
• Architecture-Aware Generalization Bounds for Temporal Networks: Theory and Fair Comparison Methodology [8.006116553957659]
  We provide the first non-vacuous, architecture-aware generalization bounds for deep temporal models.<n>For exponentially $beta$-mixing sequences, we derive bounds scaling as $ O!Bigl(R,sqrttfracD,p,n,log NNBigr), $ where $D$ is network depth, $p$ kernel size, $n$ input dimension, and $R$ weight norm.<n>Our delayed-feedback blocking mechanism transforms dependent samples into effectively independent ones while discarding only $O (1/log N
  arXiv  Detail & Related papers  (2025-08-08T06:57:49Z)
• Coupled Entropy: A Goldilocks Generalization for Complex Systems [2.1756081703276]
  The Tsallis entropy originated from considering power probabilities $p_iq$ in which textitq independent, identically-distributed random variables share the same state.<n>Unfortunately, the $q$-exponential parameters were treated as though valid substitutes for the shape and scale.<n>This flaw causes a misinterpretation of the generalized temperature and an imprecise derivation of the generalized entropy.
  arXiv  Detail & Related papers  (2025-05-17T19:00:25Z)
• A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data [32.98264375121064]
  We consider supervised deep learning when the given explanatory variable is distributed according to an exponential family.<n>Under the assumption of an upper-bounded density of the explanatory variables, we characterize the rate of convergence as $tildemathcalOleft( dfrac2lfloorbetarfloor(beta + d)2beta + dn-frac22beta + dn-frac22beta + dn-frac22beta + dn-
  arXiv  Detail & Related papers  (2024-12-13T01:15:17Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Effective Minkowski Dimension of Deep Nonparametric Regression: Function Approximation and Statistical Theories [70.90012822736988]
  Existing theories on deep nonparametric regression have shown that when the input data lie on a low-dimensional manifold, deep neural networks can adapt to intrinsic data structures. This paper introduces a relaxed assumption that input data are concentrated around a subset of $mathbbRd$ denoted by $mathcalS$, and the intrinsic dimension $mathcalS$ can be characterized by a new complexity notation -- effective Minkowski dimension.
  arXiv  Detail & Related papers  (2023-06-26T17:13:31Z)
• A Law of Robustness beyond Isoperimetry [84.33752026418045]
  We prove a Lipschitzness lower bound $Omega(sqrtn/p)$ of robustness of interpolating neural network parameters on arbitrary distributions. We then show the potential benefit of overparametrization for smooth data when $n=mathrmpoly(d)$. We disprove the potential existence of an $O(1)$-Lipschitz robust interpolating function when $n=exp(omega(d))$.
  arXiv  Detail & Related papers  (2022-02-23T16:10:23Z)
• Inductive Mutual Information Estimation: A Convex Maximum-Entropy Copula Approach [0.5330240017302619]
  We propose a novel estimator of the mutual information between two ordinal vectors $x$ and $y$. We prove that, so long as the constraint is feasible, this problem admits a unique solution, it is in the exponential family, and it can be learned by solving a convex optimization problem. We show that our approach may be used to mitigate mode collapse in GANs by maximizing the entropy of the copula of fake samples.
  arXiv  Detail & Related papers  (2021-02-25T21:21:40Z)
• Improving Robustness and Generality of NLP Models Using Disentangled Representations [62.08794500431367]
  Supervised neural networks first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$. We present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
  arXiv  Detail & Related papers  (2020-09-21T02:48:46Z)
• Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing [83.78668073898001]
  We introduce a family of entropy regularizers, which includes label smoothing as a special case. We find that variance in model performance can be explained largely by the resulting entropy of the model. We advise the use of other entropy regularization methods in its place.
  arXiv  Detail & Related papers  (2020-05-02T12:46:28Z)
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
  We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse. We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
  arXiv  Detail & Related papers  (2020-04-30T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.