Information-Theoretic Limits of Reliability and Scaling in Language Models

• URL: http://arxiv.org/abs/2607.14112v1
• Date: Fri, 08 May 2026 06:06:18 GMT
• Title: Information-Theoretic Limits of Reliability and Scaling in Language Models
• Abstract summary: Large language models (LLMs) are evaluated as though perfect reliability is achievable for any task given sufficient scale.<n>We show this assumption is information-theoretically unjustified.<n>Our work formalizes the resource-complexity tradeoffs that govern model performance across domains.
• Score: 6.658166782006281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are evaluated as though perfect reliability is achievable for any task given sufficient scale. We show this assumption is information-theoretically unjustified. Every generative task has a reliability ceiling that no model can exceed, determined by how much output uncertainty is resolvable from observable context. The gap decomposes into a resolvable component closable with additional context and a subjective component inherent to task ambiguity. Autoregressive generation further degrades this ceiling at a rate governed by the task's dependency kernel, which quantifies inter-token correlations in the output. From these two primitives, we derive a first-principles scaling law where LLM performance is bottlenecked by the scarcer resource: training data or model capacity. This law recovers the Chinchilla scaling law as a special case and provides a structural account of when scaling improves reliability. Beyond scaling, our framework unifies diverse practical phenomena, such as the benefits of retrieval-augmentation and the spectral mechanics of catastrophic forgetting. Our work formalizes the resource-complexity tradeoffs that govern model performance across domains, offering a unified theory of performance limits in generative language models.

Related papers

• CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation [9.80652710081054]
  CLSGen is a novel fine-tuning framework for binary classification tasks.<n>It enables robust probability estimation without sacrificing the model's inherent explanation-generation capabilities.<n>Results show that models fine-tuned with CLSGen outperform existing baselines in classification metrics.
  arXiv  Detail & Related papers  (2026-04-13T17:58:43Z)
• From Garbage to Gold: A Data-Architectural Theory of Predictive Robustness [0.3277163122167433]
  We show that high-D sets of error-prone predictors overcome both types of noise.<n>We propose "Proactive Data-Centric AI" to identify predictors that enable robustness efficiently.<n>By redefining data quality from item-level perfection to portfolio-level architecture, we provide a theoretical rationale for "Local Factories"
  arXiv  Detail & Related papers  (2026-03-08T18:58:33Z)
• Quantifying construct validity in large language model evaluations [0.0]
  LLM community often reports benchmark results as if they are synonymous with general model capabilities.<n> benchmarks can have problems that distort performance, like test set contamination and annotator error.<n>How can we know that a benchmark is a reliable indicator of some capability that we want to measure?
  arXiv  Detail & Related papers  (2026-02-17T12:15:57Z)
• Breaking the Factorization Barrier in Diffusion Language Models [59.946071582340146]
  "Factorization barrier" impedes efficient parallel generation of diffusion language models.<n>We propose Coupled Discrete Diffusion to replace the fully-factorized output distribution.<n>We show that CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead.
  arXiv  Detail & Related papers  (2026-02-09T08:36:39Z)
• Toward Faithful and Complete Answer Construction from a Single Document [1.0742675209112622]
  We present EVE, a structured framework for document-grounded reasoning.<n>Unlike free-form prompting, EVE constrains generation to a structured, verifiable pipeline that decomposes high-rigor reasoning into extraction, validation, and enumeration.
  arXiv  Detail & Related papers  (2026-02-05T18:22:08Z)
• SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment [49.86376148975563]
  Large language models (LLMs) have revolutionized natural language processing through their capabilities in understanding and executing diverse tasks.<n> supervised fine-tuning, particularly in Retrieval-Augmented Generation (RAG) scenarios, often leads to catastrophic forgetting.<n>We propose SelfAug, a self-distribution alignment method that aligns input sequence logits to preserve the model's semantic distribution.
  arXiv  Detail & Related papers  (2025-09-04T06:50:47Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Has LLM Reached the Scaling Ceiling Yet? Unified Insights into LLM Regularities and Constraints [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their scalability raises a critical question: Have we reached the scaling ceiling?<n>This paper develops a unified theoretical framework that integrates mathematical and statistical insights to explain the scaling dynamics of LLMs.<n>Future progress will require a shift from brute-force scaling to innovations in architecture, data quality, and training paradigms.
  arXiv  Detail & Related papers  (2024-12-21T02:19:07Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.