Diminishing Returns in Expanding Generative Models and Godel-Tarski-Lob Limits

• URL: http://arxiv.org/abs/2603.19687v1
• Date: Fri, 20 Mar 2026 06:42:22 GMT
• Title: Diminishing Returns in Expanding Generative Models and Godel-Tarski-Lob Limits
• Abstract summary: We develop a general task-space framework for analysing expanding generative reasoning systems.<n>We show that, under mild assumptions, the marginal improvement in solved tasks must converge to zero as system capacity increases.
• Score: 11.62669179647184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern generative modelling systems are increasingly improved by expanding model capacity, training data, and computational resources. While empirical studies have documented such scaling behaviour across architectures including generative adversarial networks, variational autoencoders, transformer-based models, and diffusion models, the theoretical limits of capability growth in expanding generative systems remain poorly understood. In this paper we develop a general task-space framework for analysing expanding generative reasoning systems. Each system induces a subset of a global task space representing the tasks it can successfully solve, and system capability is measured by the probability mass of this solved-task set under a fixed task distribution. Within this framework we prove a structural result showing that, under mild assumptions, the marginal improvement in solved tasks must converge to zero as system capacity increases. Thus expanding generative systems may continue to gain capability, but the probability mass of newly solvable tasks necessarily diminishes asymptotically. We further provide a prediction-theoretic refinement based on complexity-weighted hypothesis classes inspired by algorithmic probability, yielding quantitative bounds on marginal improvement in prediction settings. Finally, we examine logical reasoning tasks and show that classical results from mathematical logic -- including Rosser incompleteness, Tarski's undefinability theorem, and Löb's theorem -- imply the persistence of unresolved logical tasks within sufficiently expressive reasoning systems. Together these results provide a mathematical perspective on the asymptotic behaviour of expanding generative systems, showing that long-run capability growth is constrained both by diminishing marginal improvements in task coverage and by fundamental logical limitations on internal reasoning.

Related papers

• Information as Maximum-Caliber Deviation: A bridge between Integrated Information Theory and the Free Energy Principle [51.56484100374058]
  This work proposes that information can be defined as the deviation $$ of realized dynamics from a constrained maximum-caliber (MaxCal) path ensemble over a finite time horizon.<n>Under this definition, each of the cause/effect repertoires central to IIT 3.0 emerge directly from MaxCal variational principles.<n>Information $$ is shown to be equivalent to prediction error under accompanying predictive coding models.
  arXiv  Detail & Related papers  (2026-05-03T07:22:55Z)
• A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws [69.11818285240689]
  This study attempts to develop a mathematical approach to formalize emergent intelligence from the perspective of limit theory.<n>We recast emergent intelligence as existence of the limit $lim_N,P,K to infty mathcalE(N,P,K)$, with emergent abilities corresponding to the limiting behavior.
  arXiv  Detail & Related papers  (2026-04-27T04:43:42Z)
• Algebraic Quantum Intelligence: A New Framework for Reproducible Machine Creativity [1.9823694436716954]
  Large language models (LLMs) have achieved remarkable success in generating fluent and contextually appropriate text.<n>This paper proposes Algebraic Quantum Intelligence (AQI) as a computational framework that enables systematic expansion of semantic space.
  arXiv  Detail & Related papers  (2026-02-15T13:02:57Z)
• Diffusion-Based Solver for CNF Placement on the Cloud-Continuum [1.529342790344802]
  A novel theoretical framework is proposed, which is based on Denoising Diffusion Probabilistic Models (DDPM) for CNF placement.<n>The model incorporates constraint-specific losses directly into the loss function, thereby allowing it to learn feasible solution spaces.<n>The results obtained demonstrate the potential of diffusion-based generative modelling for constrained network embedding problems.
  arXiv  Detail & Related papers  (2025-11-03T08:47:58Z)
• Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought [64.43689151961054]
  We theoretically analyze the training dynamics of a simplified two-layer transformer on the directed graph reachability problem.<n>Our analysis reveals that during training using continuous thought, the index-matching logit will first increase and then remain bounded under mild assumptions.
  arXiv  Detail & Related papers  (2025-09-27T15:23:46Z)
• Neural Expectation Operators [2.1756081703276]
  This paper introduces textbfMeasure Learning, a paradigm for modeling ambiguity via non-linear expectations.<n>We define Neural Expectation Operators as solutions to Backward Differential Equations (BSDEs) whose drivers are parameterized by neural networks.<n>We provide constructive methods for enforcing key axiomatic properties, such as convexity, by architectural design.
  arXiv  Detail & Related papers  (2025-07-13T06:19:28Z)
• Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning [73.18052192964349]
  We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics.<n>By lifting neural parameters to a measure space and modeling training as Wasserstein gradient flow, we show that under geometric constraints, the parameter measure $mu_t$ undergoes two concurrent phenomena.
  arXiv  Detail & Related papers  (2025-06-26T22:40:30Z)
• Systemic Constraints of Undecidability [0.0]
  This paper presents a theory of systemic undecidability, reframing incomputability as a structural property of systems.<n>We prove a closure principle: any subsystem that participates functionally in the computation of an undecidable system inherits its undecidability.<n>Our framework disarms oracle mimicry and challenges the view that computational limits can be circumvented through architectural innovation.
  arXiv  Detail & Related papers  (2025-06-21T22:56:26Z)
• Probabilistic neural operators for functional uncertainty quantification [14.08907045605149]
  We introduce the probabilistic neural operator (PNO), a framework for learning probability distributions over the output function space of neural operators.<n>PNO extends neural operators with generative modeling based on strictly proper scoring rules, integrating uncertainty information directly into the training process.
  arXiv  Detail & Related papers  (2025-02-18T14:42:11Z)
• A Novel Neural-symbolic System under Statistical Relational Learning [47.30190559449236]
  We propose a neural-symbolic framework based on statistical relational learning, referred to as NSF-SRL.<n>Results of symbolic reasoning are utilized to refine and correct the predictions made by deep learning models, while deep learning models enhance the efficiency of the symbolic reasoning process.<n>We believe that this approach sets a new standard for neural-symbolic systems and will drive future research in the field of general artificial intelligence.
  arXiv  Detail & Related papers  (2023-09-16T09:15:37Z)
• Semantic Probabilistic Layers for Neuro-Symbolic Learning [83.25785999205932]
  We design a predictive layer for structured-output prediction (SOP) It can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints. Our Semantic Probabilistic Layer (SPL) can model intricate correlations, and hard constraints, over a structured output space.
  arXiv  Detail & Related papers  (2022-06-01T12:02:38Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.