The Need for an External Observer Formalizing the Sufficiency Gap: A Mathematical Extension of Mixture Identifiability and Contextual Grounding in Sequence Models

• URL: http://arxiv.org/abs/2605.26711v2
• Date: Fri, 29 May 2026 12:17:42 GMT
• Title: The Need for an External Observer Formalizing the Sufficiency Gap: A Mathematical Extension of Mixture Identifiability and Contextual Grounding in Sequence Models
• Abstract summary: We construct a binary mixed-regime process with one deterministic textual regime and one random regime governed by an unobserved latent state.<n>The resulting entropy difference is not an ordinary optimization error; it is a sufficiency gap caused by marginalization over an unobserved state.<n>A corrective signal reverses the posterior odds induced by the textual history exactly when its fidelity exceeds the text-only posterior weight assigned to the misleading regime.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We construct a binary mixed-regime process with one deterministic textual regime and one random regime governed by an unobserved latent state. Even an ideal infinite-capacity sequence predictor that exactly recovers the text-only marginal law can become overconfident when the observed prefix is compatible with the wrong latent regime. The resulting entropy difference is not an ordinary optimization error; it is a sufficiency gap caused by marginalization over an unobserved state. We then formalize retrieval, tool use, and external grounding through an auxiliary binary signal with fidelity $γ\in [1/2,1]$. The resulting Bayesian update yields a contextual dominance threshold: a corrective signal reverses the posterior odds induced by the textual history exactly when its fidelity exceeds the text-only posterior weight assigned to the misleading regime. This threshold reduces, but does not generally eliminate, the sufficiency gap; complete closure requires perfect revelation of the relevant latent state or an equivalent verification mechanism. The analysis clarifies why temperature scaling cannot restore missing context, why grounding mechanisms must be both informative and learnably usable by the model, and why autonomous sequence models require structurally decoupled observers or verifiers in high-stakes domains.

Related papers

• Koopman-Based Robust Model Predictive Control for Nonlinear Systems with Stochastic Intermittent Measurements [11.153670606528545]
  Intermittent state measurements pose fundamental challenges to model predictive control of constrained nonlinear systems.<n>This paper develops a Koopman-based MPC framework with probabilistically truncated soft constraints.
  arXiv  Detail & Related papers  (2026-09-02T04:08:49Z)
• Simplex Relaxation for Discrete Diffusion [58.33696160279345]
  We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable.<n>Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation.
  arXiv  Detail & Related papers  (2026-08-11T07:58:17Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• Bayesian Repetition Penalty: A Principled Adjacent-Conditional Framework for Reversing Attention Collapse in Autoregressive Language Models [9.240704665480939]
  We present a principled framework that penalises or compensates anomalous confidence arising from collapsed generation patterns.<n>The resulting self-normalising penalty ratio $R=f(m,n,p)/f(np,n,p)$ requires no ad hoc standardisation and admits a closed-form logit offset with zero approximation error.
  arXiv  Detail & Related papers  (2026-07-17T17:01:57Z)
• Replicating Belief, Not Bits: Epistemic State Replication for Agentic Systems [2.124730017640531]
  In distributed systems, the classical State Machine Replication model assumes that correct replicas execute deterministic transitions to yield identical bitwise states.<n>We propose Epistemic State Replication (ESR), a belief-replication layer for agentic distributed systems that shifts the replication boundary from data visibility to knowledge visibility.
  arXiv  Detail & Related papers  (2026-07-03T19:59:03Z)
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics [49.443264461057645]
  Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.<n>Progress in both paradigms is overwhelmingly tracked by generative perplexity (gen-PPL): the per-token negative log-likelihood of samples under a frozen autoregressive (AR) scorer.<n>We argue that this metric is unsound. By construction, gen-PPL measures only predictability under the scoring AR, not grammaticality or semantic coherence.
  arXiv  Detail & Related papers  (2026-06-07T02:35:56Z)
• Learning Context-conditioned Gaussian Overbounds for Convolution-Based Uncertainty Propagation [4.891998873486944]
  We propose a unified learning framework that trains neural networks to produce context-aware Gaussian overbounds.<n>We provide a scoped analysis of discrete-to-continuous conservatism and compact-domain objective regularity.<n>The framework is modality-agnostic and applicable to learning systems that require conservative, feature-conditioned uncertainty estimates.
  arXiv  Detail & Related papers  (2026-05-15T09:47:05Z)
• The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation [18.83395586542378]
  Multimodal Large Language Models (MLLMs) are increasingly integrated with Retrieval-Augmented Generation (RAG)<n>We identify and formalize the phenomenon of recorruption, where the introduction of even perfectly accurate "oracle" context causes a capable model to abandon an initially correct prediction.<n>We propose Bottleneck Attention Intervention for Recovery (BAIR), a parameter-free, inference-time framework that restores visual saliency and applies position-aware penalties to textual distractors.
  arXiv  Detail & Related papers  (2026-05-07T02:27:04Z)
• Out of Context: Reliability in Multimodal Anomaly Detection Requires Contextual Inference [54.774686416991386]
  We argue that multimodal anomaly detection should be reframed as a cross-modal contextual inference problem.<n>This perspective has implications for model design, evaluation protocols, and benchmark construction.
  arXiv  Detail & Related papers  (2026-04-14T19:32:55Z)
• Bias Inheritance in Neural-Symbolic Discovery of Constitutive Closures Under Function-Class Mismatch [1.6249267147413524]
  We investigate the data-driven discovery of closures in nonlinear reaction-diffusion systems with known PDE structures.<n>We propose a three-stage neural-symbolic framework: learning numerical surrogates under physical constraints; compressing these surrogates into restrictedable symbolic families; and validating the symbolic closures through explicit forward re-simulation on unseen initial conditions.
  arXiv  Detail & Related papers  (2026-04-01T19:39:04Z)
• Verifying Closed-Loop Contractivity of Learning-Based Controllers via Partitioning [52.23804865017831]
  We address the problem of verifying closed-loop contraction in nonlinear control systems whose controller and contraction metric are both parameterized by neural networks.<n>We derive a tractable and scalable sufficient condition for closed-loop contractivity that reduces to checking that the dominant eigenvalue of a symmetric Metzler matrix is nonpositive.
  arXiv  Detail & Related papers  (2025-12-01T23:06:56Z)
• The Mirror Loop: Recursive Non-Convergence in Generative Reasoning Systems [0.0]
  Recursive self-evaluation without external feedback often yields reformulation rather than progress.<n>We study 144 reasoning sequences across three models (OpenAI GPT-4o-mini, Anthropic Claude 3 Haiku, and Google Gemini 2.0 Flash) and four task families (arithmetic, code, explanation, reflection)<n>We interpret this as evidence for a structural limit on self-correction in generative reasoning.
  arXiv  Detail & Related papers  (2025-10-23T07:53:26Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting.<n>In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)
• Neuro-Symbolic Entropy Regularization [78.16196949641079]
  In structured prediction, the goal is to jointly predict many output variables that together encode a structured object. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.
  arXiv  Detail & Related papers  (2022-01-25T06:23:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.