How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL

• URL: http://arxiv.org/abs/2609.11132v1
• Date: Thu, 10 Sep 2026 06:22:53 GMT
• Title: How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL
• Abstract summary: An unbounded gap between the update maps can coexist with vanishing predictive KL for every fixed finite $Kge2$ in a stationary symmetric Gaussian HMM.<n>The construction is fixed in $K$ and does not provide a universal criterion for when compression is harmless or characterize when internal gaps must incur task loss.
• Score: 8.929322780616545
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate posterior prediction need not require accurate approximation of Bayesian updates. We prove that an unbounded gap between the update maps can coexist with vanishing predictive KL for every fixed finite $K\ge2$ in a stationary symmetric Gaussian HMM. Exact Bayesian mixing and an explicit deterministic radial filter act on the same $K-1$ belief coordinates. As $q\to0^+$, their separation in centered logits in the worst case grows at least linearly in the natural confidence scale $L_K(q)$, while their categorical $D_{\mathrm{KL}}(\mathrm{exact}\|\mathrm{radial})$ vanishes at the same explicit witness. Along stationary HMM trajectories, the expected terminal KL between filtered posteriors also converges to zero at $H(q)=\lceil-\log(q)/c\rceil+1$. Typical blocks without switches drive both filters into a common confidence cone, where softmax curvature suppresses their disagreement; a single Gaussian maximal event controls adaptive noise. A sweep with equally spaced Gaussians over $K\in\{2,4,8\}$ illustrates the opposing trends, and binary controls at long horizons compare saturating and nonsaturating recurrences. The result isolates two missing links between internal update gaps and predictive cost: the contribution of separating states to expected loss and decoder sensitivity. Thus even an unbounded internal update gap does not by itself certify predictive failure. The construction is fixed in $K$ and does not provide a universal criterion for when compression is harmless or characterize when internal gaps must incur task loss.

Related papers

• Constrained Online Learning with Noisy Constraint Values [55.29259818039367]
  LEDGER tracks observed net consumption in a nonnegative balance.<n>LEDGER achieves $O(sqrt T/V)$ expected regret and $O(sqrt V,T3/4+sqrt T)$ expected budget violation for $Vin[T-1/2,1]$.
  arXiv  Detail & Related papers  (2026-09-07T01:38:41Z)
• Demixing Sparse Signals from Nonlinear Observations using Generalized Non-convex Regularization [0.0]
  We consider the recovery of a pair of sparse vectors from a limited number of nonlinear observations of their superposition.<n>We propose a regularization-based framework combining a Huberized data fidelity with generalized folded-concave penalties.<n>We derive estimation error bounds of order $sqrtslog(n)/m$ that hold at emphevery localized stationary point.
  arXiv  Detail & Related papers  (2026-07-12T07:29:39Z)
• Asymptotic-Preserving A Posteriori Analysis of Diffusion and Flow-Matching Samplers [0.6768558752130311]
  Diffusion and flow-matching samplers integrate a learned probability-flow ODE from a large noise scale down to a small terminal floor $_min$, at which the score is stiff and the flow develops a boundary layer.<n>We treat $_min$ as a singular-perturbation parameter and determine which fixed-step samplers are solvable.
  arXiv  Detail & Related papers  (2026-07-05T04:48:33Z)
• Rao-Blackwellized Score Matching on Manifolds [0.0]
  We study denoising score matching (DSM) when the latent distribution is supported on a smooth embedded manifold.<n>We show that conditioning on the nearest-point projection $(X)$ canonically removes this singularity.<n>We then compute the small-noise expansion of this canonical target.
  arXiv  Detail & Related papers  (2026-05-25T08:18:45Z)
• Bandit Convex Optimization with Gradient Prediction Adaptivity [56.816177049016794]
  We investigate whether optimistic gradient predictions can improve worst-case regret guarantees in a prediction-adaptive manner.<n>The key idea is a novel variance-reduced gradient estimator whose variance scales with the prediction error rather than the gradient norm.<n>We establish an information-theoretic lower bound that scales as $(sqrtmathbbE[S_T])$, providing a fundamental characterization of the best achievable prediction-adaptive regret.
  arXiv  Detail & Related papers  (2026-05-21T08:57:38Z)
• Adaptive Confidence Intervals in Efron's Gaussian Two-Groups Model [28.636700996382558]
  Motivated by noise-oblivious nature of adversaries, we study confidence intervals for the null location parameter $$ in Efron's Gaussian two-groups fraction model.<n>We characterize the minimax-optimal length among confidence intervals with a prescribed coverage level uniformly over the unknown proportion contamination and all noise-oblivious adversaries.
  arXiv  Detail & Related papers  (2026-04-28T22:40:54Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Scale-Invariant Regret Matching and Online Learning with Optimal Convergence: Bridging Theory and Practice in Zero-Sum Games [60.871651115241406]
  A considerable chasm has been looming for decades between theory and practice in zero-sum game solving through first-order methods.<n>We propose a new scale-invariance and parameter-free variant of PRM$+$, which we call IREG-PRM$+$.<n>We show that it achieves $T-1/2$ best-iterate and $T-1$ optimal convergence guarantees, while also being on par with PRM$+$ on benchmark games.
  arXiv  Detail & Related papers  (2025-10-06T00:33:20Z)
• Anchor-MoE: A Mean-Anchored Mixture of Experts For Probabilistic Regression [0.0]
  We present an Anchored Mixture of Experts (Anchor-MoE) model that handles both probabilistic and point regression.<n>We show that Anchor-MoE attains the minimax-optimal $L2$ risk rate.<n>It consistently matches or surpasses the strong NG baseline in RMSE and NLL.
  arXiv  Detail & Related papers  (2025-08-22T21:12:41Z)
• Conformal Predictions under Markovian Data [50.24983453990065]
  We study the split Conformal Prediction method when applied to Markovian data. We quantify the gap in terms of coverage induced by the correlations in the data. We present $K$-split CP, a method that adapts to the mixing properties of the chain.
  arXiv  Detail & Related papers  (2024-07-21T22:01:09Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Online nonparametric regression with Sobolev kernels [99.12817345416846]
  We derive the regret upper bounds on the classes of Sobolev spaces $W_pbeta(mathcalX)$, $pgeq 2, beta>fracdp$. The upper bounds are supported by the minimax regret analysis, which reveals that in the cases $beta> fracd2$ or $p=infty$ these rates are (essentially) optimal.
  arXiv  Detail & Related papers  (2021-02-06T15:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.