Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-Context

• URL: http://arxiv.org/abs/2603.10573v1
• Date: Wed, 11 Mar 2026 09:25:53 GMT
• Title: Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-Context
• Abstract summary: In-context learning (ICL) allows Transformers to adapt to novel tasks without weight updates.<n>We adopt a statistical decision-theoretic perspective by investigating simple binary hypothesis testing.<n>We show that ICL emerges from the construction of task-adaptive statistical estimators rather than simple similarity matching.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) allows Transformers to adapt to novel tasks without weight updates, yet the underlying algorithms remain poorly understood. We adopt a statistical decision-theoretic perspective by investigating simple binary hypothesis testing, where the optimal policy is determined by the likelihood-ratio test. Notably, this setup provides a mathematically rigorous setting for mechanistic interpretability where the target algorithmic ground truth is known. By training Transformers on tasks requiring distinct geometries (linear shifted means vs. nonlinear variance estimation), we demonstrate that the models approximate the Bayes-optimal sufficient statistics from context up to some monotonic transformation, matching the performance of an ideal oracle estimator in nonlinear regimes. Leveraging this analytical ground truth, mechanistic analysis via logit lens and circuit alignment suggests that the model does not rely on a fixed kernel smoothing heuristic. Instead, it appears to adapt the point at which decisions become linearly decodable: exhibiting patterns consistent with a voting-style ensemble for linear tasks while utilizing a deeper sequential computation for nonlinear tasks. These findings suggest that ICL emerges from the construction of task-adaptive statistical estimators rather than simple similarity matching.

Related papers

• Statistical Inference for Misspecified Contextual Bandits [6.178061357164435]
  We show a previously overlooked issue: standard algorithms may fail to stabilize under misspecified working models.<n>We develop an inverse-probability-weighted Z-estimation framework for a broad class of marginal moment targets.<n>We establish sufficient conditions for scaled inverse-propensity convergence for several policy classes.
  arXiv  Detail & Related papers  (2026-06-21T18:57:13Z)
• AdaKoop: Efficient Modeling of Nonlinear Dynamics from Nonstationary Data Streams with Koopman Operator Regression [16.514388442204446]
  We present AdaKoop, an efficient streaming algorithm for modeling nonlinear dynamics over nonstationary data streams.<n>AdaKoop enables the efficient and stable modeling of nonlinear dynamics in a streaming fashion, avoiding the prohibitive computational costs of iterative nonlinear optimization.
  arXiv  Detail & Related papers  (2026-06-03T14:23:32Z)
• Non-Parametric Rehearsal Learning via Conditional Mean Embeddings [88.89267783967263]
  We propose a non-parametric rehearsal learning approach for textitavoiding undesired future (AUF) problems.<n>Specifically, we use kernel machinery to reformulate the AUF objective into a unified representation that disentangles desirability modeling from action-induced distributional changes.
  arXiv  Detail & Related papers  (2026-05-09T15:30:52Z)
• In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning [51.56484100374058]
  We introduce a principled risk decomposition that separates the total ICL risk into two components: Bayes Gap and Posterior Variance.<n>For a uniform-attention Transformer, we derive a non-asymptotic upper bound on this gap, which explicitly clarifies the dependence on the number of pretraining prompts.<n>The Posterior Variance is a model-independent risk representing the intrinsic task uncertainty.
  arXiv  Detail & Related papers  (2025-10-13T03:42:31Z)
• Statistical Inference for Misspecified Contextual Bandits [6.178061357164435]
  Contextual bandit algorithms have transformed modern experimentation by enabling real-time adaptation for personalized treatment.<n>Yet these advantages create challenges for statistical inference due to adaptivity.<n> Convergence ensures replicability of adaptive experiments and stability of online algorithms.
  arXiv  Detail & Related papers  (2025-09-08T02:19:37Z)
• Adaptive Learning-based Surrogate Method for Stochastic Programs with Implicitly Decision-dependent Uncertainty [1.5412450351033007]
  We consider a class of programming problems where the implicitly decision-dependent random variable follows a nonparametric regression model with heteroscedastic error.<n>We develop an adaptive learning-based surrogate method that integrates the simulation scheme and statistical estimates to construct estimation-based surrogate functions.
  arXiv  Detail & Related papers  (2025-05-12T07:35:06Z)
• Adaptive Conformal Inference by Betting [51.272991377903274]
  We consider the problem of adaptive conformal inference without any assumptions about the data generating process.<n>Existing approaches for adaptive conformal inference are based on optimizing the pinball loss using variants of online gradient descent.<n>We propose a different approach for adaptive conformal inference that leverages parameter-free online convex optimization techniques.
  arXiv  Detail & Related papers  (2024-12-26T18:42:08Z)
• Generalization Bounds of Surrogate Policies for Combinatorial Optimization Problems [53.03951222945921]
  We analyze smoothed (perturbed) policies, adding controlled random perturbations to the direction used by the linear oracle.<n>Our main contribution is a generalization bound that decomposes the excess risk into perturbation bias, statistical estimation error, and optimization error.<n>We illustrate the scope of the results on applications such as vehicle scheduling, highlighting how smoothing enables both tractable training and controlled generalization.
  arXiv  Detail & Related papers  (2024-07-24T12:00:30Z)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [92.90857135952231]
  Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities. We study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression.
  arXiv  Detail & Related papers  (2023-10-12T15:01:43Z)
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection [88.23337313766353]
  This work first provides a comprehensive statistical theory for transformers to perform ICL. We show that transformers can implement a broad class of standard machine learning algorithms in context. A emphsingle transformer can adaptively select different base ICL algorithms.
  arXiv  Detail & Related papers  (2023-06-07T17:59:31Z)
• The Power of Learned Locally Linear Models for Nonlinear Policy Optimization [26.45568696453259]
  This paper conducts a rigorous analysis of a simplified variant of this strategy for general nonlinear systems. We analyze an algorithm which iterates between estimating local linear models of nonlinear system dynamics and performing $mathttiLQR$-like policy updates.
  arXiv  Detail & Related papers  (2023-05-16T17:13:00Z)
• Combining Gaussian processes and polynomial chaos expansions for stochastic nonlinear model predictive control [0.0]
  We introduce a new algorithm to explicitly consider time-invariant uncertainties in optimal control problems. The main novelty in this paper is to use this combination in an efficient fashion to obtain mean and variance estimates of nonlinear transformations. It is shown how to formulate both chance-constraints and a probabilistic objective for the optimal control problem.
  arXiv  Detail & Related papers  (2021-03-09T14:25:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.