Bayesian Wind Tunnels for Model Selection

• URL: http://arxiv.org/abs/2607.19379v1
• Date: Wed, 01 Jul 2026 04:05:08 GMT
• Title: Bayesian Wind Tunnels for Model Selection
• Abstract summary: We show that transformers can perform exact Bayesian filtering within a fixed hypothesis class.<n>We then identify a sharp perceptual access condition: when the discnative statistic requires arithmetic, model selection succeeds with integer tokens.<n>A stationarity control confirms the operative factor: stable semantics, not integer identity, enable circuit compilation.
• Score: 0.4779196219827507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior work has shown that transformers can perform exact Bayesian filtering within a fixed hypothesis class. Can they also perform Bayesian model selection -- identifying the correct hypothesis class from data? We introduce model-selection Bayesian wind tunnels: controlled environments where ground-truth posteriors over hypothesis classes are available in closed form. Using fixed-point-free involutions -- whose defining property f(f(x))=x is purely relational -- a 2.8M-parameter transformer achieves 0.01-bit entropy agreement with the Bayesian optimum (3 seeds), with both integer tokens and opaque symbols whose meanings change every episode. This extends to non-nested comparisons: involutions vs. 3-cycles (where neither class is a subset of the other) achieve class-posterior MAE under 0.001, demonstrating genuine model selection beyond simplicity/subset bias. We then identify a sharp perceptual access condition: when the discriminative statistic requires arithmetic -- modular addition (rotations) or multiplication (f(x)=cx mod p) -- model selection succeeds with integer tokens but fails completely with opaque symbols, and this boundary persists under 112x scaling (2.8M to 316M parameters). A stationarity control confirms the operative factor: opaque tokens with a fixed relabeling succeed (0.009-bit MAE), showing that stable semantics, not integer identity, enable circuit compilation. Header subtask diagnostics localize the failure to the composition of header inversion with arithmetic rather than header parsing itself. Probing frontier LLMs on the same tasks shows qualitative Bayesian behavior but a large calibration gap (~55x), measured through lossy probes and therefore directional rather than exact.

Related papers

• An Omitted Mode Is a Rare Rule: The Sampling-Verification Danger Law in Continuous Code World Models [0.0]
  GPT-5.x repairs an omitted 1D clamp in 105 of 111 mode-containing synthesis draws.<n>On 2D regions no artifact recovers the rule (0/156); eight targeted interventions leave the failure in place, and positive controls locate it.<n>We prove a class of entry rules exactly consistent with every sample yet harmless at play, so identifiability is a measurable property of the instrument.
  arXiv  Detail & Related papers  (2026-08-18T16:09:51Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning [0.5854803320592717]
  We prove optimality in a linear-Gaussian model (Thm. A), necessity of range coverage for any quadratic penalty that zeros deployment drift (Thm. G)<n>Thirteen blocks (ML through Qwen2.5-7B) test matched vs isotropic vs wrong-direction penalties on geometry and deployment drift.<n>We claim a falsifiable design recipe: estimate Sigma_task, match Sigma', run the controls, report task and geometry separately.
  arXiv  Detail & Related papers  (2026-05-21T17:53:28Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Lost in State Space: Probing Frozen Mamba Representations [0.5156484100374058]
  Mamba's recurrent state h_t is a compressed summary of every token seen so far.<n>If we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free.<n>We compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone.
  arXiv  Detail & Related papers  (2026-04-30T21:35:08Z)
• Fixed-Horizon Self-Normalized Inference for Adaptive Experiments via Martingale AIPW/DML with Logged Propensities [0.0]
  Under adaptive assignment, propensities can keep changing, so the predictable quadratic variation of AIPW/DML increments may remain random.<n>We show that the Studentized statistic, with variance estimated by realized quadratic variation, is conditionally miscalibrated even without variance stabilization.
  arXiv  Detail & Related papers  (2026-02-17T13:12:31Z)
• Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost [53.746169882193456]
  Recent works have proposed various sparse attention modules to overcome the quadratic cost of self-attention. We propose a model that resolves both problems by endowing each attention head with a mixed-membership Block Model. Our model outperforms previous efficient variants as well as the original Transformer with full attention.
  arXiv  Detail & Related papers  (2022-10-27T15:30:52Z)
• SimT: Handling Open-set Noise for Domain Adaptive Semantic Segmentation [58.61946589036262]
  This paper studies a practical domain adaptive (DA) semantic segmentation problem where only pseudo-labeled target data is accessible through a black-box model. Due to the domain gap and label shift between two domains, pseudo-labeled target data contains mixed closed-set and open-set label noises. We propose a simplex noise transition matrix (SimT) to model the mixed noise distributions in DA semantic segmentation and formulate the problem as estimation of SimT.
  arXiv  Detail & Related papers  (2022-03-29T02:48:08Z)
• Bayes in Wonderland! Predictive supervised classification inference hits unpredictability [1.8814209805277506]
  We show the convergence of the sBpc and mBpc under de Finetti type of exchangeability. We also provide a parameter estimation of the generative model giving rise to the partition exchangeable sequence.
  arXiv  Detail & Related papers  (2021-12-03T12:34:52Z)
• Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation [63.77513071533095]
  Most existing sequential methods assume users are deterministic. Item-item transitions might fluctuate significantly in several item aspects and exhibit randomness of user interests. We propose a Distribution-based Transformer Sequential Recommendation (DT4SR) which injects uncertainties into sequential modeling.
  arXiv  Detail & Related papers  (2021-06-11T04:35:21Z)
• Coping with Label Shift via Distributionally Robust Optimisation [72.80971421083937]
  We propose a model that minimises an objective based on distributionally robust optimisation (DRO) We then design and analyse a gradient descent-proximal mirror ascent algorithm tailored for large-scale problems to optimise the proposed objective.
  arXiv  Detail & Related papers  (2020-10-23T08:33:04Z)
• Tractable Inference in Credal Sentential Decision Diagrams [116.6516175350871]
  Probabilistic sentential decision diagrams are logic circuits where the inputs of disjunctive gates are annotated by probability values. We develop the credal sentential decision diagrams, a generalisation of their probabilistic counterpart that allows for replacing the local probabilities with credal sets of mass functions. For a first empirical validation, we consider a simple application based on noisy seven-segment display images.
  arXiv  Detail & Related papers  (2020-08-19T16:04:34Z)
• Learning Posterior and Prior for Uncertainty Modeling in Person Re-Identification [11.651410633259543]
  We learn the sample posterior and the class prior distribution in the latent space, so that not only representative features but also the uncertainty can be built by the model. Experiments have been carried out on Market1501, DukeMTMC, MARS and noisy dataset as well.
  arXiv  Detail & Related papers  (2020-07-17T07:20:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.