CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching

• URL: http://arxiv.org/abs/2606.11473v1
• Date: Tue, 09 Jun 2026 22:07:04 GMT
• Title: CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching
• Abstract summary: Prior-fitted networks (PFNs) are a promising class of foundation models that perform in-context learning.<n>We propose CRUMB (Clustered Retrieval Using Minimised-MMD ), a three-stage inference wrapper that clusters the test queries.<n>CRUMB selects a small, distributionally matched training subset for each cluster by greedily minimising the maximum mean discrepancy.
• Score: 1.4211459456330362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior-fitted networks (PFNs) are a promising class of tabular foundation models that perform in-context learning, whereby the entire labelled training set is supplied as context, and predictions for test queries are produced in a single forward pass. However, the quadratically scaling self-attention mechanism in many PFN architectures makes inference prohibitive for very large training datasets. We propose CRUMB (Clustered Retrieval Using Minimised-MMD Batching), a three-stage inference wrapper that (i) clusters the test queries, (ii) selects a small, distributionally matched training subset for each cluster by greedily minimising the maximum mean discrepancy (MMD), and (iii) runs exact PFN inference on each reduced-context batch. CRUMB is architecture-agnostic and requires no retraining. On the 51-dataset TabArena benchmark, evaluated across three PFN architectures (TabPFNv2, TabICLv1, TabICLv2), we show that CRUMB outperforms similar state-of-the-art context selection strategies. We also show that CRUMB is resilient to covariate drift, as the MMD-minimisation step naturally helps align the training context distribution to match the current test batch distributions.

Related papers

• Residual-Guided Randomized Neural Networks [7.698602367824994]
  Randomized neural networks enable fast and analytically tractable training by fixing the input to hidden layer parameters at random and learning the output weights in closed form.<n>We propose a simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion.<n>We show that the proposed residual-guided models consistently outperform their baseline counterparts in terms of accuracy, stability, and overall ranking performance.
  arXiv  Detail & Related papers  (2026-08-28T12:30:08Z)
• sGPO: Trading Inference FLOPs for Training Efficiency in RLVR [15.69155608127456]
  Standard Reinforcement Learning with Verifiable Rewards (RLVR) training allocates a fixed rollout budget to every query.<n>This leads to two symmetric failure modes: easy queries produce near-zero advantage because the policy already solves them, while unsolvable queries produce no signal because the policy never solves them.<n>We introduce sorted Group Policy Optimization (sGPO), a compute-efficient strategy that trades a small budget of inference FLOPs for a large reduction in wasted training FLOPs.
  arXiv  Detail & Related papers  (2026-06-07T21:47:31Z)
• Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance [51.532841645285835]
  We study machine unlearning in large generative models by framing the task as density ratio estimation to a target distribution.<n>We show it can fail to faithfully unlearn with finite samples when the forget set represents a sharp, concentrated data distribution.<n>We introduce Temper-Then-Tilt Unlearning (T3-Unlearning), which freezes the base model and applies a two-step inference procedure.
  arXiv  Detail & Related papers  (2026-02-10T19:08:40Z)
• OneFlowSBI: One Model, Many Queries for Simulation-Based Inference [2.614875980890442]
  textitOneFlow SBI is a unified framework for simulation-based inference.<n>It learns a single flow-matching generative model over the joint distribution of parameters and observations.<n>It supports multiple inference tasks, including posterior sampling, likelihood estimation, and arbitrary conditional distributions.
  arXiv  Detail & Related papers  (2026-01-30T13:14:44Z)
• Learning Discrete Bayesian Networks with Hierarchical Dirichlet Shrinkage [52.914168158222765]
  We detail a comprehensive Bayesian framework for learning DBNs.<n>We give a novel Markov chain Monte Carlo (MCMC) algorithm utilizing parallel Langevin proposals to generate exact posterior samples.<n>We apply our methodology to uncover prognostic network structure from primary breast cancer samples.
  arXiv  Detail & Related papers  (2025-09-16T17:24:35Z)
• Towards Fair In-Context Learning with Tabular Foundation Models [6.4989916051093815]
  We present the first investigation of fairness in Transformer-based in-context learning (ICL)<n>We evaluate three recently proposed foundation models -- TabPFNv2, TabICL, and TabDPT -- on benchmark datasets.<n>Our experiments show that the uncertainty-based strategy consistently improves group fairness metrics with minimal impact on predictive accuracy.
  arXiv  Detail & Related papers  (2025-05-14T15:53:14Z)
• Training-Free Unsupervised Prompt for Vision-Language Models [27.13778811871694]
  We propose Training-Free Unsupervised Prompts (TFUP) to preserve inherent representation capabilities and enhance them with a residual connection to similarity-based prediction probabilities. TFUP achieves surprising performance, even surpassing the training-base method on multiple classification datasets. Our TFUP-T achieves new state-of-the-art classification performance compared to unsupervised and few-shot adaptation approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2024-04-25T05:07:50Z)
• Rethinking Few-shot 3D Point Cloud Semantic Segmentation [62.80639841429669]
  This paper revisits few-shot 3D point cloud semantic segmentation (FS-PCS) We focus on two significant issues in the state-of-the-art: foreground leakage and sparse point distribution. To address these issues, we introduce a standardized FS-PCS setting, upon which a new benchmark is built.
  arXiv  Detail & Related papers  (2024-03-01T15:14:47Z)
• Boosting Low-Data Instance Segmentation by Unsupervised Pre-training with Saliency Prompt [103.58323875748427]
  This work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models. Experimental results show that our method significantly boosts several QEIS models on three datasets.
  arXiv  Detail & Related papers  (2023-02-02T15:49:03Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.