Prior-Aligned Data Cleaning for Tabular Foundation Models

• URL: http://arxiv.org/abs/2604.25154v1
• Date: Tue, 28 Apr 2026 02:56:17 GMT
• Title: Prior-Aligned Data Cleaning for Tabular Foundation Models
• Abstract summary: Tabular Foundation Models (TFMs) achieve state-of-the-art zero-shot accuracy on small datasets by meta-learning over synthetic data-generating processes.<n>In-context learning mechanism assumes approximately clean inputs, missing values, outliers, and duplicates in the real-world data create a prior mismatch that degrades both accuracy and confidence calibration simultaneously.
• Score: 0.261072980439312
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular Foundation Models (TFMs) achieve state-of-the-art zero-shot accuracy on small tabular datasets by meta-learning over synthetic data-generating processes -- making them highly attractive for practitioners who cannot afford large annotated corpora. However, their in-context learning mechanism assumes approximately clean inputs: missing values, outliers, and duplicates in the real-world data create a prior mismatch that degrades both accuracy and confidence calibration simultaneously. Correcting this mismatch requires sequential decisions over cleaning operators whose interactions no static preprocessing rule can anticipate -a natural fit for reinforcement learning~(RL). We introduce L2C2, the first deep RL framework framing tabular data cleaning as prior alignment: a learned policy sequences operators to minimize the distributional gap between dirty input and the TFM's synthetic prior. Six experiments on ten OpenML benchmark datasets establish: 1) three of seven reward designs collapse to degenerate trivial cleaning strategies -- principled reward engineering is scientifically non-trivial; 2) the novel TFMAwareReward reward we propose selects structurally distinct pipelines on 4/10 datasets and achieves higher TabPFN accuracy on those diverging cases (mean 0.851 vs. 0.843; Wilcoxon p=0.063, n=4) while never underperforming; 3) parameterized cleaning actions improve best-found pipeline reward on 9/10 datasets (Wilcoxon p=0.004); and 4) a policy pre-trained on one single source dataset exceeds scratch training at the 2,000-step fine-tuning checkpoint on all three held-out datasets (up to +28.8% after full fine-tuning) demonstrating cross-dataset transfer of prior-alignment knowledge. These findings establish that prior alignment is a principled data preparation strategy for TFM deployment on real-world tabular data.

Related papers

• TabH2O: A Unified Foundation Model for Tabular Prediction [12.882141870947706]
  We present TabH2O, a model that performs classification and regression in a single forward pass via in-context learning.<n>We evaluate TabH2O v1 (29.2M parameters) on the TALENT benchmark, where it achieves an average rank of 2.55 out of 6 evaluated methods.
  arXiv  Detail & Related papers  (2026-05-18T13:27:54Z)
• Concordia: Self-Improving Synthetic Tables for Federated LLMs [80.03837595689608]
  Federated learning enables training large language models (LLMs) without sharing raw data.<n>We propose Concordia, a framework that aligns synthetic data generation with federated validation utility.
  arXiv  Detail & Related papers  (2026-05-11T01:17:58Z)
• KumoRFM-2: Scaling Foundation Models for Relational Learning [48.88283881350858]
  KumoRFM-2 is the next iteration of a pre-trained foundation model for relational data.<n>It supports in-context learning as well as fine-tuning and is applicable to a wide range of predictive tasks.
  arXiv  Detail & Related papers  (2026-04-14T11:24:11Z)
• Evaluating SAP RPT-1 for Enterprise Business Process Prediction: In-Context Learning vs. Traditional Machine Learning on Structured SAP Data [0.0]
  This paper presents the first independent evaluation of SAP's Retrieval Pretrained Transformer (RPT-1) from a practitioner perspective.<n>RPT-1 is a compact 64.6 MB model pretrained on 1.34 TB of structured data across 3.1 million tables.<n>We benchmark it against tuned gradient-boosted decision trees (XGBoost, LightGBM, CatBoost) on three SAP business scenarios.
  arXiv  Detail & Related papers  (2026-02-22T15:29:52Z)
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models [85.64873567417396]
  We introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world data.<n>Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks.
  arXiv  Detail & Related papers  (2025-10-24T07:15:06Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning [71.30276778807068]
  We propose a unified framework that strategically coordinates sample pruning and token pruning.<n>Q-Tuning achieves a +38% average improvement over the full-data SFT baseline using only 12.5% of the original training data.
  arXiv  Detail & Related papers  (2025-09-28T13:27:38Z)
• Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data [39.40116554523575]
  We present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network. It learns to approximate Bayesian inference on synthetic datasets drawn from a prior. It improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration.
  arXiv  Detail & Related papers  (2024-11-15T23:49:23Z)
• FedCSD: A Federated Learning Based Approach for Code-Smell Detection [7.026278088747708]
  This paper proposes a Federated Learning Code Smell Detection approach that allows organizations to collaboratively train ML models. Three experiments have leveraged three manually validated datasets aimed at detecting and examining different code smell scenarios. An accuracy of 98.34% was achieved by the global model that has been trained using 10 companies for 100 training rounds.
  arXiv  Detail & Related papers  (2023-05-31T09:51:45Z)
• DataComp: In search of the next generation of multimodal datasets [179.79323076587255]
  DataComp is a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Our benchmark consists of multiple compute scales spanning four orders of magnitude. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet.
  arXiv  Detail & Related papers  (2023-04-27T11:37:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.