Towards Evaluating Data Priors for Tabular Foundation Models
Abstract Overview
This paper studies data-generating priors as independent components of tabular foundation models rather than evaluating them only together with their original architectures and training procedures. The authors implement a unified interface covering publicly available priors from TabPFNv1, TabICL, TICL, TabForestPFN, and priors constructed from curated real datasets, then generate tasks from each prior under a shared setup. They train the same nanoTabPFN architecture with a fixed protocol on 250,000 tasks per prior and evaluate the resulting models on shared TabArena classification benchmarks. The analysis compares priors both through generated-task statistics and downstream ROC-AUC behavior, with additional examination of how data similarity relates to performance similarity.
Novelty
The main novelty is a controlled evaluation pipeline that isolates the effect of the data prior from model architecture and optimization in tabular foundation models. The work is also distinctive in unifying several public synthetic priors with real-data priors under one task-generation and evaluation interface.
Results
The experiments show that prior choice changes downstream behavior even when architecture and training budget are fixed. TICL attains the highest raw average ROC-AUC and the most dataset wins, while tabicl_mix_scm shows the best average rank and strongest normalized within-dataset competitiveness. The study also finds that data-level similarity is only a partial predictor of downstream performance similarity, and that the real-data prior with random target selection performs much better than using only default targets.
Key Points
- A unified benchmark pipeline is introduced to compare tabular data priors under the same model architecture, training protocol, and downstream evaluation tasks.
- Different priors produce different performance profiles: no single prior is best on every criterion, and dataset sensitivity to prior choice varies.
- Statistical similarity between generated tasks and real data does not fully determine downstream success; task diversity and generation choices also matter.