TabH2O: A Unified Foundation Model for Tabular Prediction

• URL: http://arxiv.org/abs/2605.18383v1
• Date: Mon, 18 May 2026 13:27:54 GMT
• Title: TabH2O: A Unified Foundation Model for Tabular Prediction
• Abstract summary: We present TabH2O, a model that performs classification and regression in a single forward pass via in-context learning.<n>We evaluate TabH2O v1 (29.2M parameters) on the TALENT benchmark, where it achieves an average rank of 2.55 out of 6 evaluated methods.
• Score: 12.882141870947706
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present TabH2O, a foundation model for tabular data that performs classification and regression in a single forward pass via in-context learning. TabH2O builds on the TabICL architecture with several key modifications: (1) unified training, a single model handles both classification and regression via a dual-head architecture, eliminating the need for separate models and reducing total pretraining cost; (2) single-stage pretraining, training stability improvements (bounded scalable softmax, inter-stage normalization, learnable residual scaling, logit soft-capping) eliminate the need for multi-stage curriculum learning, enabling training with full-length sequences from the start; and (3) noise-aware pretraining, synthetic datasets include explicit noise dimensions to teach the model robustness to irrelevant features. We evaluate TabH2O v1 (29.2M parameters) on the TALENT benchmark (300 datasets), where it achieves an average rank of 2.55 out of 6 evaluated methods, outperforming tuned CatBoost (4.07), H2O AutoML (4.18), and LightGBM (5.08), competitive with TabPFN v2.6 (2.74), and behind TabICL v2 (2.12), while placing in the top-3 on 81% of the testing datasets across classification and regression tasks.

Related papers

• EXAONE Tabular 1.0 : Technical Report [12.091751977253866]
  EXAONE Tabular is a compact model family for classification and regression.<n>It produces predictions without dataset-specific gradient updates.<n>It combines strong predictive performance with high efficiency.
  arXiv  Detail & Related papers  (2026-08-26T13:17:20Z)
• Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN [0.0]
  Localized TabICLv2 reduces the inference cost of TabICLv2 by retrieving only the k nearest training neighbours for each test point.<n>On TabArena classification tasks, the fine-tuned localized model retains 98.64% of Full TabICLv2 accuracy.
  arXiv  Detail & Related papers  (2026-08-17T11:30:56Z)
• Index SLM Technical Report [11.17580009715777]
  We present Index-1.9B, a series of open small language models developed at Bilibili.<n>The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for role-playing customization.
  arXiv  Detail & Related papers  (2026-07-10T18:19:47Z)
• StereoFactory: A Unified Merging Framework for Robust Stereo Matching [61.973843344605655]
  Stereo matching has advanced through foundation models trained on large-scale datasets, yet this paradigm suffers from a scalability bottleneck.<n>Model merging offers a scalable post-hoc alternative by integrating knowledge from specialized models after source checkpoints are available.<n>We propose StereoFactory, a coarse-to-fine evolutionary framework for adaptive model merging.
  arXiv  Detail & Related papers  (2026-06-16T03:36:59Z)
• Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization [54.749573452394664]
  We find that accuracy collapses as instance data grows, even when the formulation itself is simple.<n>We address this via a simple inference-time approach, BIND, which externalizes numeric data to structured files so the model binds data rather than from prompt prompt.<n>We validate our hypothesis by finetuning a model exclusively on binding and show that it outperforms end-to-end SFT and RL across three structurally distinct optimization categories.
  arXiv  Detail & Related papers  (2026-05-20T21:25:41Z)
• TabPFN-3: Technical Report [76.55249061301969]
  TabPFN-3 builds on TabPFN to scale state-of-the-art performance to datasets with 1M training rows and substantially reduce training and inference time.<n>Pretrained exclusively on synthetic data from our prior, TabPFN-3 dramatically pushes the frontier of tabular prediction.<n>Our API offering TabPFN-3-Plus (Thinking) exploits this to beat all non-TabPFN models by over 200 Elo on TabArena.
  arXiv  Detail & Related papers  (2026-05-13T18:01:43Z)
• Prior-Aligned Data Cleaning for Tabular Foundation Models [0.261072980439312]
  Tabular Foundation Models (TFMs) achieve state-of-the-art zero-shot accuracy on small datasets by meta-learning over synthetic data-generating processes.<n>In-context learning mechanism assumes approximately clean inputs, missing values, outliers, and duplicates in the real-world data create a prior mismatch that degrades both accuracy and confidence calibration simultaneously.
  arXiv  Detail & Related papers  (2026-04-28T02:56:17Z)
• TabICLv2: A better, faster, scalable, and open tabular foundation model [18.594859017648346]
  We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars.<n>Tabiclv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5.
  arXiv  Detail & Related papers  (2026-02-11T18:51:02Z)
• F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data [19.088644745246373]
  We introduce F2LLM, a suite of state-of-the-art embedding models in three sizes: 0.6B, 1.7B, and 4B.<n>F2LLM is directly finetuned from foundation models on 6 million query-document-negatives curated from open-source, non-synthetic datasets.<n>On the MTEB English leaderboard, F2LLM-4B ranks 2nd among models with approximately 4B parameters and 7th overall, while F2LLM-1.7B ranks 1st among models in the 1B-2B size range.
  arXiv  Detail & Related papers  (2025-10-02T17:58:49Z)
• Approximating Language Model Training Data from Weights [70.08614275061689]
  We formalize the problem of data approximation from model weights and propose several baselines and metrics.<n>We develop a gradient-based approach that selects the highest-matching data from a large public text corpus.<n>Even when none of the true training data is known, our method is able to locate a small subset of public Web documents.
  arXiv  Detail & Related papers  (2025-06-18T15:26:43Z)
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data [15.08819125687632]
  We introduce TabICL, a tabular foundation model for classification, pretrained on synthetic datasets with up to 60K samples.<n>Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times)<n>On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the potential of ICL for large data.
  arXiv  Detail & Related papers  (2025-02-08T13:25:04Z)
• Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training [68.94373533768501]
  We model knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training.<n>We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy.
  arXiv  Detail & Related papers  (2025-02-06T13:23:53Z)
• Tangent Model Composition for Ensembling and Continual Fine-tuning [69.92177580782929]
  Tangent Model Composition (TMC) is a method to combine component models independently fine-tuned around a pre-trained point. TMC improves accuracy by 4.2% compared to ensembling non-linearly fine-tuned models.
  arXiv  Detail & Related papers  (2023-07-16T17:45:33Z)
• MSeg: A Composite Dataset for Multi-domain Semantic Segmentation [100.17755160696939]
  We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains. We reconcile the generalization and bring the pixel-level annotations into alignment by relabeling more than 220,000 object masks in more than 80,000 images. A model trained on MSeg ranks first on the WildDash-v1 leaderboard for robust semantic segmentation, with no exposure to WildDash data during training.
  arXiv  Detail & Related papers  (2021-12-27T16:16:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.