TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

• URL: http://arxiv.org/abs/2606.02384v1
• Date: Mon, 01 Jun 2026 15:33:43 GMT
• Title: TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
• Abstract summary: We introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that target three specific structural data patterns.<n>We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance.<n>Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models.
• Score: 32.521087490144964
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.

Related papers

• TS2TabPFN: Time Series Classification and Extrinsic Regression through Feature Extraction and a Tabular Foundation Model [0.7913770737379631]
  This paper proposes TS2TabPFN, a framework that integrates explicit feature extraction with TabPFN 2.5.<n>We show that TS2TabPFN significantly outperforms state-of-the-art models in TSER tasks with statistical significance.<n>Results suggest that combining foundation models with structured features overcomes single-paradigm limitations.
  arXiv  Detail & Related papers  (2026-08-04T19:32:18Z)
• Beyond IID: How General Are Tabular Foundation Models, Really? [76.03669797389378]
  Research communities are increasingly evaluating foundation models on diverse datasets and tasks.<n>Standard benchmarks are mostly defined for tasks where tabular foundation models already excel.<n>We introduce BeyondArena, the first unified holistic benchmark for tabular data.
  arXiv  Detail & Related papers  (2026-06-29T14:55:52Z)
• TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention [47.480923979431886]
  Tabular foundation models perform prediction via in-context learning, inferring test labels directly from labeled training examples.<n>Recent models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment.<n>We show that a lightweight row-wise attention-only backbone can remain highly competitive with two simple enhancements.
  arXiv  Detail & Related papers  (2026-06-05T14:55:30Z)
• MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image [52.47434184153733]
  We introduce MulTaBench, a benchmark of 40 datasets split equally between image-tabular and text-tabular tasks.<n>We show that the gains from target-aware representation tuning generalize across both text and image modalities.
  arXiv  Detail & Related papers  (2026-05-11T14:12:05Z)
• Tabular Foundation Model for Generative Modelling [34.3599321018728]
  Generative modelling requires robust, holistic representation learning for a given data modality, rather than optimisation for a supervised prediction target alone.<n>Existing tabular foundation generators have not yet consistently matched strong dataset-specific generators in synthetic data quality.<n>We introduce TabFORGE, built on pretrained textbfTabular textbfFOundational textbfRepresentations for textbfGEneration.
  arXiv  Detail & Related papers  (2026-05-10T08:52:28Z)
• Revisiting the Generic Transformer: Deconstructing a Strong Baseline for Time Series Foundation Models [18.841505010078112]
  We investigate the potential of a standard patch Transformer, demonstrating that it achieves state-of-the-art zero-shot forecasting performance.<n>We conduct a comprehensive ablation study that covers model scaling, data composition, and training techniques to isolate the essential ingredients for high performance.
  arXiv  Detail & Related papers  (2026-02-06T18:01:44Z)
• Comparing Task-Agnostic Embedding Models for Tabular Data [1.6479389738270018]
  This work specifically focuses on representation learning, i.e., on transferable, task-agnostic embeddings.<n>Tableizer features achieve comparable or superior performance while being up to three orders of magnitude faster than recent foundation models.
  arXiv  Detail & Related papers  (2025-11-18T09:10:40Z)
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models [85.64873567417396]
  We introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world data.<n>Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks.
  arXiv  Detail & Related papers  (2025-10-24T07:15:06Z)
• TabArena: A Living Benchmark for Machine Learning on Tabular Data [45.52876263971067]
  We introduce TabArena, the first continuously maintained living benchmarking system.<n>We manually curate a representative collection of datasets and well-implemented models.<n>We show that deep learning methods have caught up under larger time budgets with ensembling.<n>We observe that some deep learning models are overrepresented in cross-model ensembles due to validation set overfitting.
  arXiv  Detail & Related papers  (2025-06-20T07:14:48Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• AutoBERT-Zero: Evolving BERT Backbone from Scratch [94.89102524181986]
  We propose an Operation-Priority Neural Architecture Search (OP-NAS) algorithm to automatically search for promising hybrid backbone architectures. We optimize both the search algorithm and evaluation of candidate models to boost the efficiency of our proposed OP-NAS. Experiments show that the searched architecture (named AutoBERT-Zero) significantly outperforms BERT and its variants of different model capacities in various downstream tasks.
  arXiv  Detail & Related papers  (2021-07-15T16:46:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.