TabFlex: Scaling Tabular Learning to Millions with Linear Attention

• URL: http://arxiv.org/abs/2506.05584v1
• Date: Thu, 05 Jun 2025 20:59:33 GMT
• Title: TabFlex: Scaling Tabular Learning to Millions with Linear Attention
• Authors: Yuchen Zeng, Tuan Dinh, Wonjun Kang, Andreas C Mueller,
• Abstract summary: Recent advancements, like TabPFN, excel in small-scale datasets but struggle to scale for large and complex datasets.<n>Our work enhances the efficiency and scalability of TabPFN for larger datasets by incorporating linear attention mechanisms.<n>Our model, TabFlex, efficiently handles tabular datasets with thousands of features and hundreds of classes, scaling seamlessly to millions of samples.
• Score: 8.018661387739574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Leveraging the in-context learning (ICL) capability of Large Language Models (LLMs) for tabular classification has gained significant attention for its training-free adaptability across diverse datasets. Recent advancements, like TabPFN, excel in small-scale tabular datasets but struggle to scale for large and complex datasets. Our work enhances the efficiency and scalability of TabPFN for larger datasets by incorporating linear attention mechanisms as a scalable alternative to complexity-quadratic self-attention. Our model, TabFlex, efficiently handles tabular datasets with thousands of features and hundreds of classes, scaling seamlessly to millions of samples. For instance, TabFlex processes the poker-hand dataset with over a million samples in just 5 seconds. Our extensive evaluations demonstrate that TabFlex can achieve over a 2x speedup compared to TabPFN and a 1.5x speedup over XGBoost, outperforming 25 tested baselines in terms of efficiency across a diverse range of datasets. Furthermore, TabFlex remains highly effective on large-scale datasets, delivering strong performance with significantly reduced computational costs, especially when combined with data-efficient techniques such as dimensionality reduction and data sampling.

Related papers

• Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners [82.72552644267724]
  BoostPFN can outperform standard PFNs with the same size of training samples in large datasets.<n>High performance is maintained for up to 50x of the pre-training size of PFNs.
  arXiv  Detail & Related papers  (2025-03-03T07:31:40Z)
• TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization [2.1067477213933503]
  TabGLM (Tabular Graph Language Model) is a novel multi-modal architecture designed to model both structural and semantic information from a table.<n>It transforms each row of a table into a fully connected graph and serialized text, which are encoded using a graph neural network (GNN) and a text encoder, respectively.<n> Evaluations across 25 benchmark datasets demonstrate substantial performance gains.
  arXiv  Detail & Related papers  (2025-02-26T05:32:45Z)
• A Closer Look at TabPFN v2: Strength, Limitation, and Extension [51.08999772842298]
  Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented in-context learning accuracy across multiple datasets.<n>In this paper, we evaluate TabPFN v2 on over 300 datasets, confirming its exceptional generalization capabilities on small- to medium-scale tasks.
  arXiv  Detail & Related papers  (2025-02-24T17:38:42Z)
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data [15.08819125687632]
  We introduce TabICL, a tabular foundation model for classification, pretrained on synthetic datasets with up to 60K samples.<n>Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times)<n>On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the potential of ICL for large data.
  arXiv  Detail & Related papers  (2025-02-08T13:25:04Z)
• Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes [135.68092471784516]
  We propose a simple and lightweight approach for fusing large language models and gradient-boosted decision trees.<n>We name our fusion methods LLM-Boost and PFN-Boost, respectively.<n>We demonstrate state-of-the-art performance against numerous baselines and ensembling algorithms.
  arXiv  Detail & Related papers  (2025-02-04T19:30:41Z)
• Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding [2.379669478864599]
  Current cross-dataset pruning techniques for fine-tuning often rely on computationally expensive sample ranking processes.<n>We propose Swift Cross-Dataset Pruning (SCDP), which uses TF-IDF embeddings with geometric median to rapidly evaluate sample importance.<n> Experimental results on six diverse datasets demonstrate the effectiveness of our method, spanning various tasks and scales.
  arXiv  Detail & Related papers  (2025-01-05T03:52:04Z)
• Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models [64.28420991770382]
  Data-Juicer 2.0 is a data processing system backed by data processing operators spanning text, image, video, and audio modalities.<n>It supports more critical tasks including data analysis, annotation, and foundation model post-training.<n>It has been widely adopted in diverse research fields and real-world products such as Alibaba Cloud PAI.
  arXiv  Detail & Related papers  (2024-12-23T08:29:57Z)
• A Survey on Deep Tabular Learning [0.0]
  Tabular data presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for Tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet.
  arXiv  Detail & Related papers  (2024-10-15T20:08:08Z)
• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks [90.00817095558094]
  Prior-data fitted networks (PFNs) make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective.
  arXiv  Detail & Related papers  (2024-02-17T00:02:23Z)
• In-Context Data Distillation with TabPFN [11.553950697974825]
  In-context data distillation (ICD) is a novel methodology that effectively eliminates these constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to handle significantly larger datasets with a fixed memory budget, improving TabPFN's quadratic memory complexity but at the cost of a linear number of tuning steps.
  arXiv  Detail & Related papers  (2024-02-10T15:23:45Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM)<n>The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.<n>Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second [48.87527918630822]
  We present TabPFN, a trained Transformer that can do supervised classification for small datasets in less than a second. TabPFN performs in-context learning (ICL), it learns to make predictions using sequences of labeled examples. We show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 230$times$ speedup.
  arXiv  Detail & Related papers  (2022-07-05T07:17:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.