Mixture of In-Context Prompters for Tabular PFNs

• URL: http://arxiv.org/abs/2405.16156v1
• Date: Sat, 25 May 2024 09:47:59 GMT
• Title: Mixture of In-Context Prompters for Tabular PFNs
• Authors: Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, Wei Wang,
• Abstract summary: MIXTUREPFN is the Condorcet winner across 36 diverse datasets against 19 strong deep learning and tree-based baselines. It achieves the highest mean rank among Top-10 aforementioned algorithms with statistical significance.
• Score: 33.76194735049027
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent benchmarks found In-Context Learning (ICL) outperforms both deep learning and tree-based algorithms on small tabular datasets. However, on larger datasets, ICL for tabular learning cannot run without severely compromising performance, due to its quadratic space and time complexity w.r.t. dataset size. We propose MIXTUREPFN, which both extends nearest-neighbor sampling to the state-of-the-art ICL for tabular learning model and uses bootstrapping to finetune said model on the inference-time dataset. MIXTUREPFN is the Condorcet winner across 36 diverse tabular datasets against 19 strong deep learning and tree-based baselines, achieving the highest mean rank among Top-10 aforementioned algorithms with statistical significance.

Related papers

• Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later [59.88557193062348]
  We revisit the classic Neighborhood Component Analysis (NCA), designed to learn a linear projection that captures semantic similarities between instances. We find that minor modifications, such as adjustments to the learning objectives and the integration of deep learning architectures, significantly enhance NCA's performance. We also introduce a neighbor sampling strategy that improves both the efficiency and predictive accuracy of our proposed ModernNCA.
  arXiv  Detail & Related papers  (2024-07-03T16:38:57Z)
• A Closer Look at Deep Learning on Tabular Data [52.50778536274327]
  Tabular data is prevalent across various domains in machine learning. Deep Neural Network (DNN)-based methods have shown promising performance comparable to tree-based ones.
  arXiv  Detail & Related papers  (2024-07-01T04:24:07Z)
• Retrieval & Fine-Tuning for In-Context Tabular Models [16.668695961462827]
  Recent advancements using transformer-based in-context learning have shown promise on smaller and less complex datasets, but have struggled to scale to larger and more complex ones. We propose a combination of retrieval and fine-tuning: we can adapt the transformer to a local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours in context. We show a significant boost in performance compared to the base in-context model.
  arXiv  Detail & Related papers  (2024-06-07T18:43:33Z)
• In-Context Data Distillation with TabPFN [11.553950697974825]
  In-context data distillation (ICD) is a novel methodology that effectively eliminates these constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to handle significantly larger datasets with a fixed memory budget, improving TabPFN's quadratic memory complexity but at the cost of a linear number of tuning steps.
  arXiv  Detail & Related papers  (2024-02-10T15:23:45Z)
• Minimally Supervised Learning using Topological Projections in Self-Organizing Maps [55.31182147885694]
  We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs) Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU) Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
  arXiv  Detail & Related papers  (2024-01-12T22:51:48Z)
• Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking [0.0]
  This article extensively evaluates state-of-the-art attention and contrastive learning methods on a wide selection of 28 data sets. We find that a hybrid attention-contrastive learning strategy mostly wins on hard-to-classify data sets. Traditional methods are frequently superior on easy-to-classify data sets with presumably simpler decision boundaries.
  arXiv  Detail & Related papers  (2024-01-08T22:36:05Z)
• DataComp: In search of the next generation of multimodal datasets [179.79323076587255]
  DataComp is a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Our benchmark consists of multiple compute scales spanning four orders of magnitude. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet.
  arXiv  Detail & Related papers  (2023-04-27T11:37:18Z)
• Is margin all you need? An extensive empirical study of active learning on tabular data [66.18464006872345]
  We analyze the performance of a variety of active learning algorithms on 69 real-world datasets from the OpenML-CC18 benchmark. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art.
  arXiv  Detail & Related papers  (2022-10-07T21:18:24Z)
• Why do tree-based models still outperform deep learning on tabular data? [0.0]
  We show that tree-based models remain state-of-the-art on medium-sized data. We conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs)
  arXiv  Detail & Related papers  (2022-07-18T08:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.