Speedrunning Tabular Foundation Model Pretraining

• URL: http://arxiv.org/abs/2606.03681v1
• Date: Tue, 02 Jun 2026 14:04:31 GMT
• Title: Speedrunning Tabular Foundation Model Pretraining
• Abstract summary: We introduce a community speedrun for nanoTabPFN.<n> contributors modify a single-file training script and compete to reach a fixed downstream ROC AUC target.<n>The current best record reaches the target in 0.92 minutes, an 81x speedup over the 74.32 minute baseline.
• Score: 80.20375439555964
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimization ideas. Yet the community lacks a simple way to compare and accumulate pretraining speedups. We introduce a community speedrun for nanoTabPFN: contributors modify a single-file training script and compete to reach a fixed downstream ROC AUC target on subsampled TabArena using one NVIDIA L40S GPU. The current best record reaches the target in 0.92 minutes, an 81x speedup over the 74.32 minute baseline while using 22x fewer synthetic datasets. The speedrun format provides a simple protocol for the community to add, verify, and stack pretraining improvements, with the leaderboard open to contributions. Code and records are available at https://github.com/borawhocodess/modded-nanotabpfn.

Related papers

• LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training [28.882595607766916]
  Training large language models on limited hardware is increasingly a scheduling problem.<n>We propose LazyTrain, an optimization layer over a layer-streaming executor.
  arXiv  Detail & Related papers  (2026-08-12T11:04:06Z)
• TabPFN-3: Technical Report [76.55249061301969]
  TabPFN-3 builds on TabPFN to scale state-of-the-art performance to datasets with 1M training rows and substantially reduce training and inference time.<n>Pretrained exclusively on synthetic data from our prior, TabPFN-3 dramatically pushes the frontier of tabular prediction.<n>Our API offering TabPFN-3-Plus (Thinking) exploits this to beat all non-TabPFN models by over 200 Elo on TabArena.
  arXiv  Detail & Related papers  (2026-05-13T18:01:43Z)
• Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning [19.797488880602813]
  We present a framework that computes six sample descriptors for each candidate and constructs optimized 1$times$ training subsets for different goals.<n>Under a fixed one-epoch Qwen3-VL-8B-Instruct training and evaluation recipe, GDO uses far fewer training samples than the Uni-10x baseline.<n>Relative to the fixed 512k-sample Uni-10x baseline, GDO reaches the Uni-10x reference after 35.4k samples on MVBench, 26.6k on VideoMME, 27.3k on MLVU, and 34.7k on LVBench.
  arXiv  Detail & Related papers  (2026-03-12T21:54:50Z)
• Memory-Efficient Acceleration of Block Low-Rank Foundation Models on Resource Constrained GPUs [11.45717904490388]
  Recent advances in transformer-based foundation models have made them the default choice for many tasks.<n>Their rapidly growing size makes fitting a full model on a single GPU increasingly difficult and their computational cost prohibitive.<n>Block low-rank (BLR) compression techniques address this challenge by learning compact representations of weight matrices.
  arXiv  Detail & Related papers  (2025-12-24T00:41:13Z)
• nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN [78.62756717376563]
  nanoTabPFN is a lightweight implementation of the TabPFN v2 architecture and a corresponding training loop.<n>It achieves a performance comparable to traditional machine learning baselines within one minute of pre-training on a single GPU.
  arXiv  Detail & Related papers  (2025-11-05T16:52:51Z)
• RecTable: Fast Modeling Tabular Data with Rectified Flow [0.0]
  We introduce RecTable, which uses the rectified flow modeling, applied in such as text-to-image generation and text-to-video generation.<n>Our experiments demonstrate that RecTable achieves competitive performance compared to the several state-of-the-art diffusion and score-based models.
  arXiv  Detail & Related papers  (2025-03-26T17:12:20Z)
• Speedy MASt3R [68.47052557089631]
  MASt3R redefines image matching as a 3D task by leveraging DUSt3R and introducing a fast reciprocal matching scheme.<n>Fast MASt3R achieves a 54% reduction in inference time (198 ms to 91 ms per image pair) without sacrificing accuracy.<n>This advancement enables real-time 3D understanding, benefiting applications like mixed reality navigation and large-scale 3D scene reconstruction.
  arXiv  Detail & Related papers  (2025-03-13T03:56:22Z)
• Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks [69.54774045493227]
  A drawback of adversarial training is the computational overhead introduced by the generation of adversarial examples. We propose to exploit the interior building blocks of the model to improve efficiency. Compared with previous methods, our method not only reduces the training cost but also achieves better model robustness.
  arXiv  Detail & Related papers  (2023-10-24T01:36:20Z)
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second [48.87527918630822]
  We present TabPFN, a trained Transformer that can do supervised classification for small datasets in less than a second. TabPFN performs in-context learning (ICL), it learns to make predictions using sequences of labeled examples. We show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 230$times$ speedup.
  arXiv  Detail & Related papers  (2022-07-05T07:17:43Z)
• Accelerating DNN Training with Structured Data Gradient Pruning [0.5801044612920815]
  Weight pruning is a technique to make Deep Neural Network (DNN) inference more computationally efficient. Modern accelerators such as the Nvidia A100 GPU support this type of structured sparsity for 2 nonzeros per 4 elements in a reduction. Our approach can achieve a 15-25% reduction in total training time without significant impact to performance.
  arXiv  Detail & Related papers  (2022-02-01T21:41:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.