In-context learning of evolving data streams with tabular foundational models

• URL: http://arxiv.org/abs/2502.16840v1
• Date: Mon, 24 Feb 2025 04:52:35 GMT
• Title: In-context learning of evolving data streams with tabular foundational models
• Authors: Afonso Lourenço, João Gama, Eric P. Xing, Goreti Marreiros,
• Abstract summary: This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments.<n> Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees across all non-stationary benchmarks.
• Score: 42.13420474990124
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State-of-the-art data stream mining in supervised classification has traditionally relied on ensembles of incremental decision trees. However, the emergence of large tabular models, i.e., transformers designed for structured numerical data, marks a significant paradigm shift. These models move beyond traditional weight updates, instead employing in-context learning through prompt tuning. By using on-the-fly sketches to summarize unbounded streaming data, one can feed this information into a pre-trained model for efficient processing. This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments. Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees across all non-stationary benchmarks. Several promising research directions are outlined in the paper. The authors urge the community to explore these ideas, offering valuable opportunities to advance in-context stream learning.

Related papers

• Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data [39.40116554523575]
  We present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network. It learns to approximate Bayesian inference on synthetic datasets drawn from a prior. It improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration.
  arXiv  Detail & Related papers  (2024-11-15T23:49:23Z)
• Mambular: A Sequential Model for Tabular Deep Learning [0.7184556517162347]
  This paper investigates the use of autoregressive state-space models for tabular data. We compare their performance against established benchmark models. Our findings indicate that interpreting features as a sequence and processing them can lead to significant performance improvement.
  arXiv  Detail & Related papers  (2024-08-12T16:57:57Z)
• Transformers with Stochastic Competition for Tabular Data Modelling [6.285325771390289]
  We introduce a novel deep learning model specifically designed for tabular data. The model is validated on a variety of widely-used, publicly available datasets. We demonstrate that, through the incorporation of these elements, our model yields high performance.
  arXiv  Detail & Related papers  (2024-07-18T07:48:48Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Beyond Transfer Learning: Co-finetuning for Action Localisation [64.07196901012153]
  We propose co-finetuning -- simultaneously training a single model on multiple upstream'' and downstream'' tasks. We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data. We also show how we can easily extend our approach to multiple upstream'' datasets to further improve performance.
  arXiv  Detail & Related papers  (2022-07-08T10:25:47Z)
• Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models [89.44031286278347]
  We propose a Hub-Pathway framework to enable knowledge transfer from a model hub. The proposed framework can be trained end-to-end with the target task-specific loss. Experiment results on computer vision and reinforcement learning tasks demonstrate that the framework achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-06-08T08:00:12Z)
• How Well Do Sparse Imagenet Models Transfer? [75.98123173154605]
  Transfer learning is a classic paradigm by which models pretrained on large "upstream" datasets are adapted to yield good results on "downstream" datasets. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset. We show that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities.
  arXiv  Detail & Related papers  (2021-11-26T11:58:51Z)
• Adaptive Deep Forest for Online Learning from Drifting Data Streams [15.49323098362628]
  Learning from data streams is among the most vital fields of contemporary data mining. We propose Adaptive Deep Forest (ADF) - a natural combination of the successful tree-based streaming classifiers with deep forest. The conducted experiments show that the deep forest approach can be effectively transformed into an online algorithm.
  arXiv  Detail & Related papers  (2020-10-14T18:24:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.