Bounded Context Management for Tabular Foundation Models on Stream Learning

• URL: http://arxiv.org/abs/2606.18677v1
• Date: Wed, 17 Jun 2026 04:24:26 GMT
• Title: Bounded Context Management for Tabular Foundation Models on Stream Learning
• Abstract summary: We propose a future information view that yields three practical requirements for context management.<n>We instantiate these requirements as CURE, a context-managing policy with entropy-gated admission and redundancy-aware eviction.<n>CURE shows up to 27.0% relative improvement over classical stream learners, remains robust across multiple TFM backbones, and ranks first among other policy variants.
• Score: 8.331524801839162
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular stream learning requires predictions on sequentially arriving examples under distribution shift. While standard methods adapt by updating model states, tabular foundation models (TFMs) make predictions conditioned on a labeled context in an in-context manner, making them a natural alternative for stream learning. This shifts the challenge from how to update the model to how to manage the context. We propose a future information view that yields three practical requirements for context management: preserve recent examples, retain uncertain examples, and remove redundant examples. We instantiate these requirements as CURE (Context management via Uncertainty-aware admission and Redundancy aware Eviction), a context-managing policy with entropy-gated admission and redundancy-aware eviction. Across seven streams, CURE shows up to 27.0% relative improvement over classical stream learners, remains robust across multiple TFM backbones, and ranks first among other policy variants. Code and datasets are available at https://github.com/morcellinus/CURE-ICML-FMSD.

Related papers

• Streaming Hierarchical Inference with Tabular Foundation Models [1.43494686131174]
  textitHINT is a hierarchical inference framework that combines edge-based retrieval with cloud-based TFM inference.<n>A graph-based approximate nearest neighbor memory maintained over a sliding window provides local predictions and uncertainty estimates.<n>Experiments show textitHINT consistently identifies favorable trade-offs.
  arXiv  Detail & Related papers  (2026-09-07T20:26:36Z)
• Self-Consolidating Language Models: Continual Knowledge Incorporation from Context [5.0834716824529105]
  Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context.<n>We study continual context consolidation: writing current context into model weights while limiting interference with previously consolidated information.<n>We propose textbfSelf-textbfConsolidating textbfLanguage Models (SCoL), a post-training framework in which an LLM learns to generate textual update instructions specifying which of its own Transformer layers should be updated.
  arXiv  Detail & Related papers  (2026-05-08T00:50:09Z)
• Learning from Many and Adapting to the Unknown in Open-set Test Streams [24.690368986775848]
  Large Language Models (LLMs) generalize across tasks via reusable representations and flexible reasoning.<n>SyCo is a parameter-efficient adaptation method that updates low-rank adapters through Rac1 and MAPK pathways.
  arXiv  Detail & Related papers  (2026-04-01T06:23:24Z)
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• Context-level Language Modeling by Learning Predictive Context Embeddings [79.00607069677393]
  We introduce textbfContextLM, a framework that augments standard pretraining with an inherent textbfnext-context prediction objective.<n>This mechanism trains the model to learn predictive representations of multi-token contexts, leveraging error signals derived from future token chunks.<n>Experiments on the GPT2 and Pythia model families, scaled up to $1.5$B parameters, show that ContextLM delivers consistent improvements in both perplexity and downstream task performance.
  arXiv  Detail & Related papers  (2025-10-23T07:09:45Z)
• Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning [19.677969862434708]
  We present a theoretically grounded, resource-efficient framework for data selection and reweighting.<n>At its core is an In-Context Approximation (ICA) that estimates the holdout loss a model would incur after training on a candidate example.<n>We derive per-example weights from ICA scores, dynamically reweighting gradient updates as model parameters evolve.
  arXiv  Detail & Related papers  (2025-10-16T09:00:39Z)
• PICASO: Permutation-Invariant Context Composition with State Space Models [98.91198288025117]
  State Space Models (SSMs) offer a promising solution by allowing a database of contexts to be mapped onto fixed-dimensional states.<n>We propose a simple mathematical relation derived from SSM dynamics to compose multiple states into one that efficiently approximates the effect of concatenating raw context tokens.<n>We evaluate our resulting method on WikiText and MSMARCO in both zero-shot and fine-tuned settings, and show that we can match the strongest performing baseline while enjoying on average 5.4x speedup.
  arXiv  Detail & Related papers  (2025-02-24T19:48:00Z)
• On the Loss of Context-awareness in General Instruction Fine-tuning [101.03941308894191]
  We investigate the loss of context awareness after supervised fine-tuning.<n>We find that the performance decline is associated with a bias toward different roles learned during conversational instruction fine-tuning.<n>We propose a metric to identify context-dependent examples from general instruction fine-tuning datasets.
  arXiv  Detail & Related papers  (2024-11-05T00:16:01Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.