Towards Evolving Context Parameterization for Large Language Models

• URL: http://arxiv.org/abs/2609.14168v1
• Date: Sat, 12 Sep 2026 21:55:33 GMT
• Title: Towards Evolving Context Parameterization for Large Language Models
• Abstract summary: PLUME is a training-free method that constructs a global update representation and activates memory evidence to form a local parameter view.<n>We show PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge.
• Score: 13.314849988154707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context parameterization enables large language models (LLMs) to internalize contexts into reusable model parameters, avoiding repeated processing across subsequent queries. However, existing methods typically assume static contexts and lack explicit mechanisms for distinguishing validity states under continual updates. To study this real-world scenario, we formalized the Memory Updating with Sequential Evolution (MUSE) task and constructed MUSE-bench to evaluate update incorporation and unaffected-information preservation. The resulting challenge requires preserving the global state while adjusting the contribution of memory evidence. Motivated by this, we proposed PLUME, a training-free method that constructs a global update representation, activates memory evidence to form a local parameter view, and adaptively integrates their predictions during decoding. Comprehensive evaluation on MUSE-bench demonstrated PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge. Our codes are available at: https://github.com/xiaobingshi-LLM/PLUME.

Related papers

• MemWM: Memory-Augmented Text-Based World Model [49.654556986087535]
  We introduce MemWM, a memory-augmented text-based world model.<n>MemWM uses world memory, a curated memory bank of transition rules, state caches, and hard-to-predict facts.<n>We evaluate factual state preservation with Structured State Fidelity (SSF), which scores predicted states through benchmark-specific facts and fields.<n>In the full planning setting, we keep the policy model frozen and provide policy-side world skill: retrieved task-level skills and step-wise corrective guidance for action selection.
  arXiv  Detail & Related papers  (2026-08-07T11:03:32Z)
• MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models [64.2820121997882]
  MIITA is a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage.<n>It stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues.<n>Experiments show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.
  arXiv  Detail & Related papers  (2026-05-20T03:03:04Z)
• Improving Sparse Memory Finetuning [6.491055956716639]
  Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities.<n>Standard approaches to updating models, like full finetuning or parameter-efficient methods, face a fundamental trade-off: catastrophic forgetting.<n>Sparse Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit memory layers.
  arXiv  Detail & Related papers  (2026-04-06T23:16:10Z)
• Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models [60.25003813232697]
  We propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM) as a training-free adaptive guidance framework for Masked Diffusion Models (MDMs)<n>ARAM calibrates the guidance scale during denoising according to the Signal-to-Noise Ratio (SNR) of the distributional shift induced by retrieved context.<n>Experiments on multiple knowledge-intensive QA benchmarks show that ARAM improves overall QA performance over competitive RAG baselines.
  arXiv  Detail & Related papers  (2026-03-18T12:54:50Z)
• Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams [78.9560820155271]
  We evaluate online adaptation over streaming, continually updating knowledge.<n> OAKS comprises two datasets: OAKS-BABI and OAKS-Novel.<n>Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS.
  arXiv  Detail & Related papers  (2026-03-08T00:38:39Z)
• Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories [44.46300411842271]
  Locas is a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers.<n>We show that proper initialization of such low-rank sideway-FFN-style memories is essential for fast convergence, improved generalization, and catastrophic prevention.
  arXiv  Detail & Related papers  (2026-02-04T22:09:40Z)
• LLM-as-RNN: A Recurrent Language Model for Memory Updates and Sequence Prediction [16.71499144106184]
  Large language models are strong sequence predictors, yet standard inference relies on immutable context histories.<n>We propose LLM-as-RNN, an inference-only framework that turns a frozen LLM into a recurrent predictor by representing its hidden state as natural-language memory.<n>We evaluate the method on three sequential benchmarks in healthcare, meteorology, and finance across Llama, Gemma, and GPT model families.
  arXiv  Detail & Related papers  (2026-01-19T19:41:39Z)
• Meta-Policy Reflexion: Reusable Reflective Memory and Rule Admissibility for Resource-Efficient LLM Agent [6.300669721057781]
  Meta-Policy Reflexion (MPR) is a framework that consolidates LLM-generated reflections into a structured, predicate-like Meta-Policy Memory (MPM)<n>MPR externalizes reusable corrective knowledge without model weight updates, enforces domain constraints to reduce unsafe or invalid actions, and retains the adaptability of language-based reflection.<n> Empirical results reported in the supplied material indicate consistent gains in execution accuracy and robustness when compared to Reflexion baselines; rule admissibility further improves stability.
  arXiv  Detail & Related papers  (2025-09-04T08:18:39Z)
• Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models [7.075648770762989]
  Fine-tuning large language models with low-rank adaptaion (LoRA) is a cost-effective way to incorporate information from a specific dataset.<n>It is often unclear how well the fine-tuned LLM will generalize, i.e., how well it will perform on unseen datasets.<n>We propose Amortized Bayesian Meta-Learning for LoRA (ABMLL) to improve generalization and scales to large models.
  arXiv  Detail & Related papers  (2025-08-19T21:57:59Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation [87.98063273826702]
  We propose a memory imitation meta-learning (MemIML) method that enhances the model's reliance on support sets for task adaptation. A theoretical analysis is provided to prove the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-03-22T12:41:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.