Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning

• URL: http://arxiv.org/abs/2606.11853v1
• Date: Wed, 10 Jun 2026 09:30:25 GMT
• Title: Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning
• Abstract summary: Multi-modal large language models (MLLMs) depend on in-context learning (ICL) for rapid task adaptation.<n>Existing memory compression approaches rely on rigid token removal or sample-dependent importance estimation.<n>We introduce TASM, a training-free framework that addresses these limitations through task-aware, structure-preserving, and dynamically accessible memory construction.
• Score: 40.577016426387694
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-modal large language models (MLLMs) depend on in-context learning (ICL) for rapid task adaptation, but their scalability is severely limited by finite context windows and the growing cost of key-value (KV) caches in long multi-modal sequences. Existing memory compression approaches typically rely on rigid token removal or sample-dependent importance estimation, which introduces bias, disrupts semantic structure, particularly for visual representations, and yields static memories that cannot adapt to new queries. We introduce TASM (Task-Aware Structured Memory), a training-free framework that addresses these limitations through task-aware, structure-preserving, and dynamically accessible memory construction. TASM employs task-vector guided compression to replace sample-specific signals with a task-level direction that captures shared relevance across demonstrations. To preserve the underlying manifold, it applies semantics-aware token merging via bipartite graph matching, aggregating tokens without destructive pruning. Finally, TASM structures memory into a hierarchy comprising a compact Core Memory and a Latent Bank, facilitating query-adaptive dynamic retrieval. Evaluations confirm TASM maintains high performance under heavy compression, effectively balancing efficiency with adaptability.

Related papers

• Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents [29.28371047415946]
  HORMA is a hierarchical organize-and-Retrieve Memory Agent for large language models.<n>It decomposes working memory into two stages: structured memory construction and navigation-based retrieval.<n>It consistently achieves better efficiency-performance trade-offs and generalizes effectively to unseen tasks.
  arXiv  Detail & Related papers  (2026-06-10T05:49:14Z)
• Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression [54.12624130407679]
  We propose T-Switch, which decomposes task vectors into three compact components.<n>We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors.<n>We present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.
  arXiv  Detail & Related papers  (2026-04-30T16:58:05Z)
• AMA: Adaptive Memory via Multi-Agent Collaboration [54.490349689939166]
  We propose Adaptive Memory via Multi-Agent Collaboration (AMA), a novel framework that leverages coordinated agents to manage memory across multiple granularities.<n>AMA significantly outperforms state-of-the-art baselines while reducing token consumption by approximately 80% compared to full-context methods.
  arXiv  Detail & Related papers  (2026-01-28T08:09:49Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• AR-MOT: Autoregressive Multi-object Tracking [56.09738000988466]
  We propose a novel autoregressive paradigm that formulates MOT as a sequence generation task within a large language model (LLM) framework.<n>This design enables the model to output structured results through flexible sequence construction, without requiring any task-specific heads.<n>To enhance region-level visual perception, we introduce an Object Tokenizer based on a pretrained detector.
  arXiv  Detail & Related papers  (2026-01-05T09:17:28Z)
• TALM: Dynamic Tree-Structured Multi-Agent Framework with Long-Term Memory for Scalable Code Generation [0.0]
  Agentic code generation requires large language models capable of complex context management and multi-step reasoning.<n>We propose TALM, a dynamic framework that integrates structured task decomposition, localized re-reasoning, and long-term memory mechanisms.<n> Experimental results on HumanEval, BigCodeBench, and ClassEval benchmarks demonstrate that TALM consistently delivers strong reasoning performance and high token efficiency.
  arXiv  Detail & Related papers  (2025-10-27T05:07:36Z)
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension [55.29309306566238]
  Current Large Language Models (LLMs) are confronted with overwhelming information volume when comprehending long-form documents.<n>This challenge raises the imperative of a cohesive memory module, which can elevate vanilla LLMs into autonomous reading agents.<n>We draw inspiration from Jean Piaget's Constructivist Theory, illuminating three traits of the agentic memory -- structured schemata, flexible assimilation, and dynamic accommodation.
  arXiv  Detail & Related papers  (2025-10-07T02:16:30Z)
• Task Memory Engine: Spatial Memory for Robust Multi-Step LLM Agents [0.0]
  Large Language Models (LLMs) falter in multi-step interactions due to reliance on linear, unstructured context.<n>We introduce the Task Memory Engine (TME), a modular memory controller that transforms existing LLMs into robust, revision-aware agents.<n>TME implements a spatial memory framework that replaces flat context with graph-based structures to support consistent, multi-turn reasoning.
  arXiv  Detail & Related papers  (2025-05-26T02:53:22Z)
• Task Memory Engine (TME): Enhancing State Awareness for Multi-Step LLM Agent Tasks [0.0]
  We propose a lightweight and structured memory module that tracks task execution using a hierarchical Task Memory Tree (TMT)<n>TME is designed to be graph-aware, supporting reusable substeps, converging task paths, and shared dependencies.
  arXiv  Detail & Related papers  (2025-04-11T13:38:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.