Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

• URL: http://arxiv.org/abs/2608.22963v2
• Date: Thu, 27 Aug 2026 11:42:10 GMT
• Title: Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents
• Abstract summary: We propose a Kullback-Leibler-guided framework for pruning accumulated reasoning in tool-use agents.<n> SPARE achieves the highest average accuracy among pruning methods while removing 37.89-64.58% of reasoning tokens.<n>This favorable accuracy-context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.
• Score: 16.82888862043366
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning supports task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. We observe that reasoning becomes redundant once task-relevant visual evidence is grounded, while stale hypotheses can misguide later inference when grounding remains uncertain. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback-Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89-64.58\% of reasoning tokens. This favorable accuracy-context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.

Related papers

• SAGE: From Direct Answering to Evidence-Grounded Inference for Chinese Ancient Document Understanding [61.02068339726333]
  We propose SAGE, an evidence-grounded multi-agent framework for Chinese ancient document understanding.<n>SAGE coordinates specialized agents for task-aware planning, tool-mediated evidence acquisition, claim-level verification, and bounded replanning.<n> Experiments on the AncientDoc benchmark show that SAGE consistently outperforms matched direct-answering baselines.
  arXiv  Detail & Related papers  (2026-08-25T03:01:51Z)
• Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation [74.0621258662676]
  We propose Ptah, a multi-agent harness for interleaved report generation.<n>Ptah orchestrates the lifecycle from user query to rendered web report through planning, research, and writing stages.<n>A verifier agent serves as the harness's acceptance function, enforcing factual grounding, citation fidelity, and cross-modal consistency throughout the workflow.
  arXiv  Detail & Related papers  (2026-05-28T12:40:34Z)
• Decompose, Look, and Reason: Reinforced Latent Reasoning for VLMs [6.111899371682025]
  Vision-Language Models often struggle with complex visual reasoning due to the visual information loss in textual CoT.<n>We propose emph"Decompose, Look, and Reason" (DLR), a reinforced latent reasoning framework.<n>Experiments on vision-centric benchmarks show that DLR consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-04-08T18:52:27Z)
• Parallel LLM Reasoning for Bias-Resilient, Robust Conceptual Abstraction [0.0]
  This study proposes a structured framework combining parallel chunk-level processing with evidence-anchored consolidation.<n>Experiments with multiple model types and sizes indicate that parallel processing significantly reduces omission error by approximately 84%.<n>Smaller models benefited most, suggesting that efficient parallel chunking and consolidation play a critical role in achieving reliable and scalable textual analysis.
  arXiv  Detail & Related papers  (2026-04-04T05:11:20Z)
• PLUME: Latent Reasoning Based Universal Multimodal Embedding [52.35354073629127]
  Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
  arXiv  Detail & Related papers  (2026-04-02T14:04:53Z)
• Saliency-Aware Multi-Route Thinking: Revisiting Vision-Language Reasoning [50.62037276161025]
  Vision-language models (VLMs) aim to reason by jointly leveraging visual and textual modalities.<n>Key obstacle is that visual inputs are typically provided only once at the start of generation.<n>We propose emphSaliency-Aware Principle (SAP) selection.
  arXiv  Detail & Related papers  (2026-02-18T18:49:56Z)
• Reasoning-Augmented Representations for Multimodal Retrieval [27.4146940988752]
  Universal Multimodal Retrieval (UMR) seeks any-to-any search across text and vision.<n>We argue this brittleness is often data-specified: when images carry "silent" evidence and queries leave key semantics implicit, a single embedding pass must both reason and compress.<n>We propose a data-centric framework that decouples these roles by externalizing reasoning before retrieval.
  arXiv  Detail & Related papers  (2026-02-06T19:01:54Z)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)
• TS-Agent: A Time Series Reasoning Agent with Iterative Statistical Insight Gathering [16.95452463476229]
  We propose TS-Agent, a time series reasoning agent for large language models (LLMs)<n>Instead of mapping time series into text tokens, images, or embeddings, our agent interacts with raw numeric sequences through atomic operators.<n>Our experiments show that TS-Agent achieves performance comparable to state-of-the-art LLMs on understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-08T18:31:53Z)
• Context-Aware Hierarchical Merging for Long Document Summarization [56.96619074316232]
  We propose different approaches to enrich hierarchical merging with context from the source document.<n> Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines.
  arXiv  Detail & Related papers  (2025-02-03T01:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.