Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents

• URL: http://arxiv.org/abs/2605.31354v1
• Date: Fri, 29 May 2026 14:29:56 GMT
• Title: Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents
• Abstract summary: We study failure modes of collaborative reasoning with weak learners through the lens of noise accumulation.<n>We introduce CoSee, an auditing framework that formalizes the read-write-verify loop to trace information flow in document visual question answering.<n>We identify two dominant failure modes: Noise Reinforcement, where ungrounded notes are reused as evidence, and Policy Collapse, where added context shifts the model toward under-specified workspaces.
• Score: 2.5754366051855837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modular visual reasoning systems increasingly rely on shared working memory for multi-step collaboration, yet the failure dynamics of intermediate state evolution in low-capacity regimes remain underexplored. We study failure modes of collaborative reasoning with weak learners (4B--8B models) through the lens of noise accumulation. We introduce CoSee, an auditing framework that formalizes the read-write-verify loop to trace information flow in document visual question answering. Across multi-page, chart, and web-based benchmarks, we find a counter-intuitive degradation: naive shared workspaces often amplify hallucinations rather than resolve them. We identify two dominant failure modes: Noise Reinforcement, where ungrounded notes are reused as evidence, and Policy Collapse, where added context shifts the model toward under-specified, short-form answers. Using cost-accuracy Pareto frontiers, we show that increased compute can correlate negatively with performance without explicit verification. Our findings suggest that for resource-constrained agents, the bottleneck lies not in reasoning depth but in communication fidelity, providing trace-level diagnostics and a mechanistic baseline for reliable modular design.

Related papers

• The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context [22.202409807556517]
  We introduce Computational Reality Monitoring to address the attribution blind spot.<n> CRM operationalizes a principle adapted from cognitive science's reality monitoring framework.<n>It detects whether pretraining exposure leaves a measurable internal trajectory signature.
  arXiv  Detail & Related papers  (2026-05-26T09:48:51Z)
• Why Retrieval-Augmented Generation Fails: A Graph Perspective [40.34482080762174]
  We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation.<n>We develop a graph-based error detection framework that uses attribution-graph topology features.
  arXiv  Detail & Related papers  (2026-05-13T23:18:07Z)
• Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs [68.58207076756237]
  We introduce ProCauEval, a perturbation-based evaluation protocol that shifts from outcome assessment to mechanism diagnosis.<n>We find that models faithfully perceive video content yet systematically underexploit it during causal reasoning.<n>We propose Anti-Distillation Policy Optimization (ADPO), a reinforcement learning framework built on negative teacher alignment.
  arXiv  Detail & Related papers  (2026-05-10T08:48:58Z)
• Guided Verifier: Collaborative Multimodal Reasoning via Dynamic Process Supervision [11.159231524113764]
  Reinforcement Learning (RL) has emerged as a pivotal mechanism for enhancing the complex reasoning capabilities of Multimodal Large Language Models (MLLMs)<n>In this paper, we propose the textbfGuided Verifier framework to address these structural limitations.<n>We develop a specialized data synthesis pipeline targeting multimodal hallucinations, constructing textbfCoRe dataset of process-level negatives and textbfCorrect-guide textbfReasoning trajectories to train the guided verifier.
  arXiv  Detail & Related papers  (2026-02-04T07:38:42Z)
• CoG: Controllable Graph Reasoning via Relational Blueprints and Failure-Aware Refinement over Knowledge Graphs [53.199517625701475]
  CoG is a training-free framework inspired by Dual-Process Theory that mimics the interplay between intuition and deliberation.<n>CoG significantly outperforms state-of-the-art approaches in both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-01-16T07:27:40Z)
• Lost in the Noise: How Reasoning Models Fail with Contextual Distractors [57.31788955167306]
  Recent advances in reasoning models and agentic AI systems have led to an increased reliance on diverse external information.<n>We introduce NoisyBench, a comprehensive benchmark that systematically evaluates model robustness across 11 datasets in RAG, reasoning, alignment, and tool-use tasks.<n>Our evaluation reveals a catastrophic performance drop of up to 80% in state-of-the-art models when faced with contextual distractors.
  arXiv  Detail & Related papers  (2026-01-12T05:43:51Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Graphing the Truth: Structured Visualizations for Automated Hallucination Detection in LLMs [0.0]
  This paper introduces a framework that organizes proprietary knowledge and model-generated content into interactive visual knowledge graphs.<n>Users can diagnose inconsistencies, identify weak reasoning chains, and supply corrective feedback.
  arXiv  Detail & Related papers  (2025-11-29T23:09:15Z)
• When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning [22.39245479538899]
  We introduce modality sabotage, a diagnostic failure mode in which a high-confidence unimodal error overrides other evidence and misleads the fused result.<n>A model-agnostic evaluation layer treats each modality as an agent, producing candidate labels and a brief self-assessment used for auditing.<n>A simple fusion mechanism aggregates these outputs, exposing contributors (modalities supporting correct outcomes) and saboteurs (modalities that mislead)
  arXiv  Detail & Related papers  (2025-11-04T18:20:13Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning [49.17801010041155]
  Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio.<n>Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance.<n>We categorize multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined.
  arXiv  Detail & Related papers  (2025-09-28T08:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.