Silent Failures in Multimodal Agentic Search:A Diagnostic Taxonomy and Cross-Judge Evaluation

• URL: http://arxiv.org/abs/2607.19793v1
• Date: Wed, 22 Jul 2026 06:19:06 GMT
• Title: Silent Failures in Multimodal Agentic Search:A Diagnostic Taxonomy and Cross-Judge Evaluation
• Abstract summary: Existing evaluations mainly focus on final-answer accuracy and may miss failures in the search trajectory.<n>We introduce a six-category taxonomy covering shortcuts, phantom grounding, wrong-evidence-right-answer cases, over-retrieval laundering, cross-modal contradiction, and provenance.<n>We build a trajectory-level diagnostic pipeline that evaluates both answer correctness and evidence-grounding quality under a unified ReAct-style scaffold.
• Score: 7.095219140870008
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal agentic search systems increasingly rely on external tools to answer knowledge-intensive visual questions. However, existing evaluations mainly focus on final-answer accuracy and may miss failures in the search trajectory. In this work, we study such hidden reliability issues as silent failures. We introduce a six-category taxonomy covering modality shortcuts, phantom grounding, wrong-evidence-right-answer cases, over-retrieval laundering, cross-modal contradiction, and provenance hallucination. Based on this taxonomy, we build a trajectory-level diagnostic pipeline that evaluates both answer correctness and evidence-grounding quality under a unified ReAct-style scaffold. Experiments on MMSearch-Plus trajectories across four frontier multimodal models show that surface accuracy consistently overestimates true trajectory-level correctness. We further use cross-judge validation, blank-image stress tests, and tool ablations to show that silent failures are capability-dependent and often shift rather than disappear. Home-page: https://github.com/DingWu1021/silent-failures-multimodal-agentic-search

Related papers

• Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures [11.365690419023425]
  Existing root-cause attribution methods suffer from low diagnostic accuracy as execution traces grow larger.<n>Continual Search nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence.<n>MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks.
  arXiv  Detail & Related papers  (2026-09-11T19:29:21Z)
• SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents [56.400332595452966]
  We introduce SearchAuditBench, a benchmark that evaluates whether auditors can localize, attribute, and repair search-agent failures.<n>SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks.<n>We propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures.
  arXiv  Detail & Related papers  (2026-08-05T09:09:00Z)
• Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents [63.2133766267587]
  Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence.<n>We study these questions through a trajectory-level diagnosis of long-horizon search agents.<n>We evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence.
  arXiv  Detail & Related papers  (2026-08-03T08:46:48Z)
• HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning [14.484579581234327]
  We introduce HopRefusalBench, the first benchmark of refusal within multi-hop search.<n>It crosses three causes of unanswerability with root, middle, and terminal topologies.<n>We propose a final-outcome taxonomy spanning target-aware refusal, pseudo-refusal, hallucinated completion, and search-budget exhaustion.
  arXiv  Detail & Related papers  (2026-08-02T16:20:55Z)
• AREX: Towards a Recursively Self-Improving Agent for Deep Research [75.96468303743958]
  We introduce AREX, a family of Recursively Self-Improving deep research agents.<n>AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer.<n>We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning.
  arXiv  Detail & Related papers  (2026-07-23T16:05:46Z)
• Affordance Agent Harness: Verification-Gated Skill Orchestration [45.231685718099264]
  Affordance grounding requires identifying where and how an agent should interact in open-world scenes.<n>We propose Affordance Agent Harness, a closed-loop runtime that unifies skills with an evidence store and cost control.
  arXiv  Detail & Related papers  (2026-05-01T13:45:16Z)
• When Is Enough Not Enough? Illusory Completion in Search Agents [56.98225130959051]
  We study whether search agents reliably reason across all requirements by tracking, verifying, and maintaining multiple conditions.<n>We find that illusory completion frequently occurs, wherein agents believe tasks are complete despite unresolved or violated constraints, leading to underverified answers.<n>We examine whether explicit constraint-state tracking during execution mitigates these failures via LiveLedger, an inference-time tracker.
  arXiv  Detail & Related papers  (2026-02-07T13:50:38Z)
• CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution [20.823419395675412]
  CrossCheck-Bench is a diagnostic benchmark for evaluating contradiction detection in multimodal inputs.<n>We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection.
  arXiv  Detail & Related papers  (2025-11-19T12:17:15Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [89.92916473403108]
  This paper proposes a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods.<n>The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics.<n>We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• ReDFeat: Recoupling Detection and Description for Multimodal Feature Learning [51.07496081296863]
  We recouple independent constraints of detection and description of multimodal feature learning with a mutual weighting strategy. We propose a detector that possesses a large receptive field and is equipped with learnable non-maximum suppression layers. We build a benchmark that contains cross visible, infrared, near-infrared and synthetic aperture radar image pairs for evaluating the performance of features in feature matching and image registration tasks.
  arXiv  Detail & Related papers  (2022-05-16T04:24:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.