InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations

• URL: http://arxiv.org/abs/2609.01383v1
• Date: Tue, 01 Sep 2026 15:15:30 GMT
• Title: InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
• Abstract summary: In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations.<n>The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments.<n>Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence.
• Score: 5.169269482194285
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interactive environments. Existing benchmarks are predominantly constrained to static imagery and one-shot question answering and fail to capture the epistemic demands of this domain, where evidence is frequently occluded, distributed across linked views, or conditionally revealed through user agency. In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations. The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments. Agents must navigate these environments to determine whether a natural language claim is supported, refuted or not verifiable given the available evidence. Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence. We evaluate state-of-the-art models, revealing that interactive verification remains a non-trivial challenge. We release InSight at https://github.com/maevehutch/insight.

Related papers

• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• Active Perception for Embodied Disambiguation [5.8993656688823135]
  We propose an active-perception framework for embodied target disambiguation.<n>We show that the framework combines physical information acquisition and userintent clarification.<n>Real-robot experiments show that the framework combines physical information acquisition and userintent clarification.
  arXiv  Detail & Related papers  (2026-08-11T09:47:20Z)
• Behaviorally Adaptive Visual Diversion for Inclusive and Resilient Digital Assessment Delivery [0.0]
  This paper introduces Behaviorally-Adaptive Visual Diversion (BAVD), a theoretical framework in which a synthetic, non-semantic visual field is composited with assessment content.<n>BAVD incorporates an accessibility-aware mechanism that reduces or suppresses diversion intensity for candidates with approved visual-processing accommodations.
  arXiv  Detail & Related papers  (2026-08-04T12:13:25Z)
• Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions [15.505991441542372]
  We propose a framework that recasts evaluation as a three-party interaction between an evaluatee model, an automated examiner, and a grader.<n>By navigating state-space transitions, CEDI deploys diverse strategies, from clarification requests to adversarial probes, to elicit performance evidence.<n>We show that hallucinations often accumulate over long contexts, through self-reinforcing dialogue history, and models are particularly vulnerable to questions requiring premise rejection or refusal.
  arXiv  Detail & Related papers  (2026-07-16T02:25:03Z)
• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Using Machine Mental Imagery for Representing Common Ground in Situated Dialogue [3.1039961644960186]
  Situated dialogue requires speakers to maintain a reliable representation of shared context.<n>Current conversational agents often struggle with this requirement.<n>We introduce an active visual scaffolding framework that incrementally converts dialogue state into a persistent visual history.
  arXiv  Detail & Related papers  (2026-04-22T23:15:42Z)
• Interactive Benchmarks [45.705288760439636]
  We propose Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints.<n>We instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities.
  arXiv  Detail & Related papers  (2026-03-05T02:18:26Z)
• DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories [52.57197752244638]
  We introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task.<n>Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues.<n>We construct DISBench, a challenging benchmark built on interconnected visual data.
  arXiv  Detail & Related papers  (2026-02-11T12:51:10Z)
• Automated Detection of Visual Attribute Reliance with a Self-Reflective Agent [58.90049897180927]
  We introduce an automated framework for detecting unintended reliance on visual features in vision models.<n>A self-reflective agent generates and tests hypotheses about visual attributes that a model may rely on.<n>We evaluate our approach on a novel benchmark of 130 models designed to exhibit diverse visual attribute dependencies.
  arXiv  Detail & Related papers  (2025-10-24T17:59:02Z)
• Generalized Visual Relation Detection with Diffusion Models [94.62313788626128]
  Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image.<n>We propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner.<n>Our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets.
  arXiv  Detail & Related papers  (2025-04-16T14:03:24Z)
• Exploiting Multi-Object Relationships for Detecting Adversarial Attacks in Complex Scenes [51.65308857232767]
  Vision systems that deploy Deep Neural Networks (DNNs) are known to be vulnerable to adversarial examples. Recent research has shown that checking the intrinsic consistencies in the input data is a promising way to detect adversarial attacks. We develop a novel approach to perform context consistency checks using language models.
  arXiv  Detail & Related papers  (2021-08-19T00:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.