DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards

• URL: http://arxiv.org/abs/2508.17398v1
• Date: Sun, 24 Aug 2025 15:11:44 GMT
• Title: DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards
• Abstract summary: DashboardQA is a benchmark designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards.<n>It includes 112 interactive dashboards from Tableau Public and 405 question-answer pairs with interactive dashboards spanning five categories: multiple-choice, factoid, hypothetical, multi-dashboard, and conversational.<n>Our findings indicate that interactive dashboard reasoning is a challenging task overall for all the VLMs evaluated.
• Score: 44.69783955774917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dashboards are powerful visualization tools for data-driven decision-making, integrating multiple interactive views that allow users to explore, filter, and navigate data. Unlike static charts, dashboards support rich interactivity, which is essential for uncovering insights in real-world analytical workflows. However, existing question-answering benchmarks for data visualizations largely overlook this interactivity, focusing instead on static charts. This limitation severely constrains their ability to evaluate the capabilities of modern multimodal agents designed for GUI-based reasoning. To address this gap, we introduce DashboardQA, the first benchmark explicitly designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards. The benchmark includes 112 interactive dashboards from Tableau Public and 405 question-answer pairs with interactive dashboards spanning five categories: multiple-choice, factoid, hypothetical, multi-dashboard, and conversational. By assessing a variety of leading closed- and open-source GUI agents, our analysis reveals their key limitations, particularly in grounding dashboard elements, planning interaction trajectories, and performing reasoning. Our findings indicate that interactive dashboard reasoning is a challenging task overall for all the VLMs evaluated. Even the top-performing agents struggle; for instance, the best agent based on Gemini-Pro-2.5 achieves only 38.69% accuracy, while the OpenAI CUA agent reaches just 22.69%, demonstrating the benchmark's significant difficulty. We release DashboardQA at https://github.com/vis-nlp/DashboardQA

Related papers

• DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation [8.452268647068166]
  We introduce DashArena, to our knowledge the first benchmark for open-ended, task-grounded generation of interactive analytic dashboards.<n>DashArena requires each system to generate both a dashboard and a replayable interaction trajectory.<n>A browser replays the trajectory and turns the system's intended analytical workflow into reproducible visual and execution evidence.
  arXiv  Detail & Related papers  (2026-08-11T06:54:37Z)
• Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards [50.38481199112112]
  We introduce Dashboard2Code, a novel task that requires a model to proactively explore an interactive dashboard, acquire and integrate feedback from its own interactions, and generate code that reproduces the target dashboard.<n>We present DashboardMimic, the first Plotly+Dash benchmark for Dashboard2Code, comprising 180 carefully designed and manually verified dashboard-code pairs spanning three difficulty levels and covering eight common real-world interaction patterns.<n>We propose an automated evaluation framework tailored to dashboards that combines code semantic analysis with dynamic interaction-based testing to assess visual and interaction consistency.
  arXiv  Detail & Related papers  (2026-07-06T06:58:43Z)
• AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark [32.66632642377623]
  AutoGUI-v2 is a benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction.<n>We construct the benchmark using a novel VLM-human collaborative pipeline that parses screenshots into hierarchical functional regions.<n>AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction.
  arXiv  Detail & Related papers  (2026-04-27T13:06:27Z)
• PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis [35.75779621306803]
  We introduce PaveBench, a benchmark for pavement distress perception and interactive vision-language analysis.<n>PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering.<n>On the multimodal side, we introduce PaveVQA, a real-image question answering dataset.
  arXiv  Detail & Related papers  (2026-04-03T07:15:41Z)
• UIPro: Unleashing Superior Interaction Capability For GUI Agents [33.77980648230746]
  Building autonomous agents that perceive and operate graphical user interfaces (GUIs) like humans has long been a vision in the field of artificial intelligence.<n>Existing methods have tried developing GUI agents based on the multi-modal comprehension ability of vision-language models (VLMs)<n>This paper proposes textUIPro, a novel generalist GUI agent trained with extensive multi-platform and multi-task GUI interaction data.
  arXiv  Detail & Related papers  (2025-09-22T03:04:53Z)
• FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agents [12.315613848863784]
  We introduce FineState-Bench, the first evaluation and diagnostic standard for fine-grained GUI proxy operations.<n>FineState-Bench includes 2257 task benchmarks in four components and uses a four-phase indicator for perception-to-control assessment.<n>Our diagnostic framework confirms for the first time that the primary bottleneck for current GUI proxies is basic visual positioning capability.
  arXiv  Detail & Related papers  (2025-08-12T15:12:42Z)
• MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning [83.81404871748438]
  MagicGUI is a foundational mobile GUI agent designed to address critical challenges in perception, grounding, and reasoning within real-world mobile GUI environments.<n>The framework is underpinned by six key components, including a comprehensive and accurate dataset, enhanced perception and grounding capabilities, a comprehensive and unified action space, and planning-oriented reasoning mechanisms.
  arXiv  Detail & Related papers  (2025-07-19T12:33:43Z)
• GTA1: GUI Test-time Scaling Agent [77.60727242084971]
  This paper investigates the two main challenges with our GUI Test-time Scaling Agent, GTA1.<n>First, to select the most appropriate action proposal, we introduce a test-time scaling method.<n>Second, we propose a model that achieves improved accuracy when grounding the selected action proposal to its corresponding visual elements.
  arXiv  Detail & Related papers  (2025-07-08T08:52:18Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning [63.599057862999]
  RefChartQA is a novel benchmark that integrates Chart Question Answering (ChartQA) with visual grounding.<n>Our experiments demonstrate that incorporating spatial awareness via grounding improves response accuracy by over 15%.
  arXiv  Detail & Related papers  (2025-03-29T15:50:08Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  Aguvis is a vision-based framework for autonomous GUI agents.<n>It standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.<n>It achieves state-of-the-art performance across offline and real-world online benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent [80.50062396585004]
  Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. We develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations. ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding.
  arXiv  Detail & Related papers  (2024-11-26T14:29:47Z)
• Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces [1.3107174618549584]
  Instruction Visual Grounding (IVG) is a multi-modal approach to object identification within a Graphical User Interface (GUI)<n>We propose IVGocr, which combines a Large Language Model (LLM), an object detection model, and an Optical Character Recognition (OCR) module; and IVGdirect, which uses a multimodal architecture for end-to-end grounding.<n>Our final test dataset is publicly released to support future research.
  arXiv  Detail & Related papers  (2024-05-05T19:10:19Z)
• An Extensible Dashboard Architecture For Visualizing Base And Analyzed Data [2.169919643934826]
  This paper focuses on an architecture for visualization of base as well as analyzed data. This paper proposes a modular architecture of a dashboard for user-interaction, visualization management, and complex analysis of base data.
  arXiv  Detail & Related papers  (2021-06-09T19:45:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.