Toward AI VIS Co-Scientists: A General and End-to-End Agent Harness for Solving Complex Data Visualization Tasks

• URL: http://arxiv.org/abs/2605.21825v1
• Date: Wed, 20 May 2026 23:49:28 GMT
• Title: Toward AI VIS Co-Scientists: A General and End-to-End Agent Harness for Solving Complex Data Visualization Tasks
• Abstract summary: We present an end-to-end agentic harness that independently designs custom visual analysis applications.<n>This represents an important step towards a general AI co-scientist envisioned by many as an autonomous system.<n>We validate this approach on IEEE SciVis Contests spanning multiple science and engineering fields.
• Score: 16.320569262892196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability to inspect, interpret, and communicate complex data is crucial for virtually any scientific endeavor, but often requires significant expertise outside the core domain ranging from data management and analysis to visualization design and implementation. We present an end-to-end agentic harness that, based on only the data and a high level description of the tasks, independently designs custom visual analysis applications (VIS apps). This represents an important step towards a general AI co-scientist envisioned by many as an autonomous system that can autonomously execute long horizon tasks based on high-level directions. Our proposed VIS co-scientist is an essential component of this broader AI co-scientist vision: a harness that can autonomously analyze data and design visualization solutions using a collection of agents and specialized skills that coordinate exploratory analysis, plan, configure the environment, implement, validate the interface, and most importantly evaluate the overall task completion. Each stage produces document and instruction artifacts that guide downstream work and enable iterative refinement. We validate this approach on IEEE SciVis Contests spanning multiple science and engineering fields. These contests serve as ideal proving grounds because they encode real-world complexity: ambiguous requirements, diverse data modalities, design trade-offs, and task-driven validation. Given only the data and target tasks, our system autonomously produces functional single-page VIS Apps with verified linked-view behavior, highly customized to domain experts' specified tasks and needs.

Related papers

• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe [52.210953881548996]
  We show that a vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks.<n>Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding.<n>Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks.
  arXiv  Detail & Related papers  (2026-07-17T13:25:44Z)
• Agentic Authoring of Interactive Multiview Visualizations in Genomics [10.74947096548303]
  Agentic and large language model (LLM) approaches are increasingly applied to complex scientific tasks.<n>Natural-language conversational interfaces offer a promising path to democratizing the authoring of complex visualizations.<n>These approaches face additional challenges: genomics visualizations integrate heterogeneous data types and are composed of multiple linked interactive views.
  arXiv  Detail & Related papers  (2026-05-29T21:22:16Z)
• AIDABench: AI Data Analytics Benchmark [62.45908988324612]
  AIDABench is a benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner.<n>AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation.<n>We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families.
  arXiv  Detail & Related papers  (2026-02-27T08:58:05Z)
• InferA: A Smart Assistant for Cosmological Ensemble Data [0.5130440339897478]
  InferA is a multi-agent system that enables scalable and efficient scientific data analysis.<n>At the core of the architecture is a supervisor agent that orchestrates a team of specialized agents responsible for distinct phases of the data retrieval and analysis.<n>To demonstrate the framework's usability, we evaluate the system using ensemble runs from the HACC cosmology simulation which comprises several terabytes.
  arXiv  Detail & Related papers  (2025-10-14T18:47:22Z)
• CoDA: Agentic Systems for Collaborative Data Visualization [57.270599188947294]
  Deep research has revolutionized data analysis, yet data scientists still devote substantial time to manually crafting visualizations.<n>Existing approaches, including simple single- or multi-agent systems, often oversimplify the task.<n>We introduce CoDA, a multi-agent system that employs specialized LLM agents for metadata analysis, task planning, code generation, and self-reflection.
  arXiv  Detail & Related papers  (2025-10-03T17:30:16Z)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks [94.19506319646376]
  We introduce Agent-X, a benchmark for evaluating vision-centric agents in real-world, multimodal settings.<n>Agent-X features 828 agentic tasks with authentic visual contexts, including images, multi-image comparisons, videos, and instructional text.<n>Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle to solve multi-step vision tasks.
  arXiv  Detail & Related papers  (2025-05-30T17:59:53Z)
• VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use [74.39058448757645]
  We present VipAct, an agent framework that enhances vision-language models (VLMs) VipAct consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks. We evaluate VipAct on benchmarks featuring a diverse set of visual perception tasks, with experimental results demonstrating significant performance improvements.
  arXiv  Detail & Related papers  (2024-10-21T18:10:26Z)
• AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Data [14.328402787379538]
  We introduce AGENTiGraph (Adaptive Generative ENgine for Task-based Interaction and Graphical Representation), a platform for knowledge management through natural language interaction. AGENTiGraph employs a multi-agent architecture to dynamically interpret user intents, manage tasks, and integrate new knowledge. Experimental results on a dataset of 3,500 test cases show AGENTiGraph significantly outperforms state-of-the-art zero-shot baselines.
  arXiv  Detail & Related papers  (2024-10-15T12:05:58Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour [6.716560115378451]
  We introduce a modular, flexible, yet user-friendly software framework specifically developed to streamline computational-driven data exploration for human behavior analysis. Our primary objective is to democratize access to advanced computational methodologies, thereby enabling researchers across disciplines to engage in detailed behavioral analysis without the need for extensive technical proficiency.
  arXiv  Detail & Related papers  (2024-07-18T11:28:52Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.