SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

• URL: http://arxiv.org/abs/2603.29139v1
• Date: Tue, 31 Mar 2026 01:41:28 GMT
• Title: SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
• Abstract summary: We present SciVisAgentBench, a principled benchmark for evaluating scientific data analysis and visualization agents.<n>Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation.<n>It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios.
• Score: 12.966844873205048
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in large language models (LLMs) have enabled agentic systems that translate natural language intent into executable scientific visualization (SciVis) tasks. Despite rapid progress, the community lacks a principled and reproducible benchmark for evaluating these emerging SciVis agents in realistic, multi-step analysis settings. We present SciVisAgentBench, a comprehensive and extensible benchmark for evaluating scientific data analysis and visualization agents. Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation. It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios. To enable reliable assessment, we introduce a multimodal outcome-centric evaluation pipeline that combines LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. We also conduct a validity study with 12 SciVis experts to examine the agreement between human and LLM judges. Using this framework, we evaluate representative SciVis agents and general-purpose coding agents to establish initial baselines and reveal capability gaps. SciVisAgentBench is designed as a living benchmark to support systematic comparison, diagnose failure modes, and drive progress in agentic SciVis. The benchmark is available at https://scivisagentbench.github.io/.

Related papers

• AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers [39.39345650320786]
  We introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories.<n>We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol.<n>Results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.
  arXiv  Detail & Related papers  (2026-07-31T16:58:00Z)
• SciVisAgentSkills: Design and Evaluation of Agent Skills for Scientific Data Analysis and Visualization [8.855427243372116]
  We present SciVisAgentSkills, a collection of reusable agent skills that augment coding agents for scientific data analysis and visualization.<n>We evaluate these skills on Codex and Claude Code using SciVisAgentBench, a benchmark of 108 expert-designed multi-step tasks.
  arXiv  Detail & Related papers  (2026-06-04T00:14:25Z)
• SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models [63.357710580101894]
  Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice.<n>We propose a new framework named SciCustom to address the problem.<n>It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs.
  arXiv  Detail & Related papers  (2026-05-19T04:41:43Z)
• HeurekaBench: A Benchmarking Framework for AI Co-scientist [2.206319727896241]
  HeurekaBench is a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets.<n>We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents.<n>We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts.
  arXiv  Detail & Related papers  (2026-01-04T22:16:42Z)
• An Evaluation-Centric Paradigm for Scientific Visualization Agents [11.974221351149687]
  This position paper examines the various types of evaluation required for SciVis agents.<n>We advocate for a broader collaboration to develop a SciVis agentic evaluation benchmark.
  arXiv  Detail & Related papers  (2025-09-18T17:08:23Z)
• A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [251.23085679210206]
  Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
  arXiv  Detail & Related papers  (2025-08-28T18:30:52Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• BLADE: Benchmarking Language Model Agents for Data-Driven Science [21.682416167339635]
  LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science.<n>We present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions.
  arXiv  Detail & Related papers  (2024-08-19T02:59:35Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization [86.61052121715689]
  MatPlotAgent is a model-agnostic framework designed to automate scientific data visualization tasks. MatPlotBench is a high-quality benchmark consisting of 100 human-verified test cases.
  arXiv  Detail & Related papers  (2024-02-18T04:28:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.