ABRA: Agent Benchmark for Radiology Applications

• URL: http://arxiv.org/abs/2605.11224v1
• Date: Mon, 11 May 2026 20:34:20 GMT
• Title: ABRA: Agent Benchmark for Radiology Applications
• Abstract summary: We introduce ABRA, a radiology-agent benchmark in which the agent operates a viewer and an Orthanc DICOM server.<n>ABRA contains 655 655 generated tasks across three difficulty tiers and eight types.<n>Each episode is scored along Planning, Execution, and Outcome (Blueth et al., 2025) by tasktype-specific automatic scorers.
• Score: 46.39696206872633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing medical-agent benchmarks deliver imaging as pre-selected samples, never as an environment the agent must navigate. We introduce ABRA, a radiology-agent benchmark in which the agent operates an OHIF viewer and an Orthanc DICOM server through twenty-one function-calling tools that span slice navigation, windowing, series selection, pixel-coordinate annotation, and structured reporting. ABRA contains 655 programmatically generated tasks across three difficulty tiers and eight types (viewer control, metadata QA, vision probe, annotation, longitudinal comparison, BI-RADS reporting, and oracle variants of annotation and BI-RADS reporting), drawn from LIDC-IDRI, Duke Breast Cancer MRI, and NLST New-Lesion LongCT. Each episode is scored along Planning, Execution, and Outcome (Bluethgen et al., 2025) by task-type-specific automatic scorers. Ten current models, five closed-weight and five open-weight, reach at least 89% Execution on real annotation but only 0-25% Outcome; on the paired oracle variant where a simulated detector supplies the finding, Outcome on the same task reaches 69-100% across the models evaluated, localising the bottleneck to perception rather than tool orchestration. Code, task generators, and scorers are released at https://github.com/Luab/ABRA

Related papers

• BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click [0.0]
  We present BekchiAI, a benchmark for measuring agentic skill and a platform for observing and controlling live agents.<n>The BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and telemetry as well as remote run termination.
  arXiv  Detail & Related papers  (2026-08-27T09:30:08Z)
• Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents [14.940946245813336]
  We identify a critical yet underexplored capability within this space - dexterous visual tool use.<n>Existing benchmarks cover web navigation, GUI operation, and software engineering.<n>We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use.
  arXiv  Detail & Related papers  (2026-08-26T06:19:28Z)
• AgentOmnia: Scaling Agentic Models for Full-Scenario Applications [36.656762500581216]
  Large language model agents have advanced rapidly, yet progress remains fragmented across domains, capabilities, task difficulty, and interaction settings.<n>We frame this as full-scenario agentic scaling and present Agent Omnia, a framework coordinating task-space definition, data synthesis, post-training, evaluation, and improvement across To-Consumer (ToC), To-Business (ToB), and To-Employee (ToE) applications.
  arXiv  Detail & Related papers  (2026-07-25T09:58:24Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models [2.3004655342211078]
  Large Language Models generate plausible backend code, but a single-pass paradigm provides no guarantee of correctness or runtime reliability.<n>We present ExecuGraph, a multi-agent framework that places execution-based validation at the center of backend code.<n>A per-agent ablation, retry-budget sweep, error-class taxonomy, and test-source audit are reported.
  arXiv  Detail & Related papers  (2026-06-18T05:58:17Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• TADI: Tool-Augmented Drilling Intelligence via Agentic LLM Orchestration over Heterogeneous Wellsite Data [0.0]
  TADI (Tool-Augmented Drilling Intelligence) is an agentic AI system that transforms drilling operational data into evidence-based analytical intelligence.<n>TADI integrates 1,759 daily drilling reports, selected WIT real-time objects, 15,634 production records, formation tops, and perforations into a dual-store architecture.
  arXiv  Detail & Related papers  (2026-04-30T03:19:39Z)
• CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend [2.9612444540570113]
  We present CUJBench, the first benchmark to combine browser-visible failure evidence with backend observability in a diagnostic framing.<n>The benchmark yields an overall accuracy of 19.7% with a ceiling of 52%, well below saturation.
  arXiv  Detail & Related papers  (2026-04-25T22:10:53Z)
• AgentRaft: Automated Detection of Data Over-Exposure in LLM Agents [40.88941407116349]
  We present AgentRaft, the first automated framework for detecting Data Over-Exposure (DOE) risks in Large Language Model (LLM) agents.<n>We evaluate AgentRaft on a testing environment of 6,675 real-world agent tools.
  arXiv  Detail & Related papers  (2026-03-08T09:40:54Z)
• BUSTR: Breast Ultrasound Text Reporting with a Descriptor-Aware Vision-Language Model [0.0]
  Automated radiology report generation (RRG) for breast ultrasound (BUS) is limited by the lack of paired image-report datasets.<n>We propose BUSTR, a multitask vision-language framework that generates BUS reports without requiring paired image-report supervision.
  arXiv  Detail & Related papers  (2025-11-26T01:22:29Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning [68.00304954972232]
  Multimodal agents, which integrate a controller e.g., a vision language model, with external tools, have demonstrated remarkable capabilities in tackling complex multimodal tasks.<n>Existing approaches for training these agents depend on extensive human-annotated task-answer pairs and tool trajectories.<n>We propose an iterative tool usage exploration method for multimodal agents without any pre-collected data, namely SPORT.<n>SPORT has four iterative components: task synthesis, step sampling, step verification, and preference tuning.
  arXiv  Detail & Related papers  (2025-04-30T12:01:27Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• How Well Can Modern LLMs Act as Agent Cores in Radiology Environments? [54.36730060680139]
  RadA-BenchPlat is an evaluation platform that benchmarks the performance of large language models (LLMs) in radiology environments.<n>The platform also defines ten categories of tools for agent-driven task solving and evaluates seven leading LLMs.
  arXiv  Detail & Related papers  (2024-12-12T18:20:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.