MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI

• URL: http://arxiv.org/abs/2603.19993v1
• Date: Fri, 20 Mar 2026 14:43:53 GMT
• Title: MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
• Abstract summary: We introduce MedSPOT, a workflow-aware sequential grounding benchmark for clinical GUI environments.<n> benchmark comprises 216 task-driven videos with 597 annotateds, in which each task consists of 2 to 3 interdependent grounding steps.<n>We also introduce a comprehensive failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion.
• Score: 0.7552557021953206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the rapid progress of Multimodal Large Language Models (MLLMs), their ability to perform reliable visual grounding in high-stakes clinical software environments remains underexplored. Existing GUI benchmarks largely focus on isolated, single-step grounding queries, overlooking the sequential, workflow-driven reasoning required in real-world medical interfaces, where tasks evolve across independent steps and dynamic interface states. We introduce MedSPOT, a workflow-aware sequential grounding benchmark for clinical GUI environments. Unlike prior benchmarks that treat grounding as a standalone prediction task, MedSPOT models procedural interaction as a sequence of structured spatial decisions. The benchmark comprises 216 task-driven videos with 597 annotated keyframes, in which each task consists of 2 to 3 interdependent grounding steps within realistic medical workflows. This design captures interface hierarchies, contextual dependencies, and fine-grained spatial precision under evolving conditions. To evaluate procedural robustness, we propose a strict sequential evaluation protocol that terminates task assessment upon the first incorrect grounding prediction, explicitly measuring error propagation in multi-step workflows. We further introduce a comprehensive failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion, to enable systematic diagnosis of model behavior in clinical GUI settings. By shifting evaluation from isolated grounding to workflow-aware sequential reasoning, MedSPOT establishes a realistic and safety-critical benchmark for assessing multimodal models in medical software environments. Code and data are available at: https://github.com/Tajamul21/MedSPOT.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• UniETP: Unifying Environments for Generalizable Embodied Task Planning [42.65363241295469]
  Embodied Task Planning is where an agent is required to execute a sequence of atomic actions within an interactive environment to complete a user-specified task.<n>UniETP is characterized by both standardization and diversity. On one hand, it formalizes all the simulators into a consistent observation and action space, and builds an evaluation system to support complicated task goal.
  arXiv  Detail & Related papers  (2026-07-20T15:30:16Z)
• Towards Autonomous and Auditable Medical Imaging Model Development [53.326510777116766]
  We introduce AMID, an autonomous multi-agent framework for medical imaging model development.<n>AMID first proposes Data-Conditioned Method Planning, which refines coarse task-level search spaces into executable, parallelizable method lanes.<n>It then develops Verification-Guided Two-Stage Optimization, moving from broad early exploration of diverse method lanes to selective exploitation of promising candidates.
  arXiv  Detail & Related papers  (2026-07-12T00:54:06Z)
• Adaptive Multimodal Agents-Based Framework for Automatic Workflow Execution [1.2132786136735432]
  We propose a novel multi-agent framework that achieves automatic workflow execution through a distinct two-phase pipeline.<n>During inference, agents leverage Adaptive Retrieval-Augmented Generation (RAG) over this fixed, pre-established graph.<n>We validate our framework in a real-world context, demonstrating its ability to maintain high reliability and semantic awareness even with limited training data.
  arXiv  Detail & Related papers  (2026-05-27T15:23:22Z)
• Towards a Virtual Neuroscientist: Autonomous Neuroimaging Analysis via Multi-Agent Collaboration [53.772014300855375]
  We introduce NIAgent, a multi-agent system for autonomous end-to-end neuroimaging analysis.<n>Unlike conventional flat tool-calling agents, NIAgent adopts a code-centric execution paradigm.<n>We propose a hierarchical verification framework for autonomous quality control, integrating cohort-level metric screening with agentic visual inspection.
  arXiv  Detail & Related papers  (2026-05-10T06:30:19Z)
• Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling [24.896312231457653]
  We develop a framework to profile and handle missing modalities while addressing the canonical desideratum of safe, transparent clinical AI.<n>We show that autoregressive sequence modeling with transformer-based architectures outperforms baselines on the MIMIC-IV and eICU fine-tuning benchmarks.
  arXiv  Detail & Related papers  (2026-04-20T18:55:56Z)
• CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare [37.42599407869901]
  Multimodal agentic pipelines are transforming human-computer interaction by enabling efficient and accessible automation of complex, real-world tasks.<n>Recent efforts have focused on short-horizon or general-purpose applications, leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored.<n>We introduce CarePilot, a multi-agent framework based on the actor-critic paradigm.<n>Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38%, respectively.
  arXiv  Detail & Related papers  (2026-03-25T10:25:48Z)
• Data-Local Autonomous LLM-Guided Neural Architecture Search for Multiclass Multimodal Time-Series Classification [0.4925906256430175]
  LLM-guided neural architecture search (NAS) can automate this exploration, but most assume cloud execution or access to data-derived artifacts that cannot be exposed.<n>We present a novel data-local, LLM-guided search framework that handles candidate pipelines remotely while executing all training and evaluation locally under a fixed protocol.<n>Our framework targets multiclass, multimodal learning via one-vs-rest binary experts per class and modality, a lightweight fusion, and joint search over expert architectures and modality-specific preprocessing.
  arXiv  Detail & Related papers  (2026-03-16T21:46:05Z)
• AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild [30.138230316314534]
  We introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction to bidirectional intent alignment.<n>We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols.<n>We also develop MUSE, an automated framework utilizing an MLLM-as-a-judge multi-agent architecture.
  arXiv  Detail & Related papers  (2026-02-12T09:25:15Z)
• SUNSET -- A Sensor-fUsioN based semantic SegmEnTation exemplar for ROS-based self-adaptation [2.8210523386413087]
  We present SUNSET, a ROS2-based exemplar that enables rigorous, repeatable evaluation of architecture-based self-adaptation.<n> SUNSET includes the segmentation pipeline, a trained ML model, uncertainty-injection scripts, a baseline controller, and step-by-step integration and evaluation documentation.
  arXiv  Detail & Related papers  (2026-01-20T08:40:57Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• Clairvoyance: A Pipeline Toolkit for Medical Time Series [95.22483029602921]
  Time-series learning is the bread and butter of data-driven *clinical decision support* Clairvoyance proposes a unified, end-to-end, autoML-friendly pipeline that serves as a software toolkit. Clairvoyance is the first to demonstrate viability of a comprehensive and automatable pipeline for clinical time-series ML.
  arXiv  Detail & Related papers  (2023-10-28T12:08:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.