Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents

• URL: http://arxiv.org/abs/2608.25417v1
• Date: Wed, 26 Aug 2026 06:19:28 GMT
• Title: Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents
• Abstract summary: We identify a critical yet underexplored capability within this space - dexterous visual tool use.<n>Existing benchmarks cover web navigation, GUI operation, and software engineering.<n>We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use.
• Score: 14.940946245813336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.

Related papers

• Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models [74.1961755473754]
  Existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations.<n>We propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations.<n>We show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results.
  arXiv  Detail & Related papers  (2026-08-10T14:42:10Z)
• MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents [50.27505102982638]
  We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents.<n>The framework provides a stateful execution environment spanning 500+ tools across 16 application domains.<n>It supports multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls.
  arXiv  Detail & Related papers  (2026-07-13T17:13:09Z)
• VESTA: Visual Exploration with Statistical Tool Agents [56.76643750765825]
  We introduce VESTA: Visual Exploration with Statistical Tool Agents.<n>We show that VESTA's dynamic tool creation outperforms prior agentic pipelines.<n>We also introduce DAWN, a benchmark targeting distribution fitting and time series.
  arXiv  Detail & Related papers  (2026-05-29T21:57:37Z)
• Employing Vision-Language Models for Face Image Quality Assessment [0.023141219541346198]
  Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines.<n>State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes"<n>We investigate the potential of off-the-shelf Vision-Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-05-17T14:57:52Z)
• ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation [14.85015534787527]
  Interleaved text-and-image generation offers a more intuitive way to convey complex information.<n>Current paradigms rely on either image generation or retrieval augmentation, failing to unify factuality with creativity.<n>We argue that the next milestone in this field is Agentic Tool Planning, where the model serves as a central controller that autonomously determines when, where, and which tools to invoke to produce interleaved responses for visual-critical queries.
  arXiv  Detail & Related papers  (2026-03-31T15:47:59Z)
• AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesis [30.512393568258105]
  Large Language Model agents demonstrate potential in solving real-world problems via tools, yet generalist intelligence is bottlenecked by scarce high-quality, long-horizon data.<n>We propose AgentSkiller, a fully automated framework synthesizing multi-turn interaction data across realistic, semantically linked domains.
  arXiv  Detail & Related papers  (2026-02-10T03:21:42Z)
• Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance [92.72779885657373]
  We propose a framework that grounds model selection in the internal functional dynamics of the visual encoder.<n>Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment.<n>Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks.
  arXiv  Detail & Related papers  (2026-02-01T17:29:43Z)
• ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement [58.957050610762565]
  ShowTable is a pipeline that synergizes MLLMs with diffusion models via a progressive self-correcting process.<n> MLLM acts as the central orchestrator for reasoning the visual plan and judging visual errors.<n>We introduce TableVisBench, a new benchmark with 800 challenging instances across 5 evaluation dimensions.
  arXiv  Detail & Related papers  (2025-12-15T13:21:50Z)
• ASTRA: A Scene-aware TRAnsformer-based model for trajectory prediction [15.624698974735654]
  ASTRA (A Scene-aware TRAnsformer-based model for trajectory prediction) is a light-weight pedestrian trajectory forecasting model.<n>We utilise a U-Net-based feature extractor, via its latent vector representation, to capture scene representations and a graph-aware transformer encoder for capturing social interactions.
  arXiv  Detail & Related papers  (2025-01-16T23:28:30Z)
• Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
  TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution. We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
  arXiv  Detail & Related papers  (2023-06-09T06:02:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.