RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation

• URL: http://arxiv.org/abs/2606.13040v1
• Date: Thu, 11 Jun 2026 08:20:42 GMT
• Title: RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation
• Abstract summary: Vision-language models (VLMs) are increasingly explored as visual critics, reward generators, and failure detectors in robotic manipulation.<n>Existing evaluations fail to test whether VLMs possess fine-grained process understanding.<n>We present RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation.
• Score: 59.10505906182782
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) are increasingly explored as visual critics, reward generators, and failure detectors in robotic manipulation. These roles implicitly require models to judge not only final task success, but also how a manipulation execution is physically and temporally progressing. However, existing evaluations fail to test whether VLMs possess fine-grained process understanding. To address this gap, we present RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation. RoboProcessBench decomposes such capability into two complementary dimensions, \emph{static monitoring} and \emph{dynamic reasoning}, instantiated as 12 diagnostic question families covering phase, contact, motion, coordination, primitive-local progress, temporal order, outcome, and primitive-level transitions. Built from physically grounded execution traces, the curated benchmark corpus ProcessData contains \textasciitilde 58k question-answer pairs across 260 manipulation tasks, which is further split into ProcessData-SFT and ProcessData-Eval for post-training and evaluation purposes. Extensive evaluation of various VLMs on ProcessData-Eval reveals broad limitations across 12 diagnostic task families, suggesting current models still lack robust process-aware understanding of manipulation executions. But with ProcessData-SFT, the post-trained \textit{Qwen2.5-VL-7B} and \textit{InternVL-3-8B} exhibit consistent gains on local state, motion, progress, and primitive-aware cues. These results demonstrate that RoboProcessBench serves as both an evaluation benchmark and a learnable supervision source for developing VLMs capable of monitoring and evaluating robotic manipulation processes. Project webpage: \href{https://processbench-2026.github.io/RoboProcessBench-Web/}{https://processbench-2026.github.io}.

Related papers

• RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks? [65.16829463527377]
  Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation.<n>We introduce textbfRoboSPA (textbfRobot textbfSpatial-textbfProcedural textbfAssessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning.
  arXiv  Detail & Related papers  (2026-09-04T16:19:01Z)
• FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation [43.89036551517797]
  Vision-Language-Action Models (VLAs) have demonstrated significant promise in generalizing to complex, long-horizon robotic manipulation tasks.<n>Their performance remains brittle, as they are typically trained on trajectory-monotonic, failure-free demonstrations.<n>We propose FLARE, a novel framework that endows VLAs with robust error recovery capabilities through a Retry" and Reset" paradigm.
  arXiv  Detail & Related papers  (2026-08-27T05:58:06Z)
• FacProcessTwin: An LLM-Based System for Process Twin Development [2.71939929570542]
  Process twins provide real-time representations of entire production processes.<n>We present FacProcessTwin, a system that builds a process twin from a plant's process documentation and natural-language input from an operator.<n>We evaluate FacProcessTwin through a real-world case study of an Australian food manufacturer.
  arXiv  Detail & Related papers  (2026-06-16T08:27:43Z)
• Employing Vision-Language Models for Face Image Quality Assessment [0.023141219541346198]
  Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines.<n>State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes"<n>We investigate the potential of off-the-shelf Vision-Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-05-17T14:57:52Z)
• ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation [25.420455584290465]
  Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages.<n>We present ProcVLM, a progress-aware vision-language model that learns procedure-grounded progress as a dense reward signal for manipulation.
  arXiv  Detail & Related papers  (2026-05-09T08:00:59Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining [56.62125584296097]
  Keyframe-Chaining VLA is a framework that extracts and links key historical frames to model long-horizon dependencies.<n>We design a progress-aware mechanism that dynamically retrieves historical frames based on their temporal relevance to the current execution phase.<n>We introduce a suite of four Non-Markovian manipulation tasks built upon the ManiSkill simulator to measure task success rates.
  arXiv  Detail & Related papers  (2026-03-02T05:26:29Z)
• AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act [27.922630781100864]
  We present AgenticLab, a model-agnostic robot agent platform and benchmark for open-world manipulation.<n>We benchmark state-of-the-art VLM-based agents on real-robot tasks in unstructured environments.<n>Our benchmark reveals several failure modes that offline vision-language tests fail to capture.
  arXiv  Detail & Related papers  (2026-02-02T05:30:14Z)
• TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks [12.573674060643787]
  TimeMachine-bench is a benchmark designed to evaluate software migration in real-world Python projects.<n>Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates.
  arXiv  Detail & Related papers  (2026-01-30T05:42:45Z)
• RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation [59.9896841079005]
  We introduce RoboCerebra, a benchmark for evaluating high-level reasoning in long-horizon robotic manipulation.<n>The dataset is constructed via a top-down pipeline, where GPT generates task instructions and decomposes them into subtask sequences.<n>Compared to prior benchmarks, RoboCerebra features significantly longer action sequences and denser annotations.
  arXiv  Detail & Related papers  (2025-06-07T06:15:49Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills [24.150758623016195]
  We present ManiSkill2, the next generation of the SAPIEN ManiSkill benchmark for generalizable manipulation skills. ManiSkill2 includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames. It defines a unified interface and evaluation protocol to support a wide range of algorithms. It empowers fast visual input learning algorithms so that a CNN-based policy can collect samples at about 2000 FPS.
  arXiv  Detail & Related papers  (2023-02-09T14:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.