LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratory

• URL: http://arxiv.org/abs/2607.23704v2
• Date: Wed, 29 Jul 2026 11:00:47 GMT
• Title: LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratory
• Abstract summary: We introduce LabRobFail, a failure-centric framework for learning and evaluating robotic failure analysis in chemical laboratories.<n>LabRobFail-Sim injects controllable failures at the control, physics, and semantic levels.<n>LabRobFail-Bench evaluates six capabilities spanning task understanding, failure detection, temporal localization, severity assessment, failure classification, and actionable correction.
• Score: 52.26893354670174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The deployment of embodied agents in self-driving laboratories could accelerate scientific discovery, yet their reliability is constrained by the irreversible and safety-critical nature of chemical experiments. Progress is further hindered by scarce failure data and the lack of fine-grained evaluation protocols. To address these challenges, we introduce LabRobFail, a failure-centric framework for learning and evaluating robotic failure analysis in chemical laboratories. LabRobFail-Sim injects controllable failures at the control, physics, and semantic levels, enabling the construction of LabRobFail-Data, which contains over 20,000 trajectories across 70+ task scenarios, five failure categories, and 11 fine-grained failure types. LabRobFail-Bench evaluates six capabilities spanning task understanding, failure detection, temporal localization, severity assessment, failure classification, and actionable correction. We further develop LabRobFail-VLM, a domain-specialized vision-language model that generates structured failure diagnoses and recovery instructions. On seen environments, it achieves 90.83% failure-detection accuracy and 77.21% temporal-localization accuracy, substantially outperforming general-purpose VLMs. When integrated as a real-time supervisor, it improves downstream task success rates by 4-16 percentage points, demonstrating the value of fine-grained failure understanding for closed-loop recovery and reliable laboratory autonomy. Our code and data are available at https://github.com/Su-ISE-2001/SciRobo

Related papers

• LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories [40.311566677309536]
  LabDex is a large-scale real-world dataset and benchmark for dexterous manipulation in chemistry laboratories.<n>It is organized around a hierarchical task taxonomy spanning atomic skills, compositional tasks, and long-horizon experiments.<n>We conduct cross-level evaluations of representative robot learning methods in both real-world and simulation environments.
  arXiv  Detail & Related papers  (2026-08-19T07:11:06Z)
• SAFE-CHEM: Uncertainty-Aware Policy Switching for Robust Robotic Chemistry [0.47248250311484113]
  SAFE-CHEM is an uncertainty-aware framework designed for robust, learning-based robotic chemists.<n>Our approach leverages an ensemble of recurrent neural network-based imitation learning policies to quantify uncertainty online.<n>We introduce a hybrid control architecture that autonomously switches from the learned policy to a deterministic, rule-based backup controller.
  arXiv  Detail & Related papers  (2026-08-10T08:51:43Z)
• LabEvolver: Training-Free Experience Evolution for Safe and Grounded Wet-Lab Agents [55.791551822756524]
  We introduce LabEvolver, a training-free framework that equips safe and grounded wet-lab agents with episodic memory from execution experience.<n>LabEvolver couples a state-grounded inner trial loop for adaptive perception, online planning, and safety validation with an outer evolution loop that distills completed trajectories into reusable skill, strategy, and safety experience.
  arXiv  Detail & Related papers  (2026-07-30T05:19:33Z)
• Labimus: A Simulation and Benchmark for Humanoid Dexterous Manipulation in Chemical Laboratory [47.77200587195896]
  Labimus is the first benchmark for humanoid dexterous manipulation in organic chemistry laboratories.<n>It reconstructs over 30 functionally faithful assets from real organic chemistry workstations through real-to-sim modeling.<n>We introduce a precision-aware evaluation protocol designed to jointly measure task completion, experimental precision, and long-horizon execution.
  arXiv  Detail & Related papers  (2026-06-30T02:14:28Z)
• LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories [95.20367571157679]
  Vision-Language-Action (VLA) models provide one possible interface between written protocols and robot execution.<n>Existing policies are trained mostly on household and tabletop demonstrations and rarely encounter the instruments, transparent liquids, or fixed protocol found in scientific laboratories.<n>We present LabVLA, trained with a two-stage recipe: FAST action token pretraining first makes the Qwen3-VL-4B-Instruct backbone action aware before any continuous control is learned, and flow matching posttraining then attaches a DiT action expert under knowledge insulation.
  arXiv  Detail & Related papers  (2026-06-11T17:03:53Z)
• LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories [41.392364324753224]
  multimodal large language model (MLLM) agents evolve from lab assistants into self-driving lab operators.<n>We introduce LABSHIELD, a realistic multi-view benchmark designed to assess MLLMs in hazard identification and safety-critical reasoning.<n>We evaluate 20 proprietary models, 9 open-source models, and 3 embodied models under a dual-track evaluation framework.
  arXiv  Detail & Related papers  (2026-03-12T14:38:13Z)
• RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation [68.7948300643741]
  We propose Robot-Conditioned Normalizing Flow (RC-NF), a real-time monitoring model for robotic anomaly detection and intervention.<n>RC-NF decouples the processing of task-aware robot and object states within the normalizing flow.<n>It achieves state-of-the-art performance across all anomaly types compared to previous methods in monitoring robotic tasks.
  arXiv  Detail & Related papers  (2026-03-11T10:14:37Z)
• LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents [103.65422553044816]
  LabUtopia is a comprehensive simulation and benchmarking suite designed to facilitate the development of generalizable, reasoning-capable embodied agents.<n>It supports 30 distinct tasks and includes more than 200 scene and instrument assets.<n>We demonstrate that LabUtopia offers a powerful platform for advancing the integration of perception, planning, and control in scientific-purpose agents.
  arXiv  Detail & Related papers  (2025-05-28T17:50:53Z)
• LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs [78.99703366417661]
  Large language models (LLMs) increasingly assist in tasks ranging from procedural guidance to autonomous experiment orchestration.<n>Such overreliance is particularly dangerous in high-stakes laboratory settings, where failures in hazard identification or risk assessment can result in severe accidents.<n>We propose the Laboratory Safety Benchmark (LabSafety Bench) to evaluate models on their ability to identify potential hazards, assess risks, and predict the consequences of unsafe actions in lab environments.
  arXiv  Detail & Related papers  (2024-10-18T05:21:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.