In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators

• URL: http://arxiv.org/abs/2410.19277v1
• Date: Fri, 25 Oct 2024 03:10:42 GMT
• Title: In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators
• Authors: Dmytro Humeniuk, Houssem Ben Braiek, Thomas Reid, Foutse Khomh,
• Abstract summary: Testing autonomous robotic manipulators is challenging due to the complex software interactions between vision and control components. A crucial element of modern robotic manipulators is the deep learning based object detection model. We propose the MARTENS framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios.
• Score: 11.389756788049944
• License:
• Abstract: Testing autonomous robotic manipulators is challenging due to the complex software interactions between vision and control components. A crucial element of modern robotic manipulators is the deep learning based object detection model. The creation and assessment of this model requires real world data, which can be hard to label and collect, especially when the hardware setup is not available. The current techniques primarily focus on using synthetic data to train deep neural networks (DDNs) and identifying failures through offline or online simulation-based testing. However, the process of exploiting the identified failures to uncover design flaws early on, and leveraging the optimized DNN within the simulation to accelerate the engineering of the DNN for real-world tasks remains unclear. To address these challenges, we propose the MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the deep learning vision model and uncovering system design flaws. Evaluation of two industrial case studies demonstrated that MARTENS effectively reveals robotic manipulator system failures, detecting 25 % to 50 % more failures with greater diversity compared to random test generation. The model trained and repaired using the MARTENS approach achieved mean average precision (mAP) scores of 0.91 and 0.82 on real-world images with no prior retraining. Further fine-tuning on real-world images for a few epochs (less than 10) increased the mAP to 0.95 and 0.89 for the first and second use cases, respectively. In contrast, a model trained solely on real-world data achieved mAPs of 0.8 and 0.75 for use case 1 and use case 2 after more than 25 epochs.

Related papers

• Accelerating Domain-Aware Electron Microscopy Analysis Using Deep Learning Models with Synthetic Data and Image-Wide Confidence Scoring [0.0]
  We create a physics-based synthetic image and data generator, resulting in a machine learning model that achieves comparable precision (0.86), recall (0.63), F1 scores (0.71), and engineering property predictions (R2=0.82) Our study demonstrates that synthetic data can eliminate human reliance in ML and provides a means for domain awareness in cases where many feature detections per image are needed.
  arXiv  Detail & Related papers  (2024-08-02T20:15:15Z)
• Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap [6.393953433174051]
  We propose a novel perspective for addressing the real-to-simulated data gap. We conduct the first large-scale investigation into the real-to-simulated data gap in an autonomous driving setting. Our results show notable improvements in model robustness to simulated data, even improving real-world performance in some cases.
  arXiv  Detail & Related papers  (2024-03-24T11:09:41Z)
• VR-LENS: Super Learning-based Cybersickness Detection and Explainable AI-Guided Deployment in Virtual Reality [1.9642496463491053]
  This work presents an explainable artificial intelligence (XAI)-based framework VR-LENS for developing cybersickness detection ML models. We first develop a novel super learning-based ensemble ML model for cybersickness detection. Our proposed method identified eye tracking, player position, and galvanic skin/heart rate response as the most dominant features for the integrated sensor, gameplay, and bio-physiological datasets.
  arXiv  Detail & Related papers  (2023-02-03T20:15:51Z)
• Towards Precise Model-free Robotic Grasping with Sim-to-Real Transfer Learning [11.470950882435927]
  We present an end-to-end robotic grasping network with a grasp. In physical robotic experiments, our grasping framework grasped single known objects and novel complex-shaped household objects with a success rate of 90.91%. The proposed grasping framework outperformed two state-of-the-art methods in both known and unknown object robotic grasping.
  arXiv  Detail & Related papers  (2023-01-28T16:57:19Z)
• Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly [55.41644538483948]
  We present an industrial application case that uses sim2real transfer learning for point cloud data. We provide insights on how to generate and process synthetic point cloud data. A novel patch-based attention network is proposed additionally to tackle this problem.
  arXiv  Detail & Related papers  (2023-01-12T14:00:37Z)
• Real-to-Sim: Predicting Residual Errors of Robotic Systems with Sparse Data using a Learning-based Unscented Kalman Filter [65.93205328894608]
  We learn the residual errors between a dynamic and/or simulator model and the real robot. We show that with the learned residual errors, we can further close the reality gap between dynamic models, simulations, and actual hardware.
  arXiv  Detail & Related papers  (2022-09-07T15:15:12Z)
• ProcTHOR: Large-Scale Embodied AI Using Procedural Generation [55.485985317538194]
  ProcTHOR is a framework for procedural generation of Embodied AI environments. We demonstrate state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation.
  arXiv  Detail & Related papers  (2022-06-14T17:09:35Z)
• Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin-picking [98.5984733963713]
  We propose an iterative self-training framework for sim-to-real 6D object pose estimation to facilitate cost-effective robotic grasping. We establish a photo-realistic simulator to synthesize abundant virtual data, and use this to train an initial pose estimation network. This network then takes the role of a teacher model, which generates pose predictions for unlabeled real data.
  arXiv  Detail & Related papers  (2022-04-14T15:54:01Z)
• Robot Learning from Randomized Simulations: A Review [59.992761565399185]
  Deep learning has caused a paradigm shift in robotics research, favoring methods that require large amounts of data. State-of-the-art approaches learn in simulation where data generation is fast as well as inexpensive. We focus on a technique named 'domain randomization' which is a method for learning from randomized simulations.
  arXiv  Detail & Related papers  (2021-11-01T13:55:41Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.