3D Printed Brain-Controlled Robot-Arm Prosthetic via Embedded Deep Learning from sEMG Sensors

• URL: http://arxiv.org/abs/2005.01797v1
• Date: Mon, 4 May 2020 19:14:44 GMT
• Title: 3D Printed Brain-Controlled Robot-Arm Prosthetic via Embedded Deep Learning from sEMG Sensors
• Authors: David Lonsdale, Li Zhang and Richard Jiang
• Abstract summary: Our work proposes to use transfer learning techniques applied to the Google Inception model to retrain the final layer for surface electromyography (sEMG) classification. Data have been collected using the Thalmic Labs Myo Armband and used to generate graph images comprised of 8 subplots per image. Deep learning model, Inception-v3, with transfer learning to train the model for accurate prediction of each on real-time input of new data. Brain-controlled robot arm was produced using a 3D printer and off-the-shelf hardware to control it.
• Score: 4.901124285608471
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we present our work on developing robot arm prosthetic via deep learning. Our work proposes to use transfer learning techniques applied to the Google Inception model to retrain the final layer for surface electromyography (sEMG) classification. Data have been collected using the Thalmic Labs Myo Armband and used to generate graph images comprised of 8 subplots per image containing sEMG data captured from 40 data points per sensor, corresponding to the array of 8 sEMG sensors in the armband. Data captured were then classified into four categories (Fist, Thumbs Up, Open Hand, Rest) via using a deep learning model, Inception-v3, with transfer learning to train the model for accurate prediction of each on real-time input of new data. This trained model was then downloaded to the ARM processor based embedding system to enable the brain-controlled robot-arm prosthetic manufactured from our 3D printer. Testing of the functionality of the method, a robotic arm was produced using a 3D printer and off-the-shelf hardware to control it. SSH communication protocols are employed to execute python files hosted on an embedded Raspberry Pi with ARM processors to trigger movement on the robot arm of the predicted gesture.

Related papers

• SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding [78.12178144115224]
  Robotic Foundation Models (RFMs) hold great promise as generalist, end-to-end systems for robot control.<n>We propose to enrich easy-to-collect non-robotic image data with 3D annotations and enhance a pretrained VLM with 3D understanding capabilities.<n>We introduce our main contribution, $textbfSPEAR-1$: a robotic foundation model that integrates grounded 3D perception with language-instructed embodied control.
  arXiv  Detail & Related papers  (2025-11-21T17:09:43Z)
• RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph [62.270763554624615]
  Estimating robot pose from a monocular RGB image is a challenge in robotics and computer vision.<n>Existing methods typically build networks on top of 2D visual backbones and depend heavily on labeled data for training.<n>We propose Robot Topological Alignment Graph (RoboTAG), which incorporates a 3D branch to inject 3D priors while enabling co-evolution of the 2D and 3D representations.
  arXiv  Detail & Related papers  (2025-11-11T00:49:15Z)
• ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrations [32.570602111692914]
  We present ActiveUMI, a framework for a data collection system that transfers in-the-wild human demonstrations to robots capable of complex bimanual manipulation.<n>ActiveUMI couples a portable VR teleoperation kit with sensorized controllers that mirror the robot's end-effectors.<n>By recording an operator's deliberate head movements via a head-mounted display, our system learns the crucial link between visual attention and manipulation.
  arXiv  Detail & Related papers  (2025-10-02T02:44:21Z)
• Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging [17.607640140471936]
  "Digital twins" are useful for simulations, virtual reality, marketing, robot policy fine-tuning, and part inspection.<n>We propose Omni-Scan, a pipeline for producing high-quality 3D Gaussian Splat models using a bi-manual robot that grasps an object with one gripper and rotates the object with respect to a stationary camera.<n>We present the Omni-Scan robot pipeline using DepthAny-thing, Segment Anything, as well as RAFT optical flow models to identify and isolate objects held by a robot gripper while removing the gripper and the background.
  arXiv  Detail & Related papers  (2025-08-01T06:36:19Z)
• 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model [40.730112146035076]
  A key reason is the lack of a large and uniform dataset for teaching robots manipulation skills.<n>Current robot datasets often record robot action in different action spaces within a simple scene.<n>We learn a 3D flow world model from both human and robot manipulation data.
  arXiv  Detail & Related papers  (2025-06-06T16:00:31Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos. VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Pre-training Auto-regressive Robotic Models with 4D Representations [43.80798244473759]
  ARM4R is an Auto-regressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better pre-trained robotic model. Our experiments show that ARM4R can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations.
  arXiv  Detail & Related papers  (2025-02-18T18:59:01Z)
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [51.49400490437258]
  This work develops a method for imitating articulated object manipulation from a single monocular RGB human demonstration. We first propose 4D Differentiable Part Models (4D-DPM), a method for recovering 3D part motion from a monocular video. Given this 4D reconstruction, the robot replicates object trajectories by planning bimanual arm motions that induce the demonstrated object part motion. We evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part trajectories and RSRD's physical execution performance on 9 objects across 10 trials each on a bimanual YuMi robot.
  arXiv  Detail & Related papers  (2024-09-26T17:57:16Z)
• 3D-MVP: 3D Multiview Pretraining for Robotic Manipulation [53.45111493465405]
  We propose 3D-MVP, a novel approach for 3D multi-view pretraining using masked autoencoders. We leverage Robotic View Transformer (RVT), which uses a multi-view transformer to understand the 3D scene and predict pose actions. We show promising results on a real robot platform with minimal finetuning.
  arXiv  Detail & Related papers  (2024-06-26T08:17:59Z)
• Exploring 3D Human Pose Estimation and Forecasting from the Robot's Perspective: The HARPER Dataset [52.22758311559]
  We introduce HARPER, a novel dataset for 3D body pose estimation and forecast in dyadic interactions between users and Spot. The key-novelty is the focus on the robot's perspective, i.e., on the data captured by the robot's sensors. The scenario underlying HARPER includes 15 actions, of which 10 involve physical contact between the robot and users.
  arXiv  Detail & Related papers  (2024-03-21T14:53:50Z)
• Care3D: An Active 3D Object Detection Dataset of Real Robotic-Care Environments [52.425280825457385]
  This paper introduces an annotated dataset of real environments. The captured environments represent areas which are already in use in the field of robotic health care research. We also provide ground truth data within one room, for assessing SLAM algorithms running directly on a health care robot.
  arXiv  Detail & Related papers  (2023-10-09T10:35:37Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Grasping Student: semi-supervised learning for robotic manipulation [0.7282230325785884]
  We design a semi-supervised grasping system that takes advantage of images of products to be picked, which are collected without any interactions with the robot. In the regime of a small number of robot training samples, taking advantage of the unlabeled data allows us to achieve performance at the level of 10-fold bigger dataset size.
  arXiv  Detail & Related papers  (2023-03-08T09:03:11Z)
• Automatically Prepare Training Data for YOLO Using Robotic In-Hand Observation and Synthesis [14.034128227585143]
  We propose combining robotic in-hand observation and data synthesis to enlarge the limited data set collected by the robot. The collected and synthetic images are combined to train a deep detection neural network. The results showed that combined observation and synthetic images led to comparable performance to manual data preparation.
  arXiv  Detail & Related papers  (2023-01-04T04:20:08Z)
• From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation [26.738893736520364]
  We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. We construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-26T17:59:51Z)
• Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation [0.0]
  We propose a novel low-complexity lidar gesture recognition system for mobile robot control. The system is lightweight and suitable for mobile robot control with limited computing power. The use of lidar contributes to the robustness of the system, allowing it to operate in most outdoor conditions.
  arXiv  Detail & Related papers  (2021-09-17T00:49:31Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System [96.56564257199474]
  Training a robotic arm to accomplish real-world tasks has been attracting increasing attention in both academia and industry.<n>This work discusses the role of computer vision algorithms in this field.<n>We present an alternative solution, which uses a 3D model to create a large number of synthetic data.
  arXiv  Detail & Related papers  (2018-12-03T13:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.