From Human Hands to Robotic Limbs: A Study in Motor Skill Embodiment for Telemanipulation

• URL: http://arxiv.org/abs/2502.02036v1
• Date: Tue, 04 Feb 2025 05:52:57 GMT
• Title: From Human Hands to Robotic Limbs: A Study in Motor Skill Embodiment for Telemanipulation
• Authors: Haoyi Shi, Mingxi Su, Ted Morris, Vassilios Morellas, Nikolaos Papanikolopoulos,
• Abstract summary: We propose a GRU-based Variational Autoencoder to learn a latent representation of the manipulator's configuration space. A fully connected neural network maps human arm configurations into this latent space, allowing the system to mimic and generate corresponding manipulator trajectories in real time.
• Score: 3.7482358401236398
• License:
• Abstract: This paper presents a teleoperation system for controlling a redundant degree of freedom robot manipulator using human arm gestures. We propose a GRU-based Variational Autoencoder to learn a latent representation of the manipulator's configuration space, capturing its complex joint kinematics. A fully connected neural network maps human arm configurations into this latent space, allowing the system to mimic and generate corresponding manipulator trajectories in real time through the VAE decoder. The proposed method shows promising results in teleoperating the manipulator, enabling the generation of novel manipulator configurations from human features that were not present during training.

Related papers

• Towards spiking analog hardware implementation of a trajectory interpolation mechanism for smooth closed-loop control of a spiking robot arm [41.54924235047016]
  We propose a closed-loop neuromorphic control system for an event-based robotic arm. The proposed system consists of a shifted Winner-Take-All spiking network and a spiking comparator network. To evaluate the system, we implemented and deployed the model on a mixed-signal analog-digital neuromorphic platform.
  arXiv  Detail & Related papers  (2025-01-23T14:11:32Z)
• Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition [48.65867987106428]
  We introduce a novel system for joint learning between human operators and robots. It enables human operators to share control of a robot end-effector with a learned assistive agent. It reduces the need for human adaptation while ensuring the collected data is of sufficient quality for downstream tasks.
  arXiv  Detail & Related papers  (2024-06-29T03:37:29Z)
• LPAC: Learnable Perception-Action-Communication Loops with Applications to Coverage Control [80.86089324742024]
  We propose a learnable Perception-Action-Communication (LPAC) architecture for the problem. CNN processes localized perception; a graph neural network (GNN) facilitates robot communications. Evaluations show that the LPAC models outperform standard decentralized and centralized coverage control algorithms.
  arXiv  Detail & Related papers  (2024-01-10T00:08:00Z)
• Naturalistic Robot Arm Trajectory Generation via Representation Learning [4.7682079066346565]
  Integration of manipulator robots in household environments suggests a need for more predictable human-like robot motion. One method of generating naturalistic motion trajectories is via imitation of human demonstrators. This paper explores a self-supervised imitation learning method using an autoregressive neural network for an assistive drinking task.
  arXiv  Detail & Related papers  (2023-09-14T09:26:03Z)
• Zero-Shot Robot Manipulation from Passive Human Videos [59.193076151832145]
  We develop a framework for extracting agent-agnostic action representations from human videos. Our framework is based on predicting plausible human hand trajectories. We deploy the trained model zero-shot for physical robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-02-03T21:39:52Z)
• Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance [71.36749876465618]
  We describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples. experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world.
  arXiv  Detail & Related papers  (2022-12-19T22:50:40Z)
• Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems [79.07468367923619]
  We propose a backpropagation-free approach to robotic control through the neuro-cognitive computational framework of neural generative coding (NGC) We design an agent built completely from powerful predictive coding/processing circuits that facilitate dynamic, online learning from sparse rewards. We show that our proposed ActPC agent performs well in the face of sparse (extrinsic) reward signals and is competitive with or outperforms several powerful backprop-based RL approaches.
  arXiv  Detail & Related papers  (2022-09-19T16:49:32Z)
• From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation [26.738893736520364]
  We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. We construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-26T17:59:51Z)
• A Transferable Legged Mobile Manipulation Framework Based on Disturbance Predictive Control [15.044159090957292]
  Legged mobile manipulation, where a quadruped robot is equipped with a robotic arm, can greatly enhance the performance of the robot. We propose a unified framework disturbance predictive control where a reinforcement learning scheme with a latent dynamic adapter is embedded into our proposed low-level controller.
  arXiv  Detail & Related papers  (2022-03-02T14:54:10Z)
• Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation [0.0]
  We propose a novel low-complexity lidar gesture recognition system for mobile robot control. The system is lightweight and suitable for mobile robot control with limited computing power. The use of lidar contributes to the robustness of the system, allowing it to operate in most outdoor conditions.
  arXiv  Detail & Related papers  (2021-09-17T00:49:31Z)
• Populations of Spiking Neurons for Reservoir Computing: Closed Loop Control of a Compliant Quadruped [64.64924554743982]
  We present a framework for implementing central pattern generators with spiking neural networks to obtain closed loop robot control. We demonstrate the learning of predefined gait patterns, speed control and gait transition on a simulated model of a compliant quadrupedal robot.
  arXiv  Detail & Related papers  (2020-04-09T14:32:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.