Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment

• URL: http://arxiv.org/abs/2202.05200v2
• Date: Fri, 11 Feb 2022 03:57:39 GMT
• Title: Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment
• Abstract summary: This letter presents a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing. A convolutional neural network is trained to predict the actuations required to achieve the desired pose. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera.
• Score: 9.599112149531557
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: For soft continuum arms, visual servoing is a popular control strategy that relies on visual feedback to close the control loop. However, robust visual servoing is challenging as it requires reliable feature extraction from the image, accurate control models and sensors to perceive the shape of the arm, both of which can be hard to implement in a soft robot. This letter circumvents these challenges by presenting a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing using a camera mounted at the distal end of the arm. A convolutional neural network is trained to predict the actuations required to achieve the desired pose in a structured environment. Integrated and modular approaches for estimating the actuations from the image are proposed and are experimentally compared. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera. The model together with the proportional feedback control makes the described approach robust to several variations such as new targets, lighting, loads, and diminution of the soft arm. Furthermore, the model lends itself to be transferred to a new environment with minimal effort.

Related papers

• EmbodiedVAE: Disentangled Video VAE for Efficient and Controllable Embodied Manipulation [35.20482569112933]
  We present EmbodiedVAE, a novel video VAE that provides compact yet controllable latent representations tailored for robotic manipulation world models.<n>Our proposed EmbodiedVAE achieves superior reconstruction quality with high compression rate, while enabling more precise action control in robotic manipulation scenarios with an average of 2dB PSNR improvement over state-of-the-art video VAEs.
  arXiv  Detail & Related papers  (2026-08-04T01:01:18Z)
• From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation [43.4347713269043]
  Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery.<n>We propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities.<n>Our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines.
  arXiv  Detail & Related papers  (2026-05-09T05:48:51Z)
• AI-Enabled Image-Based Hybrid Vision/Force Control of Tendon-Driven Aerial Continuum Manipulators [0.6999740786886535]
  This paper presents an AI-enabled cascaded hybrid vision/force control framework for tendon-driven aerial manipulators.<n>The proposed controller is designed to enable autonomous, physical interaction with a static environment while stabilizing the image feature error.
  arXiv  Detail & Related papers  (2026-04-21T01:19:11Z)
• Perception-Control Coupled Visual Servoing for Textureless Objects Using Keypoint-Based EKF [2.7711760478810157]
  We build upon learning-based keypoint detection for textureless objects and propose a method that enhances robustness.<n>We employ an Extended Kalman Filter (EKF) that integrates per-frame keypoint measurements to estimate 6D object pose.<n>Unlike standard PBVS, we propose a probabilistic control law that computes both camera velocity and its associated uncertainty.
  arXiv  Detail & Related papers  (2026-02-06T16:21:30Z)
• Dynamic Visual SLAM using a General 3D Prior [27.487134809452147]
  We propose a novel monocular visual SLAM system that can robustly estimate camera poses in dynamic scenes.<n> Specifically, we propose a feed-forward reconstruction model to precisely filter out dynamic regions.
  arXiv  Detail & Related papers  (2025-12-07T14:44:06Z)
• Exploring Conditions for Diffusion models in Robotic Control [70.27711404291573]
  We explore leveraging pre-trained text-to-image diffusion models to obtain task-adaptive visual representations for robotic control.<n>We find that naively applying textual conditions yields minimal or even negative gains in control tasks.<n>We propose ORCA, which introduces learnable task prompts that adapt to the control environment and visual prompts that capture fine-grained, frame-specific details.
  arXiv  Detail & Related papers  (2025-10-17T10:24:14Z)
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention [86.93601565563954]
  ScaleWeaver is a framework designed to achieve high-fidelity, controllable generation upon advanced visual autoregressive( VAR) models.<n>The proposed Reference Attention module discards the unnecessary attention from image$rightarrow$condition, reducing computational cost.<n>Experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods.
  arXiv  Detail & Related papers  (2025-10-16T17:00:59Z)
• Visual Odometry with Transformers [68.453547770334]
  We introduce Visual odometry Transformer (VoT), which processes sequences of monocular frames by extracting features.<n>Unlike prior methods, VoT directly predicts camera motion without estimating dense geometry and relies solely on camera poses for supervision.<n>VoT scales effectively with larger datasets, benefits substantially from stronger pre-trained backbones, generalizes across diverse camera motions and calibration settings, and outperforms traditional methods while running more than 3 times faster.
  arXiv  Detail & Related papers  (2025-10-02T17:00:14Z)
• PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control [67.17998939712326]
  We present PoseDiff, a conditional diffusion model that unifies robot state estimation and control within a single framework.<n>At its core, PoseDiff maps raw visual observations into structured robot states-such as 3D keypoints or joint angles-from a single RGB image.<n>Building upon this foundation, PoseDiff extends naturally to video-to-action inverse dynamics.
  arXiv  Detail & Related papers  (2025-09-29T10:55:48Z)
• PoseLess: Depth-Free Vision-to-Joint Control via Direct Image Mapping with VLM [0.0]
  PoseLess is a novel framework for robot hand control that eliminates the need for explicit pose estimation by directly mapping 2D images to joint angles using projected representations. Our approach leverages synthetic training data generated through randomized joint configurations, enabling zero-shot generalization to real-world scenarios and cross-morphology transfer from robotic to human hands.
  arXiv  Detail & Related papers  (2025-03-10T09:34:05Z)
• Neural Internal Model Control: Learning a Robust Control Policy via Predictive Error Feedback [16.46487826869775]
  We propose a novel framework, Neural Internal Model Control, which integrates model-based control with RL-based control to enhance robustness. Our framework streamlines the predictive model by applying Newton-Euler equations for rigid-body dynamics, eliminating the need to capture complex high-dimensional nonlinearities. We demonstrate the effectiveness of our framework on both quadrotors and quadrupedal robots, achieving superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-20T07:07:42Z)
• Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation [13.736566979493613]
  Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.
  arXiv  Detail & Related papers  (2024-09-23T10:38:20Z)
• Towards Robust and Expressive Whole-body Human Pose and Shape Estimation [51.457517178632756]
  Whole-body pose and shape estimation aims to jointly predict different behaviors of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We propose a novel framework to enhance the robustness of whole-body pose and shape estimation.
  arXiv  Detail & Related papers  (2023-12-14T08:17:42Z)
• Toward a Plug-and-Play Vision-Based Grasping Module for Robotics [0.0]
  This paper introduces a vision-based grasping framework that can easily be transferred across multiple manipulators. The framework generates diverse repertoires of open-loop grasping trajectories, enhancing adaptability while maintaining a diversity of grasps.
  arXiv  Detail & Related papers  (2023-10-06T16:16:00Z)
• Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation [59.3035531612715]
  Existing methods often struggle to generate plausible hand poses when the hand is heavily occluded or blurred. In videos, the movements of the hand allow us to observe various parts of the hand that may be occluded or blurred in a single frame. We propose the Deformer: a framework that implicitly reasons about the relationship between hand parts within the same image.
  arXiv  Detail & Related papers  (2023-03-09T02:24:30Z)
• Lateral Ego-Vehicle Control without Supervision using Point Clouds [50.40632021583213]
  Existing vision based supervised approaches to lateral vehicle control are capable of directly mapping RGB images to the appropriate steering commands. This paper proposes a framework for training a more robust and scalable model for lateral vehicle control. Online experiments show that the performance of our method is superior to that of the supervised model.
  arXiv  Detail & Related papers  (2022-03-20T21:57:32Z)
• Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation [0.0]
  We propose a novel low-complexity lidar gesture recognition system for mobile robot control. The system is lightweight and suitable for mobile robot control with limited computing power. The use of lidar contributes to the robustness of the system, allowing it to operate in most outdoor conditions.
  arXiv  Detail & Related papers  (2021-09-17T00:49:31Z)
• Self-optimizing adaptive optics control with Reinforcement Learning for high-contrast imaging [0.0]
  We describe how model-free Reinforcement Learning can be used to optimize a Recurrent Neural Network controller for closed-loop predictive control. We show in simulations that our algorithm can also be applied to the control of a high-order deformable mirror.
  arXiv  Detail & Related papers  (2021-08-24T10:02:55Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction [67.08350202974434]
  We propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. We show that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.
  arXiv  Detail & Related papers  (2020-07-08T02:26:19Z)
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives [89.34229413345541]
  We propose a conditioning scheme which avoids pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion. We report significant improvements in task success over representative MPC and IL baselines.
  arXiv  Detail & Related papers  (2020-03-19T15:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.