Perception-Control Coupled Visual Servoing for Textureless Objects Using Keypoint-Based EKF

• URL: http://arxiv.org/abs/2602.06834v2
• Date: Wed, 25 Feb 2026 15:45:01 GMT
• Title: Perception-Control Coupled Visual Servoing for Textureless Objects Using Keypoint-Based EKF
• Abstract summary: We build upon learning-based keypoint detection for textureless objects and propose a method that enhances robustness.<n>We employ an Extended Kalman Filter (EKF) that integrates per-frame keypoint measurements to estimate 6D object pose.<n>Unlike standard PBVS, we propose a probabilistic control law that computes both camera velocity and its associated uncertainty.
• Score: 2.7711760478810157
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Visual servoing is fundamental to robotic applications, enabling precise positioning and control. However, applying it to textureless objects remains a challenge due to the absence of reliable visual features. Moreover, adverse visual conditions, such as occlusions, often corrupt visual feedback, leading to reduced accuracy and instability in visual servoing. In this work, we build upon learning-based keypoint detection for textureless objects and propose a method that enhances robustness by tightly integrating perception and control in a closed loop. Specifically, we employ an Extended Kalman Filter (EKF) that integrates per-frame keypoint measurements to estimate 6D object pose, which drives pose-based visual servoing (PBVS) for control. The resulting camera motion, in turn, enhances the tracking of subsequent keypoints, effectively closing the perception-control loop. Additionally, unlike standard PBVS, we propose a probabilistic control law that computes both camera velocity and its associated uncertainty, enabling uncertainty-aware control for safe and reliable operation. We validate our approach on real-world robotic platforms using quantitative metrics and grasping experiments, demonstrating that our method outperforms traditional visual servoing techniques in both accuracy and practical application.

Related papers

• Temporally Consistent Object 6D Pose Estimation for Robot Control [16.973122634563314]
  We develop a factor graph approach to enforce temporal consistency of the object pose estimates.<n>We experimentally validate the stability of the proposed approach for a feedback-based robot control task.
  arXiv  Detail & Related papers  (2026-05-04T15:15:09Z)
• AI-Enabled Image-Based Hybrid Vision/Force Control of Tendon-Driven Aerial Continuum Manipulators [0.6999740786886535]
  This paper presents an AI-enabled cascaded hybrid vision/force control framework for tendon-driven aerial manipulators.<n>The proposed controller is designed to enable autonomous, physical interaction with a static environment while stabilizing the image feature error.
  arXiv  Detail & Related papers  (2026-04-21T01:19:11Z)
• PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control [67.17998939712326]
  We present PoseDiff, a conditional diffusion model that unifies robot state estimation and control within a single framework.<n>At its core, PoseDiff maps raw visual observations into structured robot states-such as 3D keypoints or joint angles-from a single RGB image.<n>Building upon this foundation, PoseDiff extends naturally to video-to-action inverse dynamics.
  arXiv  Detail & Related papers  (2025-09-29T10:55:48Z)
• Verification of Visual Controllers via Compositional Geometric Transformations [49.81690518952909]
  We introduce a novel verification framework for perception-based controllers that can generate outer-approximations of reachable sets.<n>We provide theoretical guarantees on the soundness of our method and demonstrate its effectiveness across benchmark control environments.
  arXiv  Detail & Related papers  (2025-07-06T20:22:58Z)
• ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation [14.575011735514522]
  ViTa-Zero is a zero-shot visuotactile pose estimation framework.<n>We validate our framework through experiments on a real-world robot setup.
  arXiv  Detail & Related papers  (2025-04-17T17:59:56Z)
• Enhancing Feature Tracking Reliability for Visual Navigation using Real-Time Safety Filter [23.90865158642276]
  Vision sensors are extensively used for localizing a robot's pose.<n>For reliable feature tracking and accurate pose estimation, it is crucial to maintain visibility of a sufficient number of features.<n>We propose a real-time safety filter based on quadratic programming.
  arXiv  Detail & Related papers  (2025-02-03T06:26:04Z)
• VICAN: Very Efficient Calibration Algorithm for Large Camera Networks [49.17165360280794]
  We introduce a novel methodology that extends Pose Graph Optimization techniques. We consider the bipartite graph encompassing cameras, object poses evolving dynamically, and camera-object relative transformations at each time step. Our framework retains compatibility with traditional PGO solvers, but its efficacy benefits from a custom-tailored optimization scheme.
  arXiv  Detail & Related papers  (2024-03-25T17:47:03Z)
• EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration [49.90228618894857]
  We introduce a new approach to hand-eye calibration called EasyHeC, which is markerless, white-box, and delivers superior accuracy and robustness. We propose to use two key technologies: differentiable rendering-based camera pose optimization and consistency-based joint space exploration. Our evaluation demonstrates superior performance in synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-02T03:49:54Z)
• Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment [9.599112149531557]
  This letter presents a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing. A convolutional neural network is trained to predict the actuations required to achieve the desired pose. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera.
  arXiv  Detail & Related papers  (2022-02-10T18:03:28Z)
• End-to-End Learning of Keypoint Representations for Continuous Control from Images [84.8536730437934]
  We show that it is possible to learn efficient keypoint representations end-to-end, without the need for unsupervised pre-training, decoders, or additional losses. Our proposed architecture consists of a differentiable keypoint extractor that feeds the coordinates directly to a soft actor-critic agent.
  arXiv  Detail & Related papers  (2021-06-15T09:17:06Z)
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives [89.34229413345541]
  We propose a conditioning scheme which avoids pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion. We report significant improvements in task success over representative MPC and IL baselines.
  arXiv  Detail & Related papers  (2020-03-19T15:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.