ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation

• URL: http://arxiv.org/abs/2504.13179v1
• Date: Thu, 17 Apr 2025 17:59:56 GMT
• Title: ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation
• Abstract summary: ViTa-Zero is a zero-shot visuotactile pose estimation framework.<n>We validate our framework through experiments on a real-world robot setup.
• Score: 14.575011735514522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object 6D pose estimation is a critical challenge in robotics, particularly for manipulation tasks. While prior research combining visual and tactile (visuotactile) information has shown promise, these approaches often struggle with generalization due to the limited availability of visuotactile data. In this paper, we introduce ViTa-Zero, a zero-shot visuotactile pose estimation framework. Our key innovation lies in leveraging a visual model as its backbone and performing feasibility checking and test-time optimization based on physical constraints derived from tactile and proprioceptive observations. Specifically, we model the gripper-object interaction as a spring-mass system, where tactile sensors induce attractive forces, and proprioception generates repulsive forces. We validate our framework through experiments on a real-world robot setup, demonstrating its effectiveness across representative visual backbones and manipulation scenarios, including grasping, object picking, and bimanual handover. Compared to the visual models, our approach overcomes some drastic failure modes while tracking the in-hand object pose. In our experiments, our approach shows an average increase of 55% in AUC of ADD-S and 60% in ADD, along with an 80% lower position error compared to FoundationPose.

Related papers

• You Only Touch Once: 6-DoF Object Pose Estimation from Single Tactile Contact [42.491016490387466]
  YOTO is a tactile-only pose estimation system that recovers the full 6-DoF object pose from a single pair of simultaneous contacts.<n>YOTO represents each tactile contact as a local 3D point cloud and localizes it on the object surface through a coarse-to-fine network.<n>Experiments on four geometrically diverse objects demonstrate accurate tactile contact localization and pose estimation, outperforming vision-based and geometric baselines.
  arXiv  Detail & Related papers  (2026-06-27T13:12:57Z)
• InvariantCloud: A Globally Invariant, Uniquely Indexed Point Cloud Framework for Robust 6-DoF Tactile Pose Tracking [7.9092948142021]
  We introduce InvariantCloud, a 6-DoF tactile object pose estimation framework.<n>We show that InvariantCloud achieves superior yaw tracking accuracy and re-localization repeatability compared to existing benchmarks.
  arXiv  Detail & Related papers  (2026-05-24T18:46:24Z)
• MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement [2.1383141762884152]
  We propose MoViD, a viewpoint-invariant 3D human pose estimation framework that disentangles viewpoint information from motion features.<n>MoViD reduces pose estimation error by over 24.2% compared to state-of-the-art methods.<n>MoViD achieves real-time inference at 15 FPS on NVIDIA edge devices.
  arXiv  Detail & Related papers  (2026-03-29T09:08:57Z)
• Active Zero: Self-Evolving Vision-Language Models through Active Environment Exploration [72.84714132070404]
  We propose a framework that shifts from passive interaction to active exploration of visual environments.<n>Active-Zero employs three co-evolving agents: a Searcher that retrieves images from open-world repositories based on the model's capability frontier.<n>On Qwen2.5-VL-7B-Instruct across 12 benchmarks, Active-Zero 53.97 average accuracy on reasoning tasks (5.7% improvement) and 59.77 on general understanding (3.9% improvement)
  arXiv  Detail & Related papers  (2026-02-11T17:29:17Z)
• A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning [67.72413262980272]
  Pre-trained vision models (PVMs) are fundamental to modern robotics, yet their optimal configuration remains unclear.<n>We develop SlotMIM, a method that induces object-centric representations by introducing a semantic bottleneck.<n>Our approach achieves significant improvements over prior work in image recognition, scene understanding, and robot learning evaluations.
  arXiv  Detail & Related papers  (2025-03-10T06:18:31Z)
• V-HOP: Visuo-Haptic 6D Object Pose Tracking [18.984396185797667]
  Humans naturally integrate vision and haptics for robust object perception during manipulation.<n>Prior object pose estimation research has attempted to combine visual and haptic/tactile feedback.<n>We introduce a new visuo-haptic transformer-based object pose tracker that seamlessly integrates visual and haptic input.
  arXiv  Detail & Related papers  (2025-02-24T18:59:50Z)
• FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors [18.88211706267447]
  We tackle the problem of estimating 3D contact forces using vision-based tactile sensors. Our goal is to estimate contact forces over a large range (up to 15 N) on any objects while generalizing across different vision-based tactile sensors.
  arXiv  Detail & Related papers  (2024-10-02T21:28:19Z)
• Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation [57.60490773016364]
  We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation.
  arXiv  Detail & Related papers  (2023-12-20T22:36:37Z)
• ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape Completion [11.483528203807312]
  We introduce ViHOPE, a novel framework for estimating the 6D pose of an in-hand object using visuotactile perception. We use a conditional Generative Adversarial Network to complete the shape of an in-hand object based on volumetric representation. We show that our framework produces models that are robust to sim-to-real transfer on a real-world robot platform.
  arXiv  Detail & Related papers  (2023-09-11T17:58:14Z)
• H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions [62.510951695174604]
  "Hypothesize, Simulate, Act, Update, and Repeat" (H-SAUR) is a probabilistic generative framework that generates hypotheses about how objects articulate given input observations. We show that the proposed model significantly outperforms the current state-of-the-art articulated object manipulation framework. We further improve the test-time efficiency of H-SAUR by integrating a learned prior from learning-based vision models.
  arXiv  Detail & Related papers  (2022-10-22T18:39:33Z)
• Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing [31.49529551069215]
  TEG-Track is a tactile-enhanced 6D pose tracking system. It can track previously unseen objects held in hand. Results demonstrate that TEG-Track consistently enhances state-of-the-art generalizable 6D pose trackers.
  arXiv  Detail & Related papers  (2022-10-08T13:47:03Z)
• Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation [88.8963330073454]
  We propose a novel monocular 6D pose estimation approach by means of self-supervised learning. We leverage current trends in noisy student training and differentiable rendering to further self-supervise the model. Our proposed self-supervision outperforms all other methods relying on synthetic data.
  arXiv  Detail & Related papers  (2022-03-19T15:12:06Z)
• VIPose: Real-time Visual-Inertial 6D Object Pose Tracking [3.44942675405441]
  We introduce a novel Deep Neural Network (DNN) called VIPose to address the object pose tracking problem in real-time. The key contribution is the design of a novel DNN architecture which fuses visual and inertial features to predict the objects' relative 6D pose. The approach presents accuracy performances comparable to state-of-the-art techniques, but with additional benefit to be real-time.
  arXiv  Detail & Related papers  (2021-07-27T06:10:23Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Learning Target Candidate Association to Keep Track of What Not to Track [100.80610986625693]
  We propose to keep track of distractor objects in order to continue tracking the target. To tackle the problem of lacking ground-truth correspondences between distractor objects in visual tracking, we propose a training strategy that combines partial annotations with self-supervision. Our tracker sets a new state-of-the-art on six benchmarks, achieving an AUC score of 67.2% on LaSOT and a +6.1% absolute gain on the OxUvA long-term dataset.
  arXiv  Detail & Related papers  (2021-03-30T17:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.