KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking

• URL: http://arxiv.org/abs/2607.14842v1
• Date: Thu, 16 Jul 2026 11:07:47 GMT
• Title: KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking
• Abstract summary: In-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera.<n>We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact.<n>We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation.
• Score: 18.013999575547427
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact, to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating, using vision exclusively for translation and dedicating one attention head to haptics for rotation, without explicit supervision; and (iii) compact finger-level tokenization with 4 tokens outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations. Our project page is available at https://cold-young.github.io/kine-fuse/.

Related papers

• DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery [51.92897182932217]
  Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
  arXiv  Detail & Related papers  (2026-08-20T17:46:24Z)
• DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation [58.47973015036709]
  DisPOSE is a self-supervised framework that approximates the inherently discrete multi-view person-assignment problem.<n>By employing differentiable Sinkhorn projections during denoising, our model learns to guide solutions toward valid and feasible assignments.<n>The proposed approach outperforms current state-of-the-art self-supervised methods on standard datasets.
  arXiv  Detail & Related papers  (2026-06-05T16:14:39Z)
• Sparse3DTrack: Monocular 3D Object Tracking Using Sparse Supervision [16.586885757497203]
  Monocular 3D object tracking aims to estimate temporally consistent 3D object poses across video frames.<n>Existing state-the-art approaches are fully supervised and rely on dense 3D annotations over long video sequences.<n>We propose the first supervised framework for monocular 3D object tracking.
  arXiv  Detail & Related papers  (2026-03-18T21:36:41Z)
• MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning [66.53533434848369]
  We propose a motion-guided self-learning framework that learns densely consistent representations.<n>We improve state-of-the-art by 1% to 6% on six image and video datasets and four evaluation benchmarks.
  arXiv  Detail & Related papers  (2025-06-10T11:20:32Z)
• Self-Supervised and Generalizable Tokenization for CLIP-Based 3D Understanding [87.68271178167373]
  We present a universal 3D tokenizer designed for scale-invariant representation learning with a frozen CLIP backbone.<n>S4Token is a tokenization pipeline that produces semantically-informed tokens regardless of scene scale.
  arXiv  Detail & Related papers  (2025-05-24T18:26:30Z)
• Uni4D: A Unified Self-Supervised Learning Framework for Point Cloud Videos [70.07088203106443]
  Existing methods rely on explicit knowledge to learn motion, resulting in suboptimal representations.<n>Prior Masked Autoentangler (MAE) frameworks struggle to bridge the gap between low-level geometry and high-level dynamics in 4D data.<n>We propose a novel self-disentangled MAE for learning expressive,riminative, and transferable 4D representations.
  arXiv  Detail & Related papers  (2025-04-07T08:47:36Z)
• Object Discovery from Motion-Guided Tokens [50.988525184497334]
  We augment the auto-encoder representation learning framework with motion-guidance and mid-level feature tokenization. Our approach enables the emergence of interpretable object-specific mid-level features.
  arXiv  Detail & Related papers  (2023-03-27T19:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.