UNIC: Learning Unified Multimodal Extrinsic Contact Estimation

• URL: http://arxiv.org/abs/2601.04356v1
• Date: Wed, 07 Jan 2026 19:43:16 GMT
• Title: UNIC: Learning Unified Multimodal Extrinsic Contact Estimation
• Abstract summary: We present UNIC, a unified framework for extrinsic contact estimation.<n>UNIC directly encodes visual observations in the camera frame and integrates them with proprioceptive and tactile modalities in a fully data-driven manner.<n>It achieves a 9.6 mm average Chamfer distance error on unseen contact locations, performs well on unseen objects, remains robust under missing modalities, and adapts to dynamic camera viewpoints.
• Score: 5.636323593047756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contact-rich manipulation requires reliable estimation of extrinsic contacts-the interactions between a grasped object and its environment which provide essential contextual information for planning, control, and policy learning. However, existing approaches often rely on restrictive assumptions, such as predefined contact types, fixed grasp configurations, or camera calibration, that hinder generalization to novel objects and deployment in unstructured environments. In this paper, we present UNIC, a unified multimodal framework for extrinsic contact estimation that operates without any prior knowledge or camera calibration. UNIC directly encodes visual observations in the camera frame and integrates them with proprioceptive and tactile modalities in a fully data-driven manner. It introduces a unified contact representation based on scene affordance maps that captures diverse contact formations and employs a multimodal fusion mechanism with random masking, enabling robust multimodal representation learning. Extensive experiments demonstrate that UNIC performs reliably. It achieves a 9.6 mm average Chamfer distance error on unseen contact locations, performs well on unseen objects, remains robust under missing modalities, and adapts to dynamic camera viewpoints. These results establish extrinsic contact estimation as a practical and versatile capability for contact-rich manipulation.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation [74.87716678195099]
  Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation.<n>Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge.<n>We propose a motion-aware tactile representation to facilitate contact-rich manipulation.
  arXiv  Detail & Related papers  (2026-06-29T08:20:18Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• Multi-Modal Semantic Communication [39.55262791529245]
  We propose a novel Multi-Modal Semantic Communication framework that integrates text-based user queries to guide the information extraction process.<n>Our proposed system employs a cross-modal attention mechanism that fuses visual features with language embeddings to produce soft relevance scores.<n>At the receiver, the patches are reconstructed and combined to preserve taskcritical information.
  arXiv  Detail & Related papers  (2025-12-17T18:47:22Z)
• Multi-Modal Manipulation via Multi-Modal Policy Consensus [62.49978559936122]
  We propose a new approach to integrate diverse sensory modalities for robotic manipulation.<n>Our method factorizes the policy into a set of diffusion models, each specialized for a single representation.<n>We evaluate our approach on simulated manipulation tasks in RLBench, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion.
  arXiv  Detail & Related papers  (2025-09-27T19:43:04Z)
• Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation [55.2480439325792]
  Safety-critical applications, such as autonomous driving and medical image analysis, require extensive multimodal data for rigorous testing.<n>This work introduces two novel methods for synthetic data generation in autonomous driving and medical image analysis, namely MObI and AnydoorMed, respectively.
  arXiv  Detail & Related papers  (2025-07-30T19:43:47Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• MObI: Multimodal Object Inpainting Using Diffusion Models [52.07640413626605]
  This paper introduces MObI, a novel framework for Multimodal Object Inpainting.<n>Using a single reference RGB image, MObI enables objects to be seamlessly inserted into existing multimodal scenes.<n>Unlike traditional inpainting methods that rely solely on edit masks, our 3D bounding box conditioning gives objects accurate spatial positioning and realistic scaling.
  arXiv  Detail & Related papers  (2025-01-06T17:43:26Z)
• Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes [51.20150148066458]
  We propose a novel method to learn to fuse the multi-view and monocular cues encoded as volumes without needing the generalizationally crafted masks. Experiments on real-world datasets prove the significant effectiveness and ability of the proposed method.
  arXiv  Detail & Related papers  (2023-04-18T13:55:24Z)
• Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition [0.0]
  Multi-sensor frameworks provide opportunities for ensemble learning and sensor fusion. We propose and analyze an imputation scheme to handle missing information. We show that a late-fusion approach between parallel convolutional neural networks can outperform even the best-placed single camera model.
  arXiv  Detail & Related papers  (2023-01-30T00:24:27Z)
• Emergent Quantized Communication [34.31732248872158]
  We propose an alternative approach to achieve discrete communication -- quantization of communicated messages. Message quantization allows us to train the model end-to-end, achieving superior performance in multiple setups.
  arXiv  Detail & Related papers  (2022-11-04T12:39:45Z)
• Stability-driven Contact Reconstruction From Monocular Color Images [7.427212296770506]
  Physical contact provides additional constraints for hand-object state reconstruction. Existing methods optimize the hand-object contact driven by distance threshold or prior from contact-labeled datasets. Our key idea is to reconstruct the contact pattern directly from monocular images, and then utilize the physical stability criterion in the simulation to optimize it.
  arXiv  Detail & Related papers  (2022-05-02T12:23:06Z)
• Exploiting Multi-Object Relationships for Detecting Adversarial Attacks in Complex Scenes [51.65308857232767]
  Vision systems that deploy Deep Neural Networks (DNNs) are known to be vulnerable to adversarial examples. Recent research has shown that checking the intrinsic consistencies in the input data is a promising way to detect adversarial attacks. We develop a novel approach to perform context consistency checks using language models.
  arXiv  Detail & Related papers  (2021-08-19T00:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.