TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks

• URL: http://arxiv.org/abs/2601.14550v1
• Date: Wed, 21 Jan 2026 00:14:28 GMT
• Title: TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks
• Abstract summary: We build on the idea of the handheld demonstration device Universal Manipulation Interface (UMI)<n>We introduce TacUMI, a multi-modal data collection system that integrates ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design.<n>We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries.
• Score: 35.05859151174601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Task decomposition is critical for understanding and learning complex long-horizon manipulation tasks. Especially for tasks involving rich physical interactions, relying solely on visual observations and robot proprioceptive information often fails to reveal the underlying event transitions. This raises the requirement for efficient collection of high-quality multi-modal data as well as robust segmentation method to decompose demonstrations into meaningful modules. Building on the idea of the handheld demonstration device Universal Manipulation Interface (UMI), we introduce TacUMI, a multi-modal data collection system that integrates additionally ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design, which enables synchronized acquisition of all these modalities during human demonstrations. We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries in sequential manipulations. Evaluation on a challenging cable mounting task shows more than 90 percent segmentation accuracy and highlights a remarkable improvement with more modalities, which validates that TacUMI establishes a practical foundation for both scalable collection and segmentation of multi-modal demonstrations in contact-rich tasks.

Related papers

• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe [52.210953881548996]
  We show that a vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks.<n>Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding.<n>Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks.
  arXiv  Detail & Related papers  (2026-07-17T13:25:44Z)
• VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation [10.08235353271524]
  Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging.<n>This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale.<n>We introduce the VTOUCH dataset to provide high-fidelity physical interaction signals, adopt a matrix-style task design, and employ automated data collection pipelines covering real-world, demand-driven scenarios.
  arXiv  Detail & Related papers  (2026-04-22T11:08:08Z)
• Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting [47.35917581534084]
  Reconstructing dynamic scenes with multiple interacting humans and objects from sparse-view inputs is critical for creating high-fidelity digital twins for robotics and VR/AR.<n>This problem, which we term Multi-Human Multi-Object rendering, presents two significant obstacles.<n>We propose MM-GS, a novel hierarchical framework built upon 3D Splatting.
  arXiv  Detail & Related papers  (2026-04-03T12:20:36Z)
• Tracking and Segmenting Anything in Any Modality [75.32774085793498]
  We propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input.<n> SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.
  arXiv  Detail & Related papers  (2025-11-22T09:09:22Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout.<n>DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder.<n>Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training [79.78201886156513]
  We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Our approach achieves comparable results to single-task state-of-the-art models and demonstrates strong generalization on downstream tasks.
  arXiv  Detail & Related papers  (2023-06-29T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.