Mobile UMI: Cross-View Diffusion Policy with Decoupled Kinematics for Mobile Manipulation

• URL: http://arxiv.org/abs/2605.20894v1
• Date: Wed, 20 May 2026 08:33:53 GMT
• Title: Mobile UMI: Cross-View Diffusion Policy with Decoupled Kinematics for Mobile Manipulation
• Abstract summary: Recent wrist-mounted interfaces lower the cost of tabletop data collection, yet a single wrist view does not capture the global context required for base navigation.<n>generative policies introduce hundreds of milliseconds of inference latency, during which the base advances past predicted waypoints.<n>This paper presents Mobile UMI, a hardware-free demonstration framework that addresses both gaps through three components.
• Score: 4.311650365696827
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mobile imitation learning on portable demonstration interfaces faces two coupled bottlenecks: locomotion-contaminated action labels and inference-induced execution latency on a continuously moving base. Recent wrist-mounted interfaces lower the cost of tabletop data collection, yet a single wrist view does not capture the global context required for base navigation. Adding a body-mounted camera entangles human walking with hand motion. Meanwhile, generative policies introduce hundreds of milliseconds of inference latency, during which the base advances past predicted waypoints, forcing backward corrections at action splices. This paper presents Mobile UMI, a hardware-free demonstration framework that addresses both gaps through three components. First, a dual-camera capture system records chest-centric global context and wrist-centric local interaction without any robot present. Second, a one-shot ChArUco-based spatial anchor unifies the chest and hand visual-inertial frames; the hand pose is then re-expressed relative to the chest to extract decoupled SE(3) manipulation and SE(2) base trajectories. Third, an asynchronous receding-horizon executor performs online state matching: each generated action chunk is realigned with the current physical pose so that expired waypoints are discarded before execution. The full system is evaluated on four long-horizon household tasks, achieving an average success rate of 83.8% over 100 trials per task. Controlled comparisons against ACT and Diffusion Policy show that the chest-relative label alone closes much of the gap; online state matching closes the remainder. These results indicate that, for mobile imitation learning under the tested conditions, explicit kinematic factorization combined with state-level latency alignment provides an effective solution without requiring architectural changes to the underlying policy class.

Related papers

• Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assembly [9.415285027294221]
  We present PACE (Proprioception-Anchored Cross-Modalception), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions.<n>Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3% and only a 2.7-percentage-point sim-to-real drop.
  arXiv  Detail & Related papers  (2026-09-07T14:20:22Z)
• RASA: Disentangled Spatial-Motional Priors for Cross-Identity Character Animation [85.41886214825894]
  Cross-identity character animation aims to drive a target identity from a reference image to follow the motion of a source character from a driving video.<n>We introduce Reference-Aware Structural Alignment (RASA), a framework that disentangles spatial mapping from motion control.<n>Our work establishes a new paradigm showing disentangled spatial and motional priors are key to robust character animation.
  arXiv  Detail & Related papers  (2026-08-28T11:36:44Z)
• DeepInsight II: One Trace from Benchmark to Robot [8.388561928330978]
  DeepInsight II keeps that substrate fixed and quantifies the embodied half.<n> MotionBench places four released whole-body controllers under one workload and metric contract.<n>A composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels.
  arXiv  Detail & Related papers  (2026-08-17T13:25:24Z)
• KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking [18.013999575547427]
  In-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera.<n>We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact.<n>We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation.
  arXiv  Detail & Related papers  (2026-07-16T11:07:47Z)
• WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models [37.59051701511746]
  We introduce WorldRoamBench, an open-world benchmark for long-horizon stability across four dimensions.<n>The benchmark comprises 600+ test cases across Nature, Urban, and Indoor scenes in first/third-person views with WASD 10-60s continuous interaction.
  arXiv  Detail & Related papers  (2026-06-30T13:51:44Z)
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games [69.57330692969543]
  We introduce RNG-Bench, a benchmark suite designed to isolate a base model's ability to reconstruct past observations.<n>RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map.<n>The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs.
  arXiv  Detail & Related papers  (2026-06-17T17:59:34Z)
• Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors [51.096845970243855]
  Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data.<n>We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction.
  arXiv  Detail & Related papers  (2026-05-21T10:15:39Z)
• MoCapAnything V2: End-to-End Motion Capture for Arbitrary Skeletons [56.68975315643491]
  We present the first fully end-to-end framework in which Videoto-Pose and Pose-to-Rotation are learnable and optimized jointly.<n>We show that our method reduces rotation error from 17 degrees to 10 degrees, and to 6.54 degrees on unseen skeletons, while achieving 20x faster inference than mesh-based pipelines.
  arXiv  Detail & Related papers  (2026-04-30T17:16:38Z)
• AG-EgoPose: Leveraging Action-Guided Motion and Kinematic Joint Encoding for Egocentric 3D Pose Estimation [11.399976049695212]
  AG-EgoPose is a novel dual-stream framework that integrates short- and long-range motion context with fine-grained spatial cues for robust pose estimation.<n>Our framework features two parallel streams: A spatial stream uses a weight-sharing ResNet-18 encoder-decoder to generate 2D joint heatmaps.<n>A temporal stream uses a ResNet-50 backbone to extract visual features, which are then processed by an action recognition backbone to capture the motion dynamics.
  arXiv  Detail & Related papers  (2026-03-26T08:46:51Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment [32.178798481694834]
  We introduce MoBind, a hierarchical contrastive learning framework designed to address three challenges.<n>To isolate motion-relevant cues, MoBind aligns IMU signals with skeletal motion sequences rather than raw pixels.<n>To capture detailed temporal correspondence, MoBind employs a hierarchical contrastive strategy that first aligns token-level temporal segments, then fuses local (body-part) alignment with global (body-wide) motion aggregation.
  arXiv  Detail & Related papers  (2026-02-22T01:54:29Z)
• Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds [46.859932208933735]
  Articulated objects are prevalent in daily life and robotic manipulation tasks.<n> pose tracking for articulated objects remains an underexplored problem due to their inherent kinematic constraints.<n>This work proposes a novel point-pair-based pose tracking framework, termed textbfPPF-Tracker
  arXiv  Detail & Related papers  (2025-11-08T12:56:21Z)
• Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model [63.336123527432136]
  We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation.<n>Unlike existing video generative models for autonomous driving, the proposed designs are tailored for interactive simulation.<n>We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-11T06:35:18Z)
• MATE: Motion-Augmented Temporal Consistency for Event-based Point Tracking [58.719310295870024]
  This paper presents an event-based framework for tracking any point.<n>To resolve ambiguities caused by event sparsity, a motion-guidance module incorporates kinematic vectors into the local matching process.<n>The method improves the $Survival_50$ metric by 17.9% over event-only tracking of any point baseline.
  arXiv  Detail & Related papers  (2024-12-02T09:13:29Z)
• Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videos [91.44553585470688]
  Multi-person 3D mesh recovery from videos is a critical first step towards automatic perception of group behavior in virtual reality, physical therapy and beyond. We propose the Coordinate transFormer (CoordFormer) that directly models multi-person spatial-temporal relations and simultaneously performs multi-mesh recovery in an end-to-end manner. Experiments on the 3DPW dataset demonstrate that CoordFormer significantly improves the state-of-the-art, outperforming the previously best results by 4.2%, 8.8% and 4.7% according to the MPJPE, PAMPJPE, and PVE metrics, respectively.
  arXiv  Detail & Related papers  (2023-08-20T18:23:07Z)
• Tsanet: Temporal and Scale Alignment for Unsupervised Video Object Segmentation [21.19216164433897]
  Unsupervised Video Object (UVOS) refers to the challenging task of segmenting the prominent object in videos without manual guidance. We propose a novel framework for UVOS that can address the aforementioned limitations of the two approaches. We present experimental results on public benchmark datasets, DAVIS 2016 and FBMS, which demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-03-08T04:59:43Z)
• Real-time Controllable Motion Transition for Characters [14.88407656218885]
  Real-time in-between motion generation is universally required in games and highly desirable in existing animation pipelines. Our approach consists of two key components: motion manifold and conditional transitioning. We show that our method is able to generate high-quality motions measured under multiple metrics.
  arXiv  Detail & Related papers  (2022-05-05T10:02:54Z)
• Learning to Segment Rigid Motions from Two Frames [72.14906744113125]
  We propose a modular network, motivated by a geometric analysis of what independent object motions can be recovered from an egomotion field. It takes two consecutive frames as input and predicts segmentation masks for the background and multiple rigidly moving objects, which are then parameterized by 3D rigid transformations. Our method achieves state-of-the-art performance for rigid motion segmentation on KITTI and Sintel.
  arXiv  Detail & Related papers  (2021-01-11T04:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.