A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models

• URL: http://arxiv.org/abs/2608.15984v1
• Date: Mon, 17 Aug 2026 00:41:26 GMT
• Title: A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models
• Abstract summary: Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model.<n>We introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models.
• Score: 14.724618805087436
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.

Related papers

• LUNA: Learning Universal 3D Human Animation Beyond Skinning [67.77753663598081]
  We propose LUNA, a universal neural animation model that maps multiple 2D controls like images, keypoints, sketches, and unseen characters into 3D Gaussian deformations.<n>Experiments show LUNA achieves photorealistic competitive visual fidelity compared to LBS-based approaches.<n>To the best of our knowledge, LUNA is the first end-to-end 3D animatable model that supports implicit 2D driving.
  arXiv  Detail & Related papers  (2026-06-30T17:19:55Z)
• Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions [23.080971732537886]
  Mimic2DM is a novel motion imitation framework that learns the control policy directly from 2D keypoint trajectories extracted from videos.<n>We show that the proposed approach is versatile and can effectively learn to synthesize physically plausible and diverse motions across a range of domains.
  arXiv  Detail & Related papers  (2025-12-09T11:30:56Z)
• MoMaps: Semantics-Aware Scene Motion Generation with Motion Maps [31.864441290577545]
  This paper addresses the challenge of learning semantically and functionally meaningful 3D motion priors from real-world videos.<n>We propose a pixel-aligned Motion Map representation for 3D scene motion, which can be generated from existing generative image models.
  arXiv  Detail & Related papers  (2025-10-13T07:56:19Z)
• Mocap-2-to-3: Multi-view Lifting for Monocular Motion Recovery with 2D Pretraining [49.223455189395025]
  Mocap-2-to-3 is a novel framework that performs multi-view lifting from monocular input.<n>To leverage abundant 2D data, we decompose complex 3D motion into multi-view syntheses.<n>Our method surpasses state-of-the-art approaches in both camera-space motion realism and world-grounded human positioning.
  arXiv  Detail & Related papers  (2025-03-05T06:32:49Z)
• Motion-2-to-3: Leveraging 2D Motion Data to Boost 3D Motion Generation [43.915871360698546]
  2D human videos offer a vast and accessible source of motion data, covering a wider range of styles and activities.<n>We introduce a novel framework that disentangles local joint motion from global movements, enabling efficient learning of local motion priors from 2D data.<n>Our method efficiently utilizes 2D data, supporting realistic 3D human motion generation and broadening the range of motion types it supports.
  arXiv  Detail & Related papers  (2024-12-17T17:34:52Z)
• Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Space [78.95579123031733]
  We present $textbfHolistic-Motion2D$, the first comprehensive and large-scale benchmark for 2D whole-body motion generation. We also highlight the utility of 2D motion for various downstream applications and its potential for lifting to 3D motion.
  arXiv  Detail & Related papers  (2024-06-17T06:31:19Z)
• Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment [45.74813582690906]
  Learning 3D human motion from 2D inputs is a fundamental task in the realms of computer vision and computer graphics. We present the Video-to-Motion Generator (VTM), which leverages motion priors through cross-modal latent feature space alignment. The VTM showcases state-of-the-art performance in reconstructing 3D human motion from monocular videos.
  arXiv  Detail & Related papers  (2024-04-15T06:38:09Z)
• DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos [76.01906393673897]
  We propose a self-supervised method to jointly learn 3D motion and depth from monocular videos. Our system contains a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion. Our model delivers superior performance in all evaluated settings.
  arXiv  Detail & Related papers  (2024-03-09T12:22:46Z)
• Cinematic Behavior Transfer via NeRF-based Differentiable Filming [63.1622492808519]
  Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections. We first introduce a reverse filming behavior estimation technique. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment.
  arXiv  Detail & Related papers  (2023-11-29T15:56:58Z)
• Delving into Motion-Aware Matching for Monocular 3D Object Tracking [81.68608983602581]
  We find that the motion cue of objects along different time frames is critical in 3D multi-object tracking. We propose MoMA-M3T, a framework that mainly consists of three motion-aware components. We conduct extensive experiments on the nuScenes and KITTI datasets to demonstrate our MoMA-M3T achieves competitive performance against state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-22T17:53:58Z)
• NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action [24.67958500694608]
  We introduce the Neural Motion (NeMo) field to represent the underlying 3D motions across a set of videos of the same action. NeMo can recover 3D motion in sports using videos from the Penn Action dataset, where NeMo outperforms existing HMR methods in terms of 2D keypoint detection.
  arXiv  Detail & Related papers  (2022-12-28T01:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.