GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes

• URL: http://arxiv.org/abs/2603.17993v1
• Date: Wed, 18 Mar 2026 17:54:35 GMT
• Title: GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes
• Abstract summary: GMT is a multimodal transformer framework that generates realistic and goal-directed object trajectories.<n>Experiments on synthetic and real-world benchmarks demonstrate that GMT outperforms state-of-the-art human motion and human-object interaction baselines.
• Score: 47.88691731631585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthesizing controllable 6-DOF object manipulation trajectories in 3D environments is essential for enabling robots to interact with complex scenes, yet remains challenging due to the need for accurate spatial reasoning, physical feasibility, and multimodal scene understanding. Existing approaches often rely on 2D or partial 3D representations, limiting their ability to capture full scene geometry and constraining trajectory precision. We present GMT, a multimodal transformer framework that generates realistic and goal-directed object trajectories by jointly leveraging 3D bounding box geometry, point cloud context, semantic object categories, and target end poses. The model represents trajectories as continuous 6-DOF pose sequences and employs a tailored conditioning strategy that fuses geometric, semantic, contextual, and goaloriented information. Extensive experiments on synthetic and real-world benchmarks demonstrate that GMT outperforms state-of-the-art human motion and human-object interaction baselines, such as CHOIS and GIMO, achieving substantial gains in spatial accuracy and orientation control. Our method establishes a new benchmark for learningbased manipulation planning and shows strong generalization to diverse objects and cluttered 3D environments. Project page: https://huajian- zeng.github. io/projects/gmt/.

Related papers

• Generative Texture Diversification of 3D Pedestrians for Robust Autonomous Driving Perception [0.0]
  In this paper, we propose a method for scaling variability in 3D pedestrian assets for synthetic scene generation.<n>We generate multiple distinct pedestrian instances by synthesizing diverse facial textures and identity-level appearance variations.<n>Using the assets, we construct synthetic datasets and study the impact of mixing real and synthetic data for RGB-based object detection.
  arXiv  Detail & Related papers  (2026-05-13T16:35:50Z)
• MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation [14.056276184323321]
  MaMi-HOI reconciles textbfMacro-level fluidity with textbfMicro-level spatial precision.<n>Experiments validate that MaMi-HOI simultaneously achieves natural motion and precise contact.
  arXiv  Detail & Related papers  (2026-05-07T06:52:14Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• Unified Semantic Transformer for 3D Scene Understanding [55.415468022487005]
  We introduce UNITE, a novel feed-forward neural network that unifies a diverse set of 3D semantic tasks within a single model.<n>Our model operates on unseen scenes in a fully end-to-end manner and only takes a few seconds to infer the full 3D semantic geometry.<n>We demonstrate that UNITE achieves state-of-the-art performance on several different semantic tasks and even outperforms task-specific models.
  arXiv  Detail & Related papers  (2025-12-16T12:49:35Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories [124.24041272390954]
  Modeling hand-object interaction priors holds significant potential to advance robotic and embodied AI systems.<n>We introduce SIGHT, a novel task focused on generating realistic and physically plausible 3D hand-object interaction trajectories from a single image.<n>We propose SIGHT-Fusion, a novel diffusion-based image-text conditioned generative model that tackles this task by retrieving the most similar 3D object mesh from a database.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.