Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward

• URL: http://arxiv.org/abs/2506.03191v1
• Date: Sat, 31 May 2025 11:02:24 GMT
• Title: Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward
• Authors: Muhammad Islam, Tao Huang, Euijoon Ahn, Usman Naseem,
• Abstract summary: This paper focuses on the use of multimodal Generative Artificial Intelligence (GenAI) and autoregressive Large Language Models (LLMs) for human motion understanding and generation.<n>It offers insights into emerging methods, architectures, and their potential to advance realistic and versatile motion synthesis.<n>This research underscores the transformative potential of text-to-motion GenAI and LLM architectures in applications such as healthcare, humanoids, gaming, animation, and assistive technologies.
• Score: 8.470241117250243
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents an in-depth survey on the use of multimodal Generative Artificial Intelligence (GenAI) and autoregressive Large Language Models (LLMs) for human motion understanding and generation, offering insights into emerging methods, architectures, and their potential to advance realistic and versatile motion synthesis. Focusing exclusively on text and motion modalities, this research investigates how textual descriptions can guide the generation of complex, human-like motion sequences. The paper explores various generative approaches, including autoregressive models, diffusion models, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and transformer-based models, by analyzing their strengths and limitations in terms of motion quality, computational efficiency, and adaptability. It highlights recent advances in text-conditioned motion generation, where textual inputs are used to control and refine motion outputs with greater precision. The integration of LLMs further enhances these models by enabling semantic alignment between instructions and motion, improving coherence and contextual relevance. This systematic survey underscores the transformative potential of text-to-motion GenAI and LLM architectures in applications such as healthcare, humanoids, gaming, animation, and assistive technologies, while addressing ongoing challenges in generating efficient and realistic human motion.

Related papers

• Motion Generation: A Survey of Generative Approaches and Benchmarks [1.4254358932994455]
  We provide an in-depth categorization of motion generation methods based on their underlying generative strategies.<n>Our main focus is on papers published in top-tier venues since 2023, reflecting the most recent advancements in the field.<n>We analyze architectural principles, conditioning mechanisms, and generation settings, and compile a detailed overview of the evaluation metrics and datasets used across the literature.
  arXiv  Detail & Related papers  (2025-07-07T19:04:56Z)
• Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset [113.25650486482762]
  We introduce the Seamless Interaction dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage.<n>This dataset enables the development of AI technologies that understand dyadic embodied dynamics.<n>We develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech.
  arXiv  Detail & Related papers  (2025-06-27T18:09:49Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• CASIM: Composite Aware Semantic Injection for Text to Motion Generation [15.53049009014166]
  We propose a composite-aware semantic injection mechanism that learns the dynamic correspondence between text and motion tokens.<n> Experiments on HumanML3D and KIT benchmarks demonstrate that CASIM consistently improves motion quality, text-motion alignment, and retrieval scores.
  arXiv  Detail & Related papers  (2025-02-04T07:22:07Z)
• COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models [14.130327598928778]
  Large language models (LLMs) and hierarchical motion-specific vector-quantized variational autoencoders (VQ-VAEs) are proposed. Our framework generates realistic and diverse collaborative human-object-human interactions, outperforming state-of-the-art methods. Our work opens up new possibilities for modeling complex interactions in various domains, such as robotics, graphics and computer vision.
  arXiv  Detail & Related papers  (2024-09-30T17:02:13Z)
• Learning Generalizable Human Motion Generator with Reinforcement Learning [95.62084727984808]
  Text-driven human motion generation is one of the vital tasks in computer-aided content creation. Existing methods often overfit specific motion expressions in the training data, hindering their ability to generalize. We present textbfInstructMotion, which incorporate the trail and error paradigm in reinforcement learning for generalizable human motion generation.
  arXiv  Detail & Related papers  (2024-05-24T13:29:12Z)
• Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation [30.245348014602577]
  We discuss the evolution of video generation from text, starting with animating MNIST numbers to simulating the physical world with Sora. Our review into the shortcomings of Sora-generated videos pinpoints the call for more in-depth studies in various enabling aspects of video generation. We conclude that the study of the text-to-video generation may still be in its infancy, requiring contribution from the cross-discipline research community.
  arXiv  Detail & Related papers  (2024-03-08T07:58:13Z)
• When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges [50.280704114978384]
  Pre-trained large language models (LLMs) exhibit powerful capabilities for generating natural text.<n> Evolutionary algorithms (EAs) can discover diverse solutions to complex real-world problems.
  arXiv  Detail & Related papers  (2024-01-19T05:58:30Z)
• AMD:Anatomical Motion Diffusion with Interpretable Motion Decomposition and Fusion [11.689663297469945]
  We propose the Adaptable Motion Diffusion model. It exploits a Large Language Model (LLM) to parse the input text into a sequence of concise and interpretable anatomical scripts. We then devise a two-branch fusion scheme that balances the influence of the input text and the anatomical scripts on the inverse diffusion process.
  arXiv  Detail & Related papers  (2023-12-20T04:49:45Z)
• Interactive Character Control with Auto-Regressive Motion Diffusion Models [18.727066177880708]
  We propose A-MDM (Auto-regressive Motion Diffusion Model) for real-time motion synthesis. Our conditional diffusion model takes an initial pose as input, and auto-regressively generates successive motion frames conditioned on previous frame. We introduce a suite of techniques for incorporating interactive controls into A-MDM, such as task-oriented sampling, in-painting, and hierarchical reinforcement learning.
  arXiv  Detail & Related papers  (2023-06-01T07:48:34Z)
• TEMOS: Generating diverse human motions from textual descriptions [53.85978336198444]
  We address the problem of generating diverse 3D human motions from textual descriptions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data. We show that TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions.
  arXiv  Detail & Related papers  (2022-04-25T14:53:06Z)
• DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations [119.1953397679783]
  We focus on advancing the state-of-the-art in interpreting multimodal models. Our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models.
  arXiv  Detail & Related papers  (2022-03-03T20:52:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.