Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation

• URL: http://arxiv.org/abs/2606.30266v1
• Date: Mon, 29 Jun 2026 13:14:57 GMT
• Title: Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation
• Abstract summary: Motion-language agents must possess the capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M)<n>We investigate the stability-plasticity trade-off in bidirectional motion-language learning under sequential task exposure.
• Score: 0.3441021278275805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M). While foundational models have achieved strong performance in static settings, autonomous agents operating in dynamic environments must continuously incorporate new motion concepts -- such as novel athletic styles or specialized gestures -- without catastrophic forgetting of previously acquired skills. We investigate the stability-plasticity trade-off in bidirectional motion-language learning under sequential task exposure. Building on a frozen large language model backbone, we introduce low-rank adaptation (LoRA) variants designed to mitigate inter-task interference. We specifically propose mixture-of-experts architectures that utilize an autoencoder-based router to select task-specific experts at inference time, so that no task-label is needed. To evaluate these methods, we establish a reproducible five-task benchmark derived from HumanML3D through semantic clustering of motion descriptions. Our experimental results demonstrate near-zero forgetting across both M2T and T2M directions while maintaining high generation and captioning quality. Furthermore, we show that hard expert selection via routing significantly outperforms soft expert blending in quality metrics, indicating that preserving expert isolation is critical for maintaining performance in our continual learning setting. Finally, we observe that a divergence between token-level accuracy and downstream generation quality may occur, highlighting the need for more comprehensive evaluation protocols in future research on lifelong motion-language agents.

Related papers

• Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World [31.64682340600284]
  We propose Open-UniMo, a unified Large Motion-Language Model (LMLM) trained on million-scale open-world motion-language data.<n>Open-UniMo promotes modality parity by extending Qwen's vocabulary of about 150K text tokens with 64K motion tokens.<n>We also introduce motion-consistent Chain-of-Thought reasoning as an intermediate representation to bridge language semantics and motion dynamics.
  arXiv  Detail & Related papers  (2026-09-13T15:46:41Z)
• Progressively Learning Heterogeneous Skills in a Unified Latent Space [69.64319218245176]
  HetSkills is a novel framework designed to learn heterogeneous skills within a unified latent space for physics-based character control.<n>HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder.<n>We introduce motion distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions.
  arXiv  Detail & Related papers  (2026-08-24T13:48:45Z)
• Re$^2$MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement [27.84741874985021]
  Re$2$MoGen is a Reasoning and Refinement open-vocabulary Motion Generation framework.<n>It generates an initial motion planning and then refines its physical plausibility via reinforcement learning (RL) post-training.<n>Our framework can generate semantically consistent and physically plausible motions and achieve state-of-the-art performance in open-vocabulary motion generation.
  arXiv  Detail & Related papers  (2026-04-20T04:59:28Z)
• KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition [89.7916653471162]
  We introduce a novel kinematics-rich vision-language-action (VLA) task, in which language commands densely encode diverse kinematic attributes.<n>In this setting, task goals remain invariant while execution trajectories must adapt to instruction-level kinematic specifications.<n>We show that KineVLA consistently outperforms strong VLA baselines on kinematics-sensitive benchmarks.
  arXiv  Detail & Related papers  (2026-03-18T09:28:49Z)
• IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation [54.36300724708094]
  Assessment and refinement tasks act as crucial bridges to enable bidirectional knowledge flow between understanding and generation.<n>We introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve generation performance.
  arXiv  Detail & Related papers  (2025-12-11T15:16:06Z)
• DEFT-LLM: Disentangled Expert Feature Tuning for Micro-Expression Recognition [16.903294278064667]
  We propose DEFT-LLM, which achieves semantic alignment by multi-expert disentanglement.<n>We first introduce Uni-MER, a motion-driven instruction designed to align text with local facial motion.<n>We then design an architecture with three experts to decouple facial dynamics into independent representations.
  arXiv  Detail & Related papers  (2025-11-14T04:21:24Z)
• HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA [19.21746661300773]
  3D human motion is being progressively integrated with foundation models to enhance multimodal understanding and cross-modal generation capabilities.<n>We propose the Human Motion-Vision-Language Model (HMVLM), a unified framework based on the Mixture of Expert Low-Rank Adaption(MoE LoRA) strategy.<n>We show that our method effectively alleviates knowledge forgetting during instruction-tuning and achieves remarkable performance across diverse human motion downstream tasks.
  arXiv  Detail & Related papers  (2025-11-03T11:22:10Z)
• Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents [43.5771856761934]
  Vision-and-Language Navigation (VLN) poses significant challenges for agents to interpret natural language instructions and navigate complex 3D environments.<n>We propose SkillNav, a modular framework that introduces structured, skill-based reasoning into Transformer-based VLN agents.
  arXiv  Detail & Related papers  (2025-08-11T05:50:30Z)
• MotionGPT3: Human Motion as a Second Modality [28.616340011811843]
  MotionGPT3 is a bimodal motion-language model for both understanding and generation.<n>A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow.<n>Experiments show that MotionGPT3 achieves 2x faster convergence in training loss and up to 4x faster convergence in validation.
  arXiv  Detail & Related papers  (2025-06-30T17:42:22Z)
• GENMO: A GENeralist Model for Human MOtion [64.16188966024542]
  We present GENMO, a unified Generalist Model for Human Motion that bridges motion estimation and generation in a single framework.<n>Our key insight is to reformulate motion estimation as constrained motion generation, where the output motion must precisely satisfy observed conditioning signals.<n>Our novel architecture handles variable-length motions and mixed multimodal conditions (text, audio, video) at different time intervals, offering flexible control.
  arXiv  Detail & Related papers  (2025-05-02T17:59:55Z)
• MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding [76.30210465222218]
  MotionGPT-2 is a unified Large Motion-Language Model (LMLMLM) It supports multimodal control conditions through pre-trained Large Language Models (LLMs) It is highly adaptable to the challenging 3D holistic motion generation task.
  arXiv  Detail & Related papers  (2024-10-29T05:25:34Z)
• MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition [94.56755080185732]
  We propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information. Our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation.
  arXiv  Detail & Related papers  (2024-05-31T08:06:05Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.