IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&A

• URL: http://arxiv.org/abs/2508.01984v1
• Date: Mon, 04 Aug 2025 01:44:41 GMT
• Title: IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&A
• Abstract summary: We propose an implicit program-guided motion reasoning (IMoRe) framework that unifies reasoning across multiple query types without manually designed modules.<n>We introduce a program-guided reading mechanism, which dynamically selects multi-level motion representations from a pretrained motion Vision Transformer (ViT)<n>Our model achieves state-of-the-art performance on Babel-QA and generalizes to a newly constructed motion Q&A dataset based on HuMMan.
• Score: 25.91990824698619
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing human motion Q\&A methods rely on explicit program execution, where the requirement for manually defined functional modules may limit the scalability and adaptability. To overcome this, we propose an implicit program-guided motion reasoning (IMoRe) framework that unifies reasoning across multiple query types without manually designed modules. Unlike existing implicit reasoning approaches that infer reasoning operations from question words, our model directly conditions on structured program functions, ensuring a more precise execution of reasoning steps. Additionally, we introduce a program-guided reading mechanism, which dynamically selects multi-level motion representations from a pretrained motion Vision Transformer (ViT), capturing both high-level semantics and fine-grained motion cues. The reasoning module iteratively refines memory representations, leveraging structured program functions to extract relevant information for different query types. Our model achieves state-of-the-art performance on Babel-QA and generalizes to a newly constructed motion Q\&A dataset based on HuMMan, demonstrating its adaptability across different motion reasoning datasets. Code and dataset are available at: https://github.com/LUNAProject22/IMoRe.

Related papers

• In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics [109.99871326497333]
  In-Context Model Predictive Generation (ICMPG) is a framework that integrates language-model planning with inference-time physical feedback.<n>ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules.
  arXiv  Detail & Related papers  (2026-06-25T12:50:33Z)
• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models [40.17845169929452]
  We propose Plug-and-Forecast (PnF), a plug-and-play approach that augments existing motion forecasting models with multimodal large language models (MLLMs)<n>PnF builds on the insight that natural language provides a more effective way to describe and handle complex scenarios, enabling quick adaptation to targeted behaviors.<n>Our method leverages the zero-shot reasoning capabilities of MLLMs to achieve significant improvements in motion prediction performance, while requiring no fine-tuning.
  arXiv  Detail & Related papers  (2025-10-20T08:01:29Z)
• MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing [53.98607267063729]
  MotionVerse is a framework to comprehend, generate, and edit human motion in both single-person and multi-person scenarios.<n>We employ a motion tokenizer with residual quantization, which converts continuous motion sequences into multi-stream discrete tokens.<n>We also introduce a textitDelay Parallel Modeling strategy, which temporally staggers the encoding of residual token streams.
  arXiv  Detail & Related papers  (2025-09-28T04:20:56Z)
• Zero-Shot Human Mobility Forecasting via Large Language Model with Hierarchical Reasoning [27.096256356447117]
  ZHMF is a framework for zero-shot human mobility forecasting.<n>It combines a semantic enhanced retrieval and reflection mechanism with a hierarchical language model based reasoning system.
  arXiv  Detail & Related papers  (2025-09-20T08:46:38Z)
• Motion-R1: Chain-of-Thought Reasoning and Reinforcement Learning for Human Motion Generation [31.484189825477877]
  Motion-R1 is a unified motion-language modeling framework that integrates a Chain-of-Thought mechanism.<n>By explicitly decomposing complex textual instructions into logically structured action paths, Motion-R1 provides high-level semantic guidance for motion generation.<n>Experiments across multiple benchmark datasets demonstrate that Motion-R1 achieves competitive or superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-12T05:21:43Z)
• Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification [22.871255950998016]
  We introduce a novel framework for inference-time visual tokens scaling that enables MLLMs to perform verifier-guided reasoning over visual content.<n>Our method significantly outperforms existing approaches across diverse visual reasoning benchmarks.<n>These results demonstrate the promise of dynamic inference mechanisms for enabling fine-grained, context-aware visual reasoning in next-generation MLLMs.
  arXiv  Detail & Related papers  (2025-06-08T17:38:49Z)
• LatentQA: Teaching LLMs to Decode Activations Into Natural Language [72.87064562349742]
  We introduce LatentQA, the task of answering open-ended questions about model activations in natural language.<n>We propose Latent Interpretation Tuning (LIT), which finetunes a decoder LLM on a dataset of activations and associated question-answer pairs.<n>Our decoder also specifies a differentiable loss that we use to control models, such as debiasing models on stereotyped sentences and controlling the sentiment of generations.
  arXiv  Detail & Related papers  (2024-12-11T18:59:33Z)
• MoReVQA: Exploring Modular Reasoning Models for Video Question Answering [101.25249395748794]
  This paper addresses the task of video question answering (videoQA) via a decomposed multi-stage, modular reasoning framework.<n>Unlike traditional single-stage planning methods, we propose a multi-stage system consisting of an event, a grounding stage, and a final reasoning stage in conjunction with an external memory.
  arXiv  Detail & Related papers  (2024-04-09T17:59:31Z)
• De-fine: Decomposing and Refining Visual Programs with Auto-Feedback [75.62712247421146]
  De-fine is a training-free framework that decomposes complex tasks into simpler subtasks and refines programs through auto-feedback. Our experiments across various visual tasks show that De-fine creates more robust programs.
  arXiv  Detail & Related papers  (2023-11-21T06:24:09Z)
• MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations [25.630268570049708]
  MoConVQ is a novel unified framework for physics-based motion control leveraging scalable discrete representations. Our approach effectively learns motion embeddings from a large, unstructured dataset spanning tens of hours of motion examples.
  arXiv  Detail & Related papers  (2023-10-16T09:09:02Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)
• Motion Question Answering via Modular Motion Programs [10.869455692535853]
  We propose the HumanMotionQA task to evaluate complex, multi-step reasoning abilities of models on long-form human motion sequences. We generate a dataset of question-answer pairs that require detecting motor cues in small portions of motion reasoning temporally about when events occur, and querying specific motion attributes. We demonstrate the suitability of NSPose for HumanMotionQA task, outperforming all baseline methods.
  arXiv  Detail & Related papers  (2023-05-15T18:45:55Z)
• Guiding the PLMs with Semantic Anchors as Intermediate Supervision: Towards Interpretable Semantic Parsing [57.11806632758607]
  We propose to incorporate the current pretrained language models with a hierarchical decoder network. By taking the first-principle structures as the semantic anchors, we propose two novel intermediate supervision tasks. We conduct intensive experiments on several semantic parsing benchmarks and demonstrate that our approach can consistently outperform the baselines.
  arXiv  Detail & Related papers  (2022-10-04T07:27:29Z)
• One for All: One-stage Referring Expression Comprehension with Dynamic Reasoning [11.141645707535599]
  We propose a Dynamic Multi-step Reasoning Network, which allows the reasoning steps to be dynamically adjusted based on the reasoning state and expression complexity. The work achieves the state-of-the-art performance or significant improvements on several REC datasets.
  arXiv  Detail & Related papers  (2022-07-31T04:51:27Z)
• Procedures as Programs: Hierarchical Control of Situated Agents through Natural Language [81.73820295186727]
  We propose a formalism of procedures as programs, a powerful yet intuitive method of representing hierarchical procedural knowledge for agent command and control. We instantiate this framework on the IQA and ALFRED datasets for NL instruction following.
  arXiv  Detail & Related papers  (2021-09-16T20:36:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.