MuSteerNet: Human Reaction Generation from Videos via Observation-Reaction Mutual Steering

• URL: http://arxiv.org/abs/2603.20187v1
• Date: Fri, 20 Mar 2026 17:59:06 GMT
• Title: MuSteerNet: Human Reaction Generation from Videos via Observation-Reaction Mutual Steering
• Abstract summary: Video-driven human reaction generation aims to synthesize 3D human motions that directly react to observed video sequences.<n>Existing methods often fail to effectively leverage video inputs to steer human reaction synthesis.<n>We propose MuSteerNet, a framework that generates 3D human reactions from videos via observation-reaction mutual steering.
• Score: 81.50722520596652
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video-driven human reaction generation aims to synthesize 3D human motions that directly react to observed video sequences, which is crucial for building human-like interactive AI systems. However, existing methods often fail to effectively leverage video inputs to steer human reaction synthesis, resulting in reaction motions that are mismatched with the content of video sequences. We reveal that this limitation arises from a severe relational distortion between visual observations and reaction types. In light of this, we propose MuSteerNet, a simple yet effective framework that generates 3D human reactions from videos via observation-reaction mutual steering. Specifically, we first propose a Prototype Feedback Steering mechanism to mitigate relational distortion by refining visual observations with a gated delta-rectification modulator and a relational margin constraint, guided by prototypical vectors learned from human reactions. We then introduce Dual-Coupled Reaction Refinement that fully leverages rectified visual cues to further steer the refinement of generated reaction motions, thereby effectively improving reaction quality and enabling MuSteerNet to achieve competitive performance. Extensive experiments and ablation studies validate the effectiveness of our method. Code coming soon: https://github.com/zhouyuan888888/MuSteerNet.

Related papers

• Learning Reactive Human Motion Generation from Paired Interaction Data Using Transformer-Based Models [0.0]
  This study addresses the problem of generating the motion of one person based on the motion of another in interaction scenarios.<n>We implement and compare three models: a simple Transformer, iTransformer, and Crossformer.<n> Experimental results show that the simple Transformer can generate plausible interaction-aware motions without suffering from posture collapse.
  arXiv  Detail & Related papers  (2026-04-24T02:27:13Z)
• EgoReAct: Egocentric Video-Driven 3D Human Reaction Generation [84.37917777533963]
  We present EgoReAct, the first framework that generates 3D-aligned human reaction motions from egocentric video streams in real-time.<n>EgoReAct achieves remarkably higher realism, spatial consistency, and generation efficiency compared with prior methods.
  arXiv  Detail & Related papers  (2025-12-28T06:44:05Z)
• MoReact: Generating Reactive Motion from Textual Descriptions [57.642436102978245]
  MoReact is a diffusion-based method designed to disentangle the generation of global trajectories and local motions sequentially.<n>Our experiments, utilizing data adapted from a two-person motion dataset, demonstrate the efficacy of our approach.
  arXiv  Detail & Related papers  (2025-09-28T14:31:41Z)
• ARFlow: Human Action-Reaction Flow Matching with Physical Guidance [36.052383171690614]
  Action-Reaction Flow Matching (ARFlow) is a novel framework that establishes direct action-to-reaction mappings.<n>Our approach introduces a physical guidance mechanism specifically designed for Flow Matching (FM) that effectively prevents body penetration artifacts during sampling.
  arXiv  Detail & Related papers  (2025-03-21T09:41:24Z)
• HERO: Human Reaction Generation from Videos [54.602947113980655]
  HERO is a framework for Human rEaction geneRation from videOs.<n> HERO considers both global and frame-level local representations of the video to extract the interaction intention.<n>Local visual representations are continuously injected into the model to maximize the exploitation of the dynamic properties inherent in videos.
  arXiv  Detail & Related papers  (2025-03-11T10:39:32Z)
• ReGenNet: Towards Human Action-Reaction Synthesis [87.57721371471536]
  We analyze the asymmetric, dynamic, synchronous, and detailed nature of human-human interactions. We propose the first multi-setting human action-reaction benchmark to generate human reactions conditioned on given human actions.
  arXiv  Detail & Related papers  (2024-03-18T15:33:06Z)
• ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions [66.87211993793807]
  We present ReMoS, a denoising diffusion based model that synthesizes full body motion of a person in two person interaction scenario. We demonstrate ReMoS across challenging two person scenarios such as pair dancing, Ninjutsu, kickboxing, and acrobatics. We also contribute the ReMoCap dataset for two person interactions containing full body and finger motions.
  arXiv  Detail & Related papers  (2023-11-28T18:59:52Z)
• LARNet: Latent Action Representation for Human Action Synthesis [3.3454373538792552]
  We present LARNet, a novel end-to-end approach for generating human action videos. We learn action dynamics in latent space avoiding the need of a driving video during inference. We evaluate the proposed approach on four real-world human action datasets.
  arXiv  Detail & Related papers  (2021-10-21T05:04:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.