Learning When to Listen: Gated Affect Fusion for Human Motion Prediction

• URL: http://arxiv.org/abs/2607.00296v1
• Date: Wed, 01 Jul 2026 00:47:02 GMT
• Title: Learning When to Listen: Gated Affect Fusion for Human Motion Prediction
• Abstract summary: We present a systematic study investigating the utility and temporal limitations of affect-conditioned forecasting in-the-wild.<n>We introduce the Gated Affect Transformer (GAT) to dynamically regulate cross-modal information flow.<n>Our findings provide empirical evidence that facial affect should be regarded as a complementary behavioral cue.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human motion forecasting in unconstrained real-world videos remains challenging due to the ambiguity of future behaviors and the presence of noisy multimodal observations. While facial affect potentially provides complementary behavioral cues, its practical utility and mechanistic boundaries within motion forecasting frameworks remain poorly understood. In this work, we present a systematic study investigating the utility and temporal limitations of affect-conditioned forecasting in-the-wild. We establish a rigorous multimodal pipeline combining MediaPipe body pose trajectories with HSEmotion facial affect representations, and introduce the Gated Affect Transformer (GAT) to dynamically regulate cross-modal information flow. Through extensive multi-horizon evaluations under a strict subject-wise protocol, we demonstrate that naive early cross-modal concatenation consistently degrades forecasting accuracy relative to pose-only baselines. Conversely, our proposed gating mechanism stabilizes cross-modal integration by adaptively controlling the affective stream. Crucially, controlled counterfactual experiments using shuffled and randomized affect inputs reveal that the learned gate successfully suppresses unstructured cross-modal noise while remaining responsive to plausible affective signals. Furthermore, our empirical results indicate that facial affect features provide bounded, horizon-dependent predictive cues strictly within short-to-medium windows (e.g., 30 frames), whereas long-term trajectories remain predominantly governed by intrinsic kinematic continuity. Our findings provide empirical evidence that facial affect should be regarded as a complementary behavioral cue rather than a dominant driver of future motion, offering practical guidance for selective multimodal fusion in unconstrained human motion forecasting.

Related papers

• SegDiff: Segmented Trajectory Diffusion for Consistent and Adaptive Robot Manipulation [65.81476183180527]
  We introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms.<n>SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose.<n>SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios.
  arXiv  Detail & Related papers  (2026-07-13T02:48:03Z)
• SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics [49.771521708359955]
  SurgVista is a surgical world model that mitigates spatial interaction incoherence and temporal fidelity collapse.<n>It consistently outperforms state-of-the-art methods across visual quality, temporal consistency, and interaction fidelity.
  arXiv  Detail & Related papers  (2026-06-18T07:47:28Z)
• Emotion-Conditioned Short-Horizon Human Pose Forecasting with a Lightweight Predictive World Model [0.0]
  Short-term human pose prediction plays a crucial role in interactive systems, assistive robots, and emotion-aware human-computer interaction.<n>This paper investigates whether facial expression-derived emotion embeddings can provide auxiliary conditional signals for short-term pose prediction.
  arXiv  Detail & Related papers  (2026-04-26T04:56:45Z)
• F2F-AP: Flow-to-Future Asynchronous Policy for Real-time Dynamic Manipulation [62.06267255986041]
  Asynchronous inference has emerged as a prevalent paradigm in robotic manipulation.<n>In this paper, we propose a novel framework that leverages predicted object flow to synthesize future observations.<n>Our approach significantly enhances responsiveness and success rates in complex dynamic manipulation tasks.
  arXiv  Detail & Related papers  (2026-04-02T17:57:15Z)
• \ extsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Neural Interaction Energy for Multi-Agent Trajectory Prediction [55.098754835213995]
  We introduce a framework called Multi-Agent Trajectory prediction via neural interaction Energy (MATE) MATE assesses the interactive motion of agents by employing neural interaction energy. To bolster temporal stability, we introduce two constraints: inter-agent interaction constraint and intra-agent motion constraint.
  arXiv  Detail & Related papers  (2024-04-25T12:47:47Z)
• Motion-Scenario Decoupling for Rat-Aware Video Position Prediction: Strategy and Benchmark [49.58762201363483]
  We introduce RatPose, a bio-robot motion prediction dataset constructed by considering the influence factors of individuals and environments. We propose a Dual-stream Motion-Scenario Decoupling framework that effectively separates scenario-oriented and motion-oriented features. We demonstrate significant performance improvements of the proposed textitDMSD framework on different difficulty-level tasks.
  arXiv  Detail & Related papers  (2023-05-17T14:14:31Z)
• BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction [26.306489700180627]
  We present BeLFusion, a model that leverages latent diffusion models in human motion prediction (HMP) to sample from a latent space where behavior is disentangled from pose and motion. Thanks to our behavior coupler's ability to transfer sampled behavior to ongoing motion, BeLFusion's predictions display a variety of behaviors that are significantly more realistic than the state of the art.
  arXiv  Detail & Related papers  (2022-11-25T18:59:03Z)
• Prediction by Anticipation: An Action-Conditional Prediction Method based on Interaction Learning [23.321627835039934]
  We propose prediction by anticipation, which views interaction in terms of a latent probabilistic generative process. Under this view, consecutive data frames can be factorized into sequential samples from an action-conditional distribution. Our proposed prediction model, variational Bayesian in nature, is trained to maximize the evidence lower bound (ELBO) of this conditional distribution.
  arXiv  Detail & Related papers  (2020-12-25T01:39:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.