Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation

• URL: http://arxiv.org/abs/2604.10397v1
• Date: Sun, 12 Apr 2026 01:07:43 GMT
• Title: Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation
• Abstract summary: Video-based human-object interaction (HOI) understanding requires both detecting ongoing interactions and anticipating their future evolution.<n>We introduce DETAnt-HOI and HOI-DA, a pair-centric framework that jointly performs subject-object localization, present HOI detection, and future anticipation.<n>Experiments show consistent improvements in both detection and anticipation, with larger gains at longer horizons.
• Score: 18.693454975393703
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video-based human-object interaction (HOI) understanding requires both detecting ongoing interactions and anticipating their future evolution. However, existing methods usually treat anticipation as a downstream forecasting task built on externally constructed human-object pairs, limiting joint reasoning between detection and prediction. In addition, sparse keyframe annotations in current benchmarks can temporally misalign nominal future labels from actual future dynamics, reducing the reliability of anticipation evaluation. To address these issues, we introduce DETAnt-HOI, a temporally corrected benchmark derived from VidHOI and Action Genome for more faithful multi-horizon evaluation, and HOI-DA, a pair-centric framework that jointly performs subject-object localization, present HOI detection, and future anticipation by modeling future interactions as residual transitions from current pair states. Experiments show consistent improvements in both detection and anticipation, with larger gains at longer horizons. Our results highlight that anticipation is most effective when learned jointly with detection as a structural constraint on pair-level video representation learning. Benchmark and code will be publicly available.

Related papers

• From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video [62.898774619819676]
  Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them.<n>We introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, comprising approximately 233K samples across three domains.<n>We propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models forecasting as a cascaded where-to-how process.
  arXiv  Detail & Related papers  (2026-09-08T12:06:23Z)
• OFlow: Injecting Object-Aware Temporal Flow Matching for Robust Robotic Manipulation [55.56490813640669]
  VLA models typically act only on the current frame, while future prediction and object-aware reasoning are often learned in separate latent spaces.<n>We propose OFlow, a framework that addresses both limitations by unifying temporal foresight and object-aware reasoning in a shared semantic latent space.<n>Our method forecasts future latents with temporal flow matching, factorizes them into object-aware representations that emphasize physically relevant cues.
  arXiv  Detail & Related papers  (2026-04-20T06:38:01Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting [53.74101174559609]
  We propose a novel streaming-based motion forecasting framework that explicitly focuses on evolving scenes.<n>Our method incrementally processes incoming observation windows and leverages an instance-aware context streaming to maintain and update latent agent representations.<n>Our model achieves state-of-the-art performance in streaming inference on the Argoverse 2 multi-agent benchmark, while maintaining minimal latency, highlighting its suitability for real-world deployment.
  arXiv  Detail & Related papers  (2026-03-30T06:47:19Z)
• FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model [73.03346643967309]
  We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling.<n>FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information.<n>In the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors.
  arXiv  Detail & Related papers  (2026-03-11T12:39:55Z)
• UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection [18.25576487115016]
  This paper focuses on Human-Object Interaction (HOI) detection. It addresses the challenge of identifying and understanding the interactions between humans and objects within a given image or video frame. We propose a novel approach textscUAHOI, Uncertainty-aware Robust Human-Object Interaction Learning.
  arXiv  Detail & Related papers  (2024-08-14T10:06:39Z)
• Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation [17.4088244981231]
  Long-term action anticipation has become an important task for many applications such as autonomous driving and human-robot interaction. We propose a novel Gated Temporal Diffusion (GTD) network that models the uncertainty of both the observation and the future predictions. Our model achieves state-of-the-art results on the Breakfast, Assembly101 and 50Salads datasets in both deterministic settings.
  arXiv  Detail & Related papers  (2024-07-16T17:48:05Z)
• Bidirectional Progressive Transformer for Interaction Intention Anticipation [20.53329698350243]
  We introduce a Bidirectional Progressive mechanism into the anticipation of interaction intention. We employ a Trajectory Unit and a C-VAE to introduce appropriate uncertainty to trajectories and interaction hotspots. Our method achieves state-of-the-art results on three benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-09T05:22:18Z)
• Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos [22.81433371521832]
  We propose Diff-IP2D to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. Our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol.
  arXiv  Detail & Related papers  (2024-05-07T14:51:05Z)
• Neural Interaction Energy for Multi-Agent Trajectory Prediction [55.098754835213995]
  We introduce a framework called Multi-Agent Trajectory prediction via neural interaction Energy (MATE) MATE assesses the interactive motion of agents by employing neural interaction energy. To bolster temporal stability, we introduce two constraints: inter-agent interaction constraint and intra-agent motion constraint.
  arXiv  Detail & Related papers  (2024-04-25T12:47:47Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• Parallel Reasoning Network for Human-Object Interaction Detection [53.422076419484945]
  We propose a new transformer-based method named Parallel Reasoning Network(PR-Net) PR-Net constructs two independent predictors for instance-level localization and relation-level understanding. Our PR-Net has achieved competitive results on HICO-DET and V-COCO benchmarks.
  arXiv  Detail & Related papers  (2023-01-09T17:00:34Z)
• Inductive Attention for Video Action Anticipation [16.240254363118016]
  We propose an inductive attention model, dubbed IAM, which leverages the current prior predictions as the query to infer future action. Our method consistently outperforms the state-of-the-art anticipation models on multiple large-scale egocentric video datasets.
  arXiv  Detail & Related papers  (2022-12-17T09:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.