Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation

• URL: http://arxiv.org/abs/2407.11954v1
• Date: Tue, 16 Jul 2024 17:48:05 GMT
• Title: Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation
• Authors: Olga Zatsarynna, Emad Bahrami, Yazan Abu Farha, Gianpiero Francesca, Juergen Gall,
• Abstract summary: Long-term action anticipation has become an important task for many applications such as autonomous driving and human-robot interaction. We propose a novel Gated Temporal Diffusion (GTD) network that models the uncertainty of both the observation and the future predictions. Our model achieves state-of-the-art results on the Breakfast, Assembly101 and 50Salads datasets in both deterministic settings.
• Score: 17.4088244981231
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-term action anticipation has become an important task for many applications such as autonomous driving and human-robot interaction. Unlike short-term anticipation, predicting more actions into the future imposes a real challenge with the increasing uncertainty in longer horizons. While there has been a significant progress in predicting more actions into the future, most of the proposed methods address the task in a deterministic setup and ignore the underlying uncertainty. In this paper, we propose a novel Gated Temporal Diffusion (GTD) network that models the uncertainty of both the observation and the future predictions. As generator, we introduce a Gated Anticipation Network (GTAN) to model both observed and unobserved frames of a video in a mutual representation. On the one hand, using a mutual representation for past and future allows us to jointly model ambiguities in the observation and future, while on the other hand GTAN can by design treat the observed and unobserved parts differently and steer the information flow between them. Our model achieves state-of-the-art results on the Breakfast, Assembly101 and 50Salads datasets in both stochastic and deterministic settings. Code: https://github.com/olga-zats/GTDA .

Related papers

• FIMP: Future Interaction Modeling for Multi-Agent Motion Prediction [18.10147252674138]
  We propose Future Interaction modeling for Motion Prediction (FIMP), which captures potential future interactions in an end-to-end manner. Experiments show that our future interaction modeling improves the performance remarkably, leading to superior performance on the Argoverse motion forecasting benchmark.
  arXiv  Detail & Related papers  (2024-01-29T14:41:55Z)
• DiffAnt: Diffusion Models for Action Anticipation [12.022815981853071]
  Anticipating future actions is inherently uncertain. Given an observed video segment containing ongoing actions, multiple subsequent actions can plausibly follow. In this work, we rethink action anticipation from a generative view, employing diffusion models to capture different possible future actions. Our code and trained models will be published on GitHub.
  arXiv  Detail & Related papers  (2023-11-27T16:40:09Z)
• Inductive Attention for Video Action Anticipation [16.240254363118016]
  We propose an inductive attention model, dubbed IAM, which leverages the current prior predictions as the query to infer future action. Our method consistently outperforms the state-of-the-art anticipation models on multiple large-scale egocentric video datasets.
  arXiv  Detail & Related papers  (2022-12-17T09:51:17Z)
• Future Transformer for Long-term Action Anticipation [33.771374384674836]
  We propose an end-to-end attention model for action anticipation, dubbed Future Transformer (FUTR) Unlike the previous autoregressive models, the proposed method learns to predict the whole sequence of future actions in parallel decoding. We evaluate our method on two standard benchmarks for long-term action anticipation, Breakfast and 50 Salads, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2022-05-27T14:47:43Z)
• StretchBEV: Stretching Future Instance Prediction Spatially and Temporally [0.0]
  In self-driving cars, predicting future in terms of location and motion of all the agents around the vehicle is a crucial requirement for planning. Recently, a new joint formulation of perception and prediction has emerged by fusing rich sensory information perceived from multiple cameras into a compact bird's-eye view representation to perform prediction. However, the quality of future predictions degrades over time while extending to longer time horizons due to multiple plausible predictions. In this work, we address this inherent uncertainty in future predictions with a temporal model.
  arXiv  Detail & Related papers  (2022-03-25T13:28:44Z)
• Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction [63.62263239934777]
  We conduct an indepth study on various pose representations with a focus on their effects on the motion prediction task. We propose a novel RNN architecture termed AHMR (Attentive Hierarchical Motion Recurrent network) for motion prediction. Our approach outperforms the state-of-the-art methods in short-term prediction and achieves much enhanced long-term prediction proficiency.
  arXiv  Detail & Related papers  (2021-12-30T10:45:22Z)
• Dyadic Human Motion Prediction [119.3376964777803]
  We introduce a motion prediction framework that explicitly reasons about the interactions of two observed subjects. Specifically, we achieve this by introducing a pairwise attention mechanism that models the mutual dependencies in the motion history of the two subjects. This allows us to preserve the long-term motion dynamics in a more realistic way and more robustly predict unusual and fast-paced movements.
  arXiv  Detail & Related papers  (2021-12-01T10:30:40Z)
• Long Term Motion Prediction Using Keyposes [122.22758311506588]
  We argue that, to achieve long term forecasting, predicting human pose at every time instant is unnecessary. We call such poses "keyposes", and approximate complex motions by linearly interpolating between subsequent keyposes. We show that learning the sequence of such keyposes allows us to predict very long term motion, up to 5 seconds in the future.
  arXiv  Detail & Related papers  (2020-12-08T20:45:51Z)
• From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting [54.273455592965355]
  Uncertainty in future trajectories stems from two sources: (a) sources known to the agent but unknown to the model, such as long term goals and (b)sources that are unknown to both the agent & the model, such as intent of other agents & irreducible randomness indecisions. We model the epistemic un-certainty through multimodality in long term goals and the aleatoric uncertainty through multimodality in waypoints& paths. To exemplify this dichotomy, we also propose a novel long term trajectory forecasting setting, with prediction horizons upto a minute, an order of magnitude longer than prior works.
  arXiv  Detail & Related papers  (2020-12-02T21:01:29Z)
• Adversarial Generative Grammars for Human Activity Prediction [141.43526239537502]
  We propose an adversarial generative grammar model for future prediction. Our grammar is designed so that it can learn production rules from the data distribution. Being able to select multiple production rules during inference leads to different predicted outcomes.
  arXiv  Detail & Related papers  (2020-08-11T17:47:53Z)
• Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
  We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
  arXiv  Detail & Related papers  (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.