Unified Recurrence Modeling for Video Action Anticipation

• URL: http://arxiv.org/abs/2206.01009v1
• Date: Thu, 2 Jun 2022 12:16:44 GMT
• Title: Unified Recurrence Modeling for Video Action Anticipation
• Authors: Tsung-Ming Tai, Giuseppe Fiameni, Cheng-Kuang Lee, Simon See, Oswald Lanz
• Abstract summary: We propose a unified recurrence modeling for video action anticipation via message passing framework. Our proposed method outperforms previous works on the large-scale EPIC-Kitchen dataset.
• Score: 16.240254363118016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Forecasting future events based on evidence of current conditions is an innate skill of human beings, and key for predicting the outcome of any decision making. In artificial vision for example, we would like to predict the next human action before it happens, without observing the future video frames associated to it. Computer vision models for action anticipation are expected to collect the subtle evidence in the preamble of the target actions. In prior studies recurrence modeling often leads to better performance, the strong temporal inference is assumed to be a key element for reasonable prediction. To this end, we propose a unified recurrence modeling for video action anticipation via message passing framework. The information flow in space-time can be described by the interaction between vertices and edges, and the changes of vertices for each incoming frame reflects the underlying dynamics. Our model leverages self-attention as the building blocks for each of the message passing functions. In addition, we introduce different edge learning strategies that can be end-to-end optimized to gain better flexibility for the connectivity between vertices. Our experimental results demonstrate that our proposed method outperforms previous works on the large-scale EPIC-Kitchen dataset.

Related papers

• E-Motion: Future Motion Simulation via Event Sequence Diffusion [86.80533612211502]
  Event-based sensors may potentially offer a unique opportunity to predict future motion with a level of detail and precision previously unachievable. We propose to integrate the strong learning capacity of the video diffusion model with the rich motion information of an event camera as a motion simulation framework. Our findings suggest a promising direction for future research in enhancing the interpretative power and predictive accuracy of computer vision systems.
  arXiv  Detail & Related papers  (2024-10-11T09:19:23Z)
• From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation [30.161471749050833]
  We propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR) ARR decomposes the action anticipation task into action recognition and reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP) In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder.
  arXiv  Detail & Related papers  (2024-08-05T18:38:29Z)
• Inductive Attention for Video Action Anticipation [16.240254363118016]
  We propose an inductive attention model, dubbed IAM, which leverages the current prior predictions as the query to infer future action. Our method consistently outperforms the state-of-the-art anticipation models on multiple large-scale egocentric video datasets.
  arXiv  Detail & Related papers  (2022-12-17T09:51:17Z)
• Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment [72.50906475214457]
  The goal of sequential event prediction is to estimate the next event based on a sequence of historical events. In practice, the next-event prediction models are trained with sequential data collected at one time. We propose a framework with hierarchical branching structures for learning context-specific representations.
  arXiv  Detail & Related papers  (2022-10-24T07:54:13Z)
• Finding Islands of Predictability in Action Forecasting [7.215559809521136]
  We show that future action sequences are more accurately modeled with variable, rather than one, levels of abstraction. We propose a combination Bayesian neural network and hierarchical convolutional segmentation model to both accurately predict future actions and optimally select abstraction levels.
  arXiv  Detail & Related papers  (2022-10-13T21:01:16Z)
• Reinforcement Learning with Action-Free Pre-Training from Videos [95.25074614579646]
  We introduce a framework that learns representations useful for understanding the dynamics via generative pre-training on videos. Our framework significantly improves both final performances and sample-efficiency of vision-based reinforcement learning.
  arXiv  Detail & Related papers  (2022-03-25T19:44:09Z)
• RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting [34.54878390622877]
  We propose a generic motion forecasting framework with dynamic key information selection and ranking based on a hybrid attention mechanism. The framework is instantiated to handle multi-agent trajectory prediction and human motion forecasting tasks. We validate the framework on both synthetic simulations and motion forecasting benchmarks in different domains.
  arXiv  Detail & Related papers  (2021-08-03T06:30:30Z)
• Panoptic Segmentation Forecasting [71.75275164959953]
  Our goal is to forecast the near future given a set of recent observations. We think this ability to forecast, i.e., to anticipate, is integral for the success of autonomous agents. We develop a two-component model: one component learns the dynamics of the background stuff by anticipating odometry, the other one anticipates the dynamics of detected things.
  arXiv  Detail & Related papers  (2021-04-08T17:59:16Z)
• Instance-Aware Predictive Navigation in Multi-Agent Environments [93.15055834395304]
  We propose an Instance-Aware Predictive Control (IPC) approach, which forecasts interactions between agents as well as future scene structures. We adopt a novel multi-instance event prediction module to estimate the possible interaction among agents in the ego-centric view. We design a sequential action sampling strategy to better leverage predicted states on both scene-level and instance-level.
  arXiv  Detail & Related papers  (2021-01-14T22:21:25Z)
• Dynamic Inference: A New Approach Toward Efficient Video Action Recognition [69.9658249941149]
  Action recognition in videos has achieved great success recently, but it remains a challenging task due to the massive computational cost. We propose a general dynamic inference idea to improve inference efficiency by leveraging the variation in the distinguishability of different videos.
  arXiv  Detail & Related papers  (2020-02-09T11:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.