MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows

• URL: http://arxiv.org/abs/2506.01119v1
• Date: Sun, 01 Jun 2025 18:53:27 GMT
• Title: MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows
• Authors: Hong Nguyen, Dung Tran, Hieu Hoang, Phong Nguyen, Shrikanth Narayanan,
• Abstract summary: MOOSE is a novel temporally-centric video encoder that integrates optical flow with spatial embeddings to model temporal information efficiently.<n>Unlike prior models, MOOSE takes advantage of rich, widely available pre-trained visual and optical flow encoders instead of training video models from scratch.
• Score: 21.969862773424314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many motion-centric video analysis tasks, such as atomic actions, detecting atypical motor behavior in individuals with autism, or analyzing articulatory motion in real-time MRI of human speech, require efficient and interpretable temporal modeling. Capturing temporal dynamics is a central challenge in video analysis, often requiring significant computational resources and fine-grained annotations that are not widely available. This paper presents MOOSE (Motion Flow Over Spatial Space), a novel temporally-centric video encoder explicitly integrating optical flow with spatial embeddings to model temporal information efficiently, inspired by human perception of motion. Unlike prior models, MOOSE takes advantage of rich, widely available pre-trained visual and optical flow encoders instead of training video models from scratch. This significantly reduces computational complexity while enhancing temporal interpretability. Our primary contributions includes (1) proposing a computationally efficient temporally-centric architecture for video understanding (2) demonstrating enhanced interpretability in modeling temporal dynamics; and (3) achieving state-of-the-art performance on diverse benchmarks, including clinical, medical, and standard action recognition datasets, confirming the broad applicability and effectiveness of our approach.

Related papers

• VLM4D: Towards Spatiotemporal Awareness in Vision Language Models [66.833085504228]
  We introduce V4DLM, the first benchmark specifically designed to evaluate visual language models (VLMs)<n>Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs.<n>We identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models.
  arXiv  Detail & Related papers  (2025-08-04T06:06:06Z)
• FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation [51.110607281391154]
  FlowMo is a training-free guidance method for enhancing motion coherence in text-to-video models.<n>It estimates motion coherence by measuring the patch-wise variance across the temporal dimension and guides the model to reduce this variance dynamically during sampling.
  arXiv  Detail & Related papers  (2025-06-01T19:55:33Z)
• Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning [73.7808110878037]
  This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++)<n>By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases.<n>Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T13:06:01Z)
• Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilities [12.409087198219693]
  We introduce a neuromorphic optical flow approach that addresses delay bottlenecks by encoding temporal information directly in a synaptic transistor array.<n>Compared to conventional spatial-only optical flow methods, our system offers the spatial-temporal consistency of motion information.<n>In software benchmarks, our system outperforms state-of-the-art algorithms with a 400% speedup.
  arXiv  Detail & Related papers  (2024-09-10T10:59:32Z)
• Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity [13.04953215936574]
  We propose a two-stage model named Mind-Animator to reconstruct human dynamic vision from brain activity.<n>During the fMRI-to-feature stage, we decouple semantic, structure, and motion features from fMRI.<n>In the feature-to-video stage, these features are integrated into videos using an inflated Stable Diffusion.
  arXiv  Detail & Related papers  (2024-05-06T08:56:41Z)
• Spatio-Temporal Branching for Motion Prediction using Motion Increments [55.68088298632865]
  Human motion prediction (HMP) has emerged as a popular research topic due to its diverse applications. Traditional methods rely on hand-crafted features and machine learning techniques. We propose a noveltemporal-temporal branching network using incremental information for HMP.
  arXiv  Detail & Related papers  (2023-08-02T12:04:28Z)
• TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognition [1.5237530964650965]
  We propose a novel approach that uses attention more effectively and does not require hand-crafted constraints.<n>TuNeS is an efficient and simple temporal model that incorporates self-attention at the core of a convolutional U-Net structure.<n>TuNeS achieves state-of-the-art results on the Cholec80 dataset.
  arXiv  Detail & Related papers  (2023-07-19T14:10:55Z)
• Modelling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network [1.9458156037869137]
  We propose an image-computable model of human motion perception by bridging the gap between biological and computer vision models. This model architecture aims to capture the computations in V1-MT, the core structure for motion perception in the biological visual system. In silico neurophysiology reveals that our model's unit responses are similar to mammalian neural recordings regarding motion pooling and speed tuning.
  arXiv  Detail & Related papers  (2023-05-16T04:16:07Z)
• Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
  We propose LEAPS, an architecture-independent method for synthesizing videos from internal models. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
  arXiv  Detail & Related papers  (2023-03-17T12:55:22Z)
• A Prospective Study on Sequence-Driven Temporal Sampling and Ego-Motion Compensation for Action Recognition in the EPIC-Kitchens Dataset [68.8204255655161]
  Action recognition is one of the top-challenging research fields in computer vision. ego-motion recorded sequences have become of important relevance. The proposed method aims to cope with it by estimating this ego-motion or camera motion.
  arXiv  Detail & Related papers  (2020-08-26T14:44:45Z)
• MotionSqueeze: Neural Motion Feature Learning for Video Understanding [46.82376603090792]
  Motion plays a crucial role in understanding videos and most state-of-the-art neural models for video classification incorporate motion information. In this work, we replace external and heavy computation of optical flows with internal and light-weight learning of motion features. We demonstrate that the proposed method provides a significant gain on four standard benchmarks for action recognition with only a small amount of additional cost.
  arXiv  Detail & Related papers  (2020-07-20T08:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.