Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection

• URL: http://arxiv.org/abs/2604.09164v1
• Date: Fri, 10 Apr 2026 09:51:15 GMT
• Title: Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection
• Abstract summary: Temporal human action detection aims to identify and localize action segments within untrimmed videos.<n>Prior architectures like CNN and Transformer models struggle with feature redundancy and degraded global dependency modeling capabilities.<n>This research constructs a novel framework for video human action detection using State Space Models.
• Score: 6.008736333630327
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temporal human action detection aims to identify and localize action segments within untrimmed videos, serving as a pivotal task in video understanding. Despite the progress achieved by prior architectures like CNN and Transformer models, these continue to struggle with feature redundancy and degraded global dependency modeling capabilities when applied to long video sequences. These limitations severely constrain their scalability in real-world video analysis. State Space Models (SSMs) offer a promising alternative with linear long-term modeling and robust global temporal reasoning capabilities. Rethinking the application of SSMs in temporal modeling, this research constructs a novel framework for video human action detection. Specifically, we introduce the Efficient Spatial-Temporal Focal (ESTF) Adapter into the pre-trained layers. This module integrates the advantages of our proposed Temporal Boundary-aware SSM(TB-SSM) for temporal feature modeling with efficient processing of spatial features. We perform comprehensive and quantitative analyses across multiple benchmarks, comparing our proposed method against previous SSM-based and other structural methods. Extensive experiments demonstrate that our improved strategy significantly enhances both localization performance and robustness, validating the effectiveness of our proposed method.

Related papers

• FRACTAL: SSM with Fractional Recurrent Architecture for Computational Temporal Analysis of Long Sequences [13.320173396455226]
  This paper proposes a Fractional Recurrent Architecture for Computational Temporal Analysis of Long sequences (FRACTAL)<n>By deriving projection operators with analytically characterized spectral properties and a tunable index, the proposed method sensitivity to recent signal perturbations while preserving the spectral structure that encodes fractional scale-invariant memory dynamics.<n>FRACTAL achieves an average score of 87.11% on the Long Range Arena benchmark, including 61.85% on the ListOps task, outperforming the S5 model.
  arXiv  Detail & Related papers  (2026-05-09T09:38:44Z)
• DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization [23.899829316926724]
  Temporal Forgery Localization (TFL) aims to precisely identify manipulated segments in video and audio, offering strong interpretability for security and forensics.<n>While recent State Space Models (SSMs) show promise in precise temporal reasoning, their use in TFL is hindered by ambiguous boundaries, sparse forgeries, and limited long-range modeling.<n>We propose DeformTrace, which enhances SSMs with deformable dynamics and relay mechanisms to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T07:19:50Z)
• Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation [50.05866669110754]
  Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames.<n>We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules.<n>We show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation.
  arXiv  Detail & Related papers  (2025-12-19T15:15:58Z)
• UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling [53.199942923818206]
  Point cloud videos capture 3D motion while reducing the effects of lighting and viewpoint variations, making them highly effective for recognizing subtle and continuous human actions.<n> Selective State Space Models (SSMs) have shown good performance in sequence modeling with linear complexity.<n>We propose the Unified Spatio-Temporal State Space Model (UST-SSM), which extends the latest advancements in SSMs to point cloud videos.
  arXiv  Detail & Related papers  (2025-08-20T10:46:01Z)
• Artificial Intelligence-Based Multiscale Temporal Modeling for Anomaly Detection in Cloud Services [10.421371572062595]
  This study proposes an anomaly detection method based on the Transformer architecture with integrated multiscale feature perception.<n>The proposed method outperforms mainstream baseline models in key metrics, including precision, recall, AUC, and F1-score.
  arXiv  Detail & Related papers  (2025-08-20T07:52:36Z)
• Long-Context State-Space Video World Models [66.28743632951218]
  We propose a novel architecture leveraging state-space models (SSMs) to extend temporal memory without compromising computational efficiency.<n>Central to our design is a block-wise SSM scanning scheme, which strategically trades off spatial consistency for extended temporal memory.<n>Experiments on Memory Maze and Minecraft datasets demonstrate that our approach surpasses baselines in preserving long-range memory.
  arXiv  Detail & Related papers  (2025-05-26T16:12:41Z)
• Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency [59.05753942719665]
  We propose a novel temporal robustness benchmark (TemRobBench) to assess the robustness of models.<n>We evaluate 16 mainstream LMMs and find that they exhibit over-reliance on prior knowledge and textual context in adversarial environments.<n>We design panoramic direct preference optimization (PanoDPO) to encourage LMMs to incorporate both visual and linguistic feature preferences simultaneously.
  arXiv  Detail & Related papers  (2025-05-20T14:18:56Z)
• MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction [0.0]
  We introduce the Multi-Attention Unit (MAUCell) which combines Generative Adrative Networks (GANs) and attention mechanisms to improve video prediction.<n>The new design system maintains equilibrium between temporal continuity and spatial accuracy to deliver reliable video prediction.
  arXiv  Detail & Related papers  (2025-01-28T14:52:10Z)
• Embedded feature selection in LSTM networks with multi-objective evolutionary ensemble learning for time series forecasting [49.1574468325115]
  We present a novel feature selection method embedded in Long Short-Term Memory networks. Our approach optimize the weights and biases of the LSTM in a partitioned manner. Experimental evaluations on air quality time series data from Italy and southeast Spain demonstrate that our method substantially improves the ability generalization of conventional LSTMs.
  arXiv  Detail & Related papers  (2023-12-29T08:42:10Z)
• Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
  We propose LEAPS, an architecture-independent method for synthesizing videos from internal models. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
  arXiv  Detail & Related papers  (2023-03-17T12:55:22Z)
• Motion-aware Memory Network for Fast Video Salient Object Detection [15.967509480432266]
  We design a space-time memory (STM)-based network, which extracts useful temporal information of the current frame from adjacent frames as the temporal branch of VSOD. In the encoding stage, we generate high-level temporal features by using high-level features from the current and its adjacent frames. In the decoding stage, we propose an effective fusion strategy for spatial and temporal branches. The proposed model does not require optical flow or other preprocessing, and can reach a speed of nearly 100 FPS during inference.
  arXiv  Detail & Related papers  (2022-08-01T15:56:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.