Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders

• URL: http://arxiv.org/abs/2604.03919v1
• Date: Sun, 05 Apr 2026 01:08:19 GMT
• Title: Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders
• Abstract summary: We present the first systematic study of Sparseencoencoders (SAEs) on video representations.<n>Standard SAEs decompose into interpretable, monosemantic features but destroy temporal coherence.<n>TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%.
• Score: 3.3192479135000426
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The contrastive loss weight controls a tunable trade-off between reconstruction and temporal coherence. A systematic ablation on two backbones and two datasets shows that different configurations excel at different goals: reconstruction fidelity, temporal coherence, action discrimination, or interpretability. Contrastive SAE features improve action classification by +3.9% over raw features and text-video retrieval by up to 2.8xR@1. A cross-backbone analysis reveals that standard monosemanticity metrics contain a backbone-alignment artifact: both DINOv2 and VideoMAE produce equally monosemantic features under neutral (CLIP) similarity. Causal ablation confirms that contrastive training concentrates predictive signal into a small number of identifiable features.

Related papers

• On the Adversarial Robustness of Remote Sensing Semantic Change Detection [19.4097466391406]
  Semantic change detection (SCD) is a bitemporal dense-prediction task that jointly identifies changed regions and their semantic states before and after change.<n>We present a task-specific evaluation framework that separates output-side attack objectives from input-side temporal perturbation access.<n>Experiments show that final semantic-change predictions can be severely corrupted even when binary change localization remains comparatively stable.
  arXiv  Detail & Related papers  (2026-08-15T15:02:42Z)
• CVA: Context-aware Video-text Alignment for Video Temporal Grounding [16.546053936240362]
  We propose Context-aware Video-text Alignment (CVA), a novel framework to achieve temporally sensitive video-text alignment.<n>Our framework is built on three key components. First, we propose Query-aware Context Diversification (QCD), a new data augmentation strategy that ensures only semantically unrelated content is mixed in.<n>Second, we introduce the Context-invariant Boundary Discrimination (CBD) loss, a contrastive loss that enforces semantic consistency at challenging temporal boundaries.<n>Third, we introduce the Context-enhanced Transformer (CTE), a hierarchical architecture that combines windowed self-attention with bidirectional cross
  arXiv  Detail & Related papers  (2026-03-26T01:53:00Z)
• PROBE: Diagnosing Residual Concept Capacity in Erased Text-to-Video Diffusion Models [9.957850061310054]
  Concept techniques for text-to-video (T2V) diffusion models report substantial suppression of sensitive content.<n>We introduce PROBE, a diagnostic protocol that quantifies the textitre potential erased concepts in T2V models.
  arXiv  Detail & Related papers  (2026-03-23T04:01:13Z)
• Multimodal Quantitative Measures for Multiparty Behaviour Evaluation [6.709251546882382]
  We introduce a unified, intervention-driven framework for objective assessment of multiparty social behaviour in skeletal motion data.<n>We validate metric sensitivity through three theory-driven perturbations.<n>Mixed-effects analyses reveal predictable, joint-independent shifts.
  arXiv  Detail & Related papers  (2025-08-01T13:46:12Z)
• SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders [7.618223798662929]
  We propose SA-DVAE -- Semantic Alignment via Disentangled Variational Autoencoders. We implement this idea via a pair of modality-specific variational autoencoders coupled with a total correction penalty. Experiments show that SA-DAVE produces improved performance over existing methods.
  arXiv  Detail & Related papers  (2024-07-18T12:35:46Z)
• Low-Light Video Enhancement via Spatial-Temporal Consistent Decomposition [52.89441679581216]
  Low-Light Video Enhancement (LLVE) seeks to restore dynamic or static scenes plagued by severe invisibility and noise.<n>We present an innovative video decomposition strategy that incorporates view-independent and view-dependent components.<n>Our framework consistently outperforms existing methods, establishing a new SOTA performance.
  arXiv  Detail & Related papers  (2024-05-24T15:56:40Z)
• A Generically Contrastive Spatiotemporal Representation Enhancement for 3D Skeleton Action Recognition [10.403751563214113]
  We propose a Contrastive Spatiotemporal Representation Enhancement (CSRE) framework to obtain more discriminative representations from the sequences.<n>Specifically, we decompose the representation into spatial- and temporal-specific features to explore fine-grained motion patterns.<n>To explicitly exploit the latent data distributions, we employ the attentive features to contrastive learning, which models the cross-sequence semantic relations.
  arXiv  Detail & Related papers  (2023-12-23T02:54:41Z)
• REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning [64.08293076551601]
  We propose a novel method of using a learned measure for identifying positive pairs. Our Retrieval-Based Reconstruction measure measures the similarity between two sequences. We show that the REBAR error is a predictor of mutual class membership.
  arXiv  Detail & Related papers  (2023-11-01T13:44:45Z)
• Disentangled Representation Learning for Text-Video Retrieval [51.861423831566626]
  Cross-modality interaction is a critical component in Text-Video Retrieval (TVR) We study the interaction paradigm in depth, where we find that its computation can be split into two terms. We propose a disentangled framework to capture a sequential and hierarchical representation.
  arXiv  Detail & Related papers  (2022-03-14T13:55:33Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• Spatio-Temporal Representation Factorization for Video-based Person Re-Identification [55.01276167336187]
  We propose Spatio-Temporal Representation Factorization module (STRF) for re-ID. STRF is a flexible new computational unit that can be used in conjunction with most existing 3D convolutional neural network architectures for re-ID. We empirically show that STRF improves performance of various existing baseline architectures while demonstrating new state-of-the-art results.
  arXiv  Detail & Related papers  (2021-07-25T19:29:37Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.