MD-BERT: Action Recognition in Dark Videos via Dynamic Multi-Stream Fusion and Temporal Modeling

• URL: http://arxiv.org/abs/2502.03724v1
• Date: Thu, 06 Feb 2025 02:26:47 GMT
• Title: MD-BERT: Action Recognition in Dark Videos via Dynamic Multi-Stream Fusion and Temporal Modeling
• Authors: Sharana Dharshikgan Suresh Dass, Hrishav Bakul Barua, Ganesh Krishnasamy, Raveendran Paramesran, Raphael C. -W. Phan,
• Abstract summary: This paper proposes a novel multi-stream approach that integrates complementary pre-processing techniques such as gamma correction and histograms alongside raw dark frames.<n>Extensive experiments on ARID V1.0 and ARID1.5 dark video datasets show that MD-BERT outperforms existing methods.
• Score: 4.736059095502584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Action recognition in dark, low-light (under-exposed) or noisy videos is a challenging task due to visibility degradation, which can hinder critical spatiotemporal details. This paper proposes MD-BERT, a novel multi-stream approach that integrates complementary pre-processing techniques such as gamma correction and histogram equalization alongside raw dark frames to address these challenges. We introduce the Dynamic Feature Fusion (DFF) module, extending existing attentional fusion methods to a three-stream setting, thereby capturing fine-grained and global contextual information across different brightness and contrast enhancements. The fused spatiotemporal features are then processed by a BERT-based temporal model, which leverages its bidirectional self-attention to effectively capture long-range dependencies and contextual relationships across frames. Extensive experiments on the ARID V1.0 and ARID V1.5 dark video datasets show that MD-BERT outperforms existing methods, establishing a new state-of-the-art performance. Ablation studies further highlight the individual contributions of each input stream and the effectiveness of the proposed DFF and BERT modules. The official website of this work is available at: https://github.com/HrishavBakulBarua/DarkBERT

Related papers

• TiS-TSL: Image-Label Supervised Surgical Video Stereo Matching via Time-Switchable Teacher-Student Learning [43.682537904776616]
  TiS-TSL is a timeswitchable teacher-student learning framework for video stereo matching under minimal supervision.<n>A unified model operates in three distinct modes: Image-Prediction (IP), Forward Video-Prediction (FVP), and Backward Video-Prediction (BVP)
  arXiv  Detail & Related papers  (2025-11-10T08:01:26Z)
• Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models [31.138079872368532]
  Fine-tuning Video Diffusion Models (VDMs) at the user level to generate videos that reflect specific attributes of training data presents notable challenges.<n>Recent work such as Representation Alignment (REPA) has shown promise in improving the convergence and quality of DiT-based image diffusion models.<n>We introduce Cross-frame Representation Alignment (CREPA), a novel regularization technique that aligns hidden states of a frame with external features from neighboring frames.
  arXiv  Detail & Related papers  (2025-06-10T20:34:47Z)
• REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training [58.33728862521732]
  Diffusion Transformers (DiTs) deliver state-of-the-art image quality, yet their training remains notoriously slow.<n>A recent remedy -- representation alignment (REPA) that matches DiT hidden features to those of a non-generative teacher (e.g. DINO) -- dramatically accelerates the early epochs but plateaus or even degrades performance later.<n>We trace this failure to a capacity mismatch: once the generative student begins modelling the joint data distribution, the teacher's lower-dimensional embeddings and attention patterns become a straitjacket rather than a guide.<n>We introduce HASTE
  arXiv  Detail & Related papers  (2025-05-22T15:34:33Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• VideoFusion: A Spatio-Temporal Collaborative Network for Mutli-modal Video Fusion and Restoration [26.59510171451438]
  Existing multi-sensor fusion research predominantly integrates complementary from multiple images rather than videos. VideoFusion exploits cross-modal complementarity and temporal dynamics to generate context-temporally coherent videos. Extensive experiments reveal that VideoFusion outperforms existing image-oriented fusion paradigms in sequential scenarios.
  arXiv  Detail & Related papers  (2025-03-30T08:27:18Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [63.87313550399871]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability. We propose Self-supervised Transfer (PST) and FrequencyDe-coupled Fusion module (FreDF) PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models. FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition [3.271109623410664]
  We propose COMODO, a cross-modal self-supervised distillation framework that transfers rich semantic knowledge from the video modality to the IMU modality without requiring labeled annotations. Our approach enables the IMU encoder to inherit rich semantic information from video while preserving its efficiency for real-world applications.
  arXiv  Detail & Related papers  (2025-03-10T12:43:51Z)
• Multi-Modality Driven LoRA for Adverse Condition Depth Estimation [61.525312117638116]
  We propose Multi-Modality Driven LoRA (MMD-LoRA) for Adverse Condition Depth Estimation.<n>It consists of two core components: Prompt Driven Domain Alignment (PDDA) and Visual-Text Consistent Contrastive Learning (VTCCL)<n>It achieves state-of-the-art performance on the nuScenes and Oxford RobotCar datasets.
  arXiv  Detail & Related papers  (2024-12-28T14:23:58Z)
• VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval [8.908777234657046]
  Large-language and vision-language models (LLM/LVLMs) have gained prominence across various domains.<n>Here we propose VideoLights, a novel HD/MR framework addressing these limitations through (i) Convolutional Projection and Feature Refinement modules.<n> Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-02T14:45:53Z)
• DAE-Fuse: An Adaptive Discriminative Autoencoder for Multi-Modality Image Fusion [10.713089596405053]
  We propose DAE-Fuse, a novel two-phase discriminative autoencoder framework that generates sharp and natural fused images.<n>We pioneer the extension of image fusion techniques from static images to the video domain.<n>DaE-Fuse achieves state-of-the-art performance on multiple benchmarks, with superior generalizability to tasks like medical image fusion.
  arXiv  Detail & Related papers  (2024-09-16T08:37:09Z)
• DemMamba: Alignment-free Raw Video Demoireing with Frequency-assisted Spatio-Temporal Mamba [18.06907326360215]
  Moire patterns, resulting from the interference of two similar repetitive patterns, are frequently observed during the capture of images or videos on screens. This paper introduces a novel alignment-free raw video demoireing network with frequency-assisted-temporal Mamba. Our proposed DemMamba surpasses state-of-the-art methods by 1.3 dB in PSNR, and also provides a satisfactory visual experience.
  arXiv  Detail & Related papers  (2024-08-20T09:31:03Z)
• DL-KDD: Dual-Light Knowledge Distillation for Action Recognition in the Dark [2.941253902145271]
  We propose a teacher-student video classification framework, named Dual-Light KnowleDge Distillation for Action Recognition in the Dark (DL-KDD) This framework enables the model to learn from both original and enhanced video without introducing additional computational cost during inference. In our experiments, the proposed DL-KDD framework outperforms state-of-the-art methods on the ARID, ARID V1.5, and Dark-48 datasets.
  arXiv  Detail & Related papers  (2024-06-04T16:38:06Z)
• MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training [57.18758272617101]
  MaeFuse is a novel autoencoder model designed for infrared and visible image fusion (IVIF) Our model utilizes a pretrained encoder from Masked Autoencoders (MAE), which facilities the omni features extraction for low-level reconstruction and high-level vision tasks. MaeFuse not only introduces a novel perspective in the realm of fusion techniques but also stands out with impressive performance across various public datasets.
  arXiv  Detail & Related papers  (2024-04-17T02:47:39Z)
• ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schrödinger Bridge [63.00793292863]
  ToddlerDiffusion is a novel approach to decomposing the complex task of RGB image generation into simpler, interpretable stages. Our method, termed ToddlerDiffusion, cascades modality-specific models, each responsible for generating an intermediate representation. ToddlerDiffusion consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-24T15:20:01Z)
• NORM: Knowledge Distillation via N-to-One Representation Matching [18.973254404242507]
  We present a new two-stage knowledge distillation method, which relies on a simple Feature Transform (FT) module consisting of two linear layers. In view of preserving the intact information learnt by the teacher network, our FT module is merely inserted after the last convolutional layer of the student network. By sequentially splitting the expanded student representation into N non-overlapping feature segments having the same number of feature channels as the teacher's, they can be readily forced to approximate the intact teacher representation simultaneously.
  arXiv  Detail & Related papers  (2023-05-23T08:15:45Z)
• Searching a Compact Architecture for Robust Multi-Exposure Image Fusion [55.37210629454589]
  Two major stumbling blocks hinder the development, including pixel misalignment and inefficient inference. This study introduces an architecture search-based paradigm incorporating self-alignment and detail repletion modules for robust multi-exposure image fusion. The proposed method outperforms various competitive schemes, achieving a noteworthy 3.19% improvement in PSNR for general scenarios and an impressive 23.5% enhancement in misaligned scenarios.
  arXiv  Detail & Related papers  (2023-05-20T17:01:52Z)
• TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition [68.53072549422775]
  We propose a student-teacher semi-supervised learning framework, TimeBalance. We distill the knowledge from a temporally-invariant and a temporally-distinctive teacher. Our method achieves state-of-the-art performance on three action recognition benchmarks.
  arXiv  Detail & Related papers  (2023-03-28T19:28:54Z)
• A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition [24.02488085447691]
  We introduce a novel video data augmentation method dubbed ShuffleMix, which acts as a supplement to MixUp, to provide additional temporal regularization for motion recognition. Secondly, a Unified Multimodal De-coupling and multi-stage Re-coupling framework, termed UMDR, is proposed for video representation learning.
  arXiv  Detail & Related papers  (2022-11-16T19:00:23Z)
• Aligning Logits Generatively for Principled Black-Box Knowledge Distillation [49.43567344782207]
  Black-Box Knowledge Distillation (B2KD) is a formulated problem for cloud-to-edge model compression with invisible data and models hosted on the server. We formalize a two-step workflow consisting of deprivatization and distillation. We propose a new method Mapping-Emulation KD (MEKD) that distills a black-box cumbersome model into a lightweight one.
  arXiv  Detail & Related papers  (2022-05-21T02:38:16Z)
• Delta Distillation for Efficient Video Processing [68.81730245303591]
  We propose a novel knowledge distillation schema coined as Delta Distillation. We demonstrate that these temporal variations can be effectively distilled due to the temporal redundancies within video frames. As a by-product, delta distillation improves the temporal consistency of the teacher model.
  arXiv  Detail & Related papers  (2022-03-17T20:13:30Z)
• End-to-end Multi-modal Video Temporal Grounding [105.36814858748285]
  We propose a multi-modal framework to extract complementary information from videos. We adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. We conduct experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-12T17:58:10Z)
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation [62.210091681352914]
  We study multi-sensor fusion for 3D semantic segmentation for many applications, such as autonomous driving and robotics. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) We propose a two-stream network to extract features from the two modalities separately. The extracted features are fused by effective residual-based fusion modules.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• TCLR: Temporal Contrastive Learning for Video Representation [49.6637562402604]
  We develop a new temporal contrastive learning framework consisting of two novel losses to improve upon existing contrastive self-supervised video representation learning methods. With the commonly used 3D-ResNet-18 architecture, we achieve 82.4% (+5.1% increase over the previous best) top-1 accuracy on UCF101 and 52.9% (+5.4% increase) on HMDB51 action classification.
  arXiv  Detail & Related papers  (2021-01-20T05:38:16Z)
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation [30.896957367331137]
  Two neural networks, namely a teacher and a student, are coupled together during training. The teacher network is supposed to be a trustworthy predictor and the student tries to mimic its predictions. In such a setting, distillation only happens for final predictions whereas the student could also benefit from teacher's supervision for internal components.
  arXiv  Detail & Related papers  (2020-12-27T22:30:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.