A Spatio-temporal Aligned SUNet Model for Low-light Video Enhancement

• URL: http://arxiv.org/abs/2403.02408v3
• Date: Fri, 12 Jul 2024 13:05:46 GMT
• Title: A Spatio-temporal Aligned SUNet Model for Low-light Video Enhancement
• Authors: Ruirui Lin, Nantheera Anantrasirichai, Alexandra Malyugina, David Bull,
• Abstract summary: STA-SUNet model is trained on a novel, fully registered dataset (BVI) It is analysed comparatively against various other models over three test datasets. It is particularly effective in extreme low-light conditions, yielding fairly good visualisation results.
• Score: 44.1973928137492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distortions caused by low-light conditions are not only visually unpleasant but also degrade the performance of computer vision tasks. The restoration and enhancement have proven to be highly beneficial. However, there are only a limited number of enhancement methods explicitly designed for videos acquired in low-light conditions. We propose a Spatio-Temporal Aligned SUNet (STA-SUNet) model using a Swin Transformer as a backbone to capture low light video features and exploit their spatio-temporal correlations. The STA-SUNet model is trained on a novel, fully registered dataset (BVI), which comprises dynamic scenes captured under varying light conditions. It is further analysed comparatively against various other models over three test datasets. The model demonstrates superior adaptivity across all datasets, obtaining the highest PSNR and SSIM values. It is particularly effective in extreme low-light conditions, yielding fairly good visualisation results.

Related papers

• Event-guided Low-light Video Semantic Segmentation [6.938849566816958]
  Event cameras can capture motion dynamics, filter out temporal-redundant information, and are robust to lighting conditions. We propose EVSNet, a lightweight framework that leverages event modality to guide the learning of a unified illumination-invariant representation. Specifically, we leverage a Motion Extraction Module to extract short-term and long-term temporal motions from event modality and a Motion Fusion Module to integrate image features and motion features adaptively.
  arXiv  Detail & Related papers  (2024-11-01T14:54:34Z)
• BVI-RLV: A Fully Registered Dataset and Benchmarks for Low-Light Video Enhancement [56.97766265018334]
  This paper introduces a low-light video dataset, consisting of 40 scenes with various motion scenarios under two distinct low-lighting conditions. We provide fully registered ground truth data captured in normal light using a programmable motorized dolly and refine it via an image-based approach for pixel-wise frame alignment across different light levels. Our experimental results demonstrate the significance of fully registered video pairs for low-light video enhancement (LLVE) and the comprehensive evaluation shows that the models trained with our dataset outperform those trained with the existing datasets.
  arXiv  Detail & Related papers  (2024-07-03T22:41:49Z)
• From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization [29.197409507402465]
  We propose a novel per-scene optimization strategy tailored for low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments.
  arXiv  Detail & Related papers  (2024-06-12T11:15:59Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• BVI-Lowlight: Fully Registered Benchmark Dataset for Low-Light Video Enhancement [44.1973928137492]
  This paper introduces a novel low-light video dataset, consisting of 40 scenes in various motion scenarios under two low-lighting conditions. We provide fully registered ground truth data captured in normal light using a programmable motorized dolly. We refine them via image-based post-processing to ensure the pixel-wise alignment of frames in different light levels.
  arXiv  Detail & Related papers  (2024-02-03T00:40:22Z)
• HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces [71.1071688018433]
  Neural radiance fields provide state-of-the-art view synthesis quality but tend to be slow to render. We propose a method, HybridNeRF, that leverages the strengths of both representations by rendering most objects as surfaces. We improve error rates by 15-30% while achieving real-time framerates (at least 36 FPS) for virtual-reality resolutions (2Kx2K)
  arXiv  Detail & Related papers  (2023-12-05T22:04:49Z)
• LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
  We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules. We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
  arXiv  Detail & Related papers  (2023-12-02T04:31:51Z)
• Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions [9.070630868911639]
  We propose a synthetic-aware adverse weather robust algorithm for video stabilization. Our model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.
  arXiv  Detail & Related papers  (2022-08-26T16:21:19Z)
• RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs [79.00855490550367]
  We show that NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but also conditional models that are extensively pre-trained on large multi-view datasets.
  arXiv  Detail & Related papers  (2021-12-01T18:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.