Exemplar-based Video Colorization with Long-term Spatiotemporal Dependency

• URL: http://arxiv.org/abs/2303.15081v1
• Date: Mon, 27 Mar 2023 10:45:00 GMT
• Title: Exemplar-based Video Colorization with Long-term Spatiotemporal Dependency
• Authors: Siqi Chen, Xueming Li, Xianlin Zhang, Mingdao Wang, Yu Zhang, Jiatong Han, Yue Zhang
• Abstract summary: Exear-based video colorization is an essential technique for applications like old movie restoration. We propose an exemplar-based video colorization framework with long-term temporal dependency dependency. Our model can generate more colorful, realistic and stabilized results, especially for scenes where objects change greatly and irregularly.
• Score: 10.223719035434586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exemplar-based video colorization is an essential technique for applications like old movie restoration. Although recent methods perform well in still scenes or scenes with regular movement, they always lack robustness in moving scenes due to their weak ability in modeling long-term dependency both spatially and temporally, leading to color fading, color discontinuity or other artifacts. To solve this problem, we propose an exemplar-based video colorization framework with long-term spatiotemporal dependency. To enhance the long-term spatial dependency, a parallelized CNN-Transformer block and a double head non-local operation are designed. The proposed CNN-Transformer block can better incorporate long-term spatial dependency with local texture and structural features, and the double head non-local operation further leverages the performance of augmented feature. While for long-term temporal dependency enhancement, we further introduce the novel linkage subnet. The linkage subnet propagate motion information across adjacent frame blocks and help to maintain temporal continuity. Experiments demonstrate that our model outperforms recent state-of-the-art methods both quantitatively and qualitatively. Also, our model can generate more colorful, realistic and stabilized results, especially for scenes where objects change greatly and irregularly.

Related papers

• LatentColorization: Latent Diffusion-Based Speaker Video Colorization [1.2641141743223379]
  We introduce a novel solution for achieving temporal consistency in video colorization. We demonstrate strong improvements on established image quality metrics compared to other existing methods. Our dataset encompasses a combination of conventional datasets and videos from television/movies.
  arXiv  Detail & Related papers  (2024-05-09T12:06:06Z)
• ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization [62.751303924391564]
  How to effectively explore spatial-temporal features is important for video colorization. We develop a memory-based feature propagation module that can establish reliable connections with features from far-apart frames. We develop a local attention module to aggregate the features from adjacent frames in a spatial-temporal neighborhood.
  arXiv  Detail & Related papers  (2024-04-09T12:23:30Z)
• Generating Long Videos of Dynamic Scenes [66.56925105992472]
  We present a video generation model that reproduces object motion, changes in camera viewpoint, and new content that arises over time. A common failure case is for content to never change due to over-reliance on inductive biases to provide temporal consistency.
  arXiv  Detail & Related papers  (2022-06-07T16:29:51Z)
• Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning [90.38674162878496]
  We propose a novel temporally consistent video colorization framework (TCVC) TCVC effectively propagates frame-level deep features in a bidirectional way to enhance the temporal consistency of colorization. Experiments demonstrate that our method can not only obtain visually pleasing colorized video, but also achieve clearly better temporal consistency than state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-10-09T13:00:14Z)
• StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN [70.31913835035206]
  We present a novel approach to the video synthesis problem that helps to greatly improve visual quality. We make use of a pre-trained StyleGAN network, the latent space of which allows control over the appearance of the objects it was trained for. Our temporal architecture is then trained not on sequences of RGB frames, but on sequences of StyleGAN latent codes.
  arXiv  Detail & Related papers  (2021-07-15T09:58:15Z)
• Decoupled Spatial-Temporal Transformer for Video Inpainting [77.8621673355983]
  Video aims to fill the given holes with realistic appearance but is still a challenging task even with prosperous deep learning approaches. Recent works introduce the promising Transformer architecture into deep video inpainting and achieve better performance. We propose a Decoupled Spatial-Temporal Transformer (DSTT) for improving video inpainting with exceptional efficiency.
  arXiv  Detail & Related papers  (2021-04-14T05:47:46Z)
• Coarse-Fine Networks for Temporal Activity Detection in Videos [45.03545172714305]
  We introduce 'Co-Fine Networks', a two-stream architecture which benefits from different abstractions of temporal resolution to learn better video representations for long-term motion. We show that our method can outperform the state-of-the-arts for action detection in public datasets with a significantly reduced compute and memory footprint.
  arXiv  Detail & Related papers  (2021-03-01T20:48:01Z)
• Short-Term and Long-Term Context Aggregation Network for Video Inpainting [126.06302824297948]
  Video inpainting aims to restore missing regions of a video and has many applications such as video editing and object removal. We present a novel context aggregation network to effectively exploit both short-term and long-term frame information for video inpainting. Experiments show that it outperforms state-of-the-art methods with better inpainting results and fast inpainting speed.
  arXiv  Detail & Related papers  (2020-09-12T03:50:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.