Two-Stage Vision Transformer for Image Restoration: Colorization Pretraining + Residual Upsampling

• URL: http://arxiv.org/abs/2512.02512v2
• Date: Wed, 03 Dec 2025 05:05:19 GMT
• Title: Two-Stage Vision Transformer for Image Restoration: Colorization Pretraining + Residual Upsampling
• Authors: Aditya Chaudhary, Prachet Dev Singh, Ankit Jha,
• Abstract summary: We present a new technique to improve the performance of a Vision Transformer (ViT) employing a two-stage training strategy.<n>ViT-SR, trained and evaluated on the DIV2K benchmark dataset, achieves an impressive Single Image Super-Resolution (SISR) of 0.712 and PSNR of 22.90 dB.
• Score: 4.365909537198615
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In computer vision, Single Image Super-Resolution (SISR) is still a difficult problem. We present ViT-SR, a new technique to improve the performance of a Vision Transformer (ViT) employing a two-stage training strategy. In our method, the model learns rich, generalizable visual representations from the data itself through a self-supervised pretraining phase on a colourization task. The pre-trained model is then adjusted for 4x super-resolution. By predicting the addition of a high-frequency residual image to an initial bicubic interpolation, this design simplifies residual learning. ViT-SR, trained and evaluated on the DIV2K benchmark dataset, achieves an impressive SSIM of 0.712 and PSNR of 22.90 dB. These results demonstrate the efficacy of our two-stage approach and highlight the potential of self-supervised pre-training for complex image restoration tasks. Further improvements may be possible with larger ViT architectures or alternative pretext tasks.

Related papers

• X-DECODE: EXtreme Deblurring with Curriculum Optimization and Domain Equalization [2.348041867134616]
  Restoring severely blurred images remains a significant challenge in computer vision.<n>This paper introduces a novel training strategy based on curriculum learning to improve the robustness of deep learning models for extreme image deblurring.
  arXiv  Detail & Related papers  (2025-04-10T18:59:26Z)
• UniViTAR: Unified Vision Transformer with Native Resolution [37.63387029787732]
  We introduce UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario.<n>A progressive training paradigm is introduced, which strategically combines two core mechanisms.<n>In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model.
  arXiv  Detail & Related papers  (2025-04-02T14:59:39Z)
• High-Resolution Image Synthesis via Next-Token Prediction [19.97037318862443]
  We introduce textbfD-JEPA$cdot$T2I, an autoregressive model based on continuous tokens to generate high-quality, photorealistic images at arbitrary resolutions, up to 4K.<n>For the first time, we achieve state-of-the-art high-resolution image synthesis via next-token prediction.
  arXiv  Detail & Related papers  (2024-11-22T09:08:58Z)
• Training Transformer Models by Wavelet Losses Improves Quantitative and Visual Performance in Single Image Super-Resolution [6.367865391518726]
  Transformer-based models have achieved remarkable results in low-level vision tasks including image super-resolution (SR) To activate more input pixels globally, hybrid attention models have been proposed. We employ wavelet losses to train Transformer models to improve quantitative and subjective performance.
  arXiv  Detail & Related papers  (2024-04-17T11:25:19Z)
• ViTAR: Vision Transformer with Any Resolution [80.95324692984903]
  Vision Transformers experience a performance decline when processing resolutions different from those seen during training. We introduce fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions. Our resulting model, ViTAR, demonstrates impressive adaptability, achieving 83.3% top-1 accuracy at a 1120x1120 resolution and 80.4% accuracy at a 4032x4032 resolution.
  arXiv  Detail & Related papers  (2024-03-27T08:53:13Z)
• Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration [79.04007257606862]
  This paper introduces an innovative method termed 'learning from history', which dynamically generates negative samples from the target model itself. Our approach, named Model Contrastive Learning for Image Restoration (MCLIR), rejuvenates latency models as negative models, making it compatible with diverse image restoration tasks.
  arXiv  Detail & Related papers  (2023-09-12T07:50:54Z)
• Real-World Image Super-Resolution by Exclusionary Dual-Learning [98.36096041099906]
  Real-world image super-resolution is a practical image restoration problem that aims to obtain high-quality images from in-the-wild input. Deep learning-based methods have achieved promising restoration quality on real-world image super-resolution datasets. We propose Real-World image Super-Resolution by Exclusionary Dual-Learning (RWSR-EDL) to address the feature diversity in perceptual- and L1-based cooperative learning.
  arXiv  Detail & Related papers  (2022-06-06T13:28:15Z)
• Vector-quantized Image Modeling with Improved VQGAN [93.8443646643864]
  We propose a Vector-quantized Image Modeling approach that involves pretraining a Transformer to predict image tokens autoregressively. We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. When trained on ImageNet at 256x256 resolution, we achieve Inception Score (IS) of 175.1 and Frechet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN.
  arXiv  Detail & Related papers  (2021-10-09T18:36:00Z)
• Efficient Self-supervised Vision Transformers for Representation Learning [86.57557009109411]
  We show that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity. We propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation.
  arXiv  Detail & Related papers  (2021-06-17T19:57:33Z)
• Pre-Trained Image Processing Transformer [95.93031793337613]
  We develop a new pre-trained model, namely, image processing transformer (IPT) We present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. IPT model is trained on these images with multi-heads and multi-tails.
  arXiv  Detail & Related papers  (2020-12-01T09:42:46Z)
• Two-shot Spatially-varying BRDF and Shape Estimation [89.29020624201708]
  We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF. We create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials. Experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
  arXiv  Detail & Related papers  (2020-04-01T12:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.