Rawformer: Unpaired Raw-to-Raw Translation for Learnable Camera ISPs

• URL: http://arxiv.org/abs/2404.10700v2
• Date: Mon, 15 Jul 2024 14:09:28 GMT
• Title: Rawformer: Unpaired Raw-to-Raw Translation for Learnable Camera ISPs
• Authors: Georgy Perevozchikov, Nancy Mehta, Mahmoud Afifi, Radu Timofte,
• Abstract summary: This paper introduces a novel method for unpaired learning of raw-to-raw translation across diverse cameras. It accurately maps raw images captured by a certain camera to the target camera, facilitating the generalization of learnable ISPs to new unseen cameras. Our method demonstrates superior performance on real camera datasets, achieving higher accuracy compared to previous state-of-the-art techniques.
• Score: 53.68932498994655
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern smartphone camera quality heavily relies on the image signal processor (ISP) to enhance captured raw images, utilizing carefully designed modules to produce final output images encoded in a standard color space (e.g., sRGB). Neural-based end-to-end learnable ISPs offer promising advancements, potentially replacing traditional ISPs with their ability to adapt without requiring extensive tuning for each new camera model, as is often the case for nearly every module in traditional ISPs. However, the key challenge with the recent learning-based ISPs is the urge to collect large paired datasets for each distinct camera model due to the influence of intrinsic camera characteristics on the formation of input raw images. This paper tackles this challenge by introducing a novel method for unpaired learning of raw-to-raw translation across diverse cameras. Specifically, we propose Rawformer, an unsupervised Transformer-based encoder-decoder method for raw-to-raw translation. It accurately maps raw images captured by a certain camera to the target camera, facilitating the generalization of learnable ISPs to new unseen cameras. Our method demonstrates superior performance on real camera datasets, achieving higher accuracy compared to previous state-of-the-art techniques, and preserving a more robust correlation between the original and translated raw images. The codes and the pretrained models are available at https://github.com/gosha20777/rawformer.

Related papers

• RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images [51.68432586065828]
  We introduce RAW-Adapter, a novel approach aimed at adapting sRGB pre-trained models to camera RAW data. Raw-Adapter comprises input-level adapters that employ learnable ISP stages to adjust RAW inputs, as well as model-level adapters to build connections between ISP stages and subsequent high-level networks.
  arXiv  Detail & Related papers  (2024-08-27T06:14:54Z)
• Simple Image Signal Processing using Global Context Guidance [56.41827271721955]
  Deep learning-based ISPs aim to transform RAW images into DSLR-like RGB images using deep neural networks. We propose a novel module that can be integrated into any neural ISP to capture the global context information from the full RAW images. Our model achieves state-of-the-art results on different benchmarks using diverse and real smartphone images.
  arXiv  Detail & Related papers  (2024-04-17T17:11:47Z)
• BSRAW: Improving Blind RAW Image Super-Resolution [63.408484584265985]
  We tackle blind image super-resolution in the RAW domain. We design a realistic degradation pipeline tailored specifically for training models with raw sensor data. Our BSRAW models trained with our pipeline can upscale real-scene RAW images and improve their quality.
  arXiv  Detail & Related papers  (2023-12-24T14:17:28Z)
• ParamISP: Learned Forward and Inverse ISPs using Camera Parameters [27.244062839494]
  ParamISP is a learning-based method for forward and inverse conversion between sRGB and RAW images. It can be effectively used for a variety of applications such as deblurring dataset synthesis, raw deblurring, HDR reconstruction, and camera-to-camera transfer.
  arXiv  Detail & Related papers  (2023-12-20T09:16:47Z)
• Transform your Smartphone into a DSLR Camera: Learning the ISP in the Wild [159.71025525493354]
  We propose a trainable Image Signal Processing framework that produces DSLR quality images given RAW images captured by a smartphone. To address the color misalignments between training image pairs, we employ a color-conditional ISP network and optimize a novel parametric color mapping between each input RAW and reference DSLR image.
  arXiv  Detail & Related papers  (2022-03-20T20:13:59Z)
• Model-Based Image Signal Processors via Learnable Dictionaries [6.766416093990318]
  Digital cameras transform sensor RAW readings into RGB images by means of their Image Signal Processor (ISP) Recent approaches have attempted to bridge this gap by estimating the RGB to RAW mapping. We present a novel hybrid model-based and data-driven ISP that is both learnable and interpretable.
  arXiv  Detail & Related papers  (2022-01-10T08:36:10Z)
• Del-Net: A Single-Stage Network for Mobile Camera ISP [14.168130234198467]
  Traditional image signal processing (ISP) pipeline in a smartphone camera consists of several image processing steps performed sequentially to reconstruct a high quality sRGB image from the raw sensor data. Deep learning methods using convolutional neural networks (CNN) have become popular in solving many image-related tasks such as image denoising, contrast enhancement, super resolution, deblurring, etc. In this paper we propose DelNet - a single end-to-end deep learning model - to learn the entire ISP pipeline within reasonable complexity for smartphone deployment.
  arXiv  Detail & Related papers  (2021-08-03T16:51:11Z)
• Replacing Mobile Camera ISP with a Single Deep Learning Model [171.49776472948957]
  PyNET is a novel pyramidal CNN architecture designed for fine-grained image restoration. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera.
  arXiv  Detail & Related papers  (2020-02-13T14:22:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.