Learned Lightweight Smartphone ISP with Unpaired Data

• URL: http://arxiv.org/abs/2505.10420v1
• Date: Thu, 15 May 2025 15:37:51 GMT
• Title: Learned Lightweight Smartphone ISP with Unpaired Data
• Authors: Andrei Arhire, Radu Timofte,
• Abstract summary: We propose a novel training method for a learnable Image Signal Processor (ISP)<n>Our unpaired approach employs a multi-term loss function guided by adversarial training.<n>Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity.
• Score: 55.2480439325792
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The Image Signal Processor (ISP) is a fundamental component in modern smartphone cameras responsible for conversion of RAW sensor image data to RGB images with a strong focus on perceptual quality. Recent work highlights the potential of deep learning approaches and their ability to capture details with a quality increasingly close to that of professional cameras. A difficult and costly step when developing a learned ISP is the acquisition of pixel-wise aligned paired data that maps the raw captured by a smartphone camera sensor to high-quality reference images. In this work, we address this challenge by proposing a novel training method for a learnable ISP that eliminates the need for direct correspondences between raw images and ground-truth data with matching content. Our unpaired approach employs a multi-term loss function guided by adversarial training with multiple discriminators processing feature maps from pre-trained networks to maintain content structure while learning color and texture characteristics from the target RGB dataset. Using lightweight neural network architectures suitable for mobile devices as backbones, we evaluated our method on the Zurich RAW to RGB and Fujifilm UltraISP datasets. Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity across multiple evaluation metrics. The code and pre-trained models are available at https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data .

Related papers

• RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images [51.68432586065828]
  We introduce RAW-Adapter, a novel approach aimed at adapting sRGB pre-trained models to camera RAW data. Raw-Adapter comprises input-level adapters that employ learnable ISP stages to adjust RAW inputs, as well as model-level adapters to build connections between ISP stages and subsequent high-level networks.
  arXiv  Detail & Related papers  (2024-08-27T06:14:54Z)
• Simple Image Signal Processing using Global Context Guidance [56.41827271721955]
  Deep learning-based ISPs aim to transform RAW images into DSLR-like RGB images using deep neural networks. We propose a novel module that can be integrated into any neural ISP to capture the global context information from the full RAW images. Our model achieves state-of-the-art results on different benchmarks using diverse and real smartphone images.
  arXiv  Detail & Related papers  (2024-04-17T17:11:47Z)
• Rawformer: Unpaired Raw-to-Raw Translation for Learnable Camera ISPs [53.68932498994655]
  This paper introduces a novel method for unpaired learning of raw-to-raw translation across diverse cameras. It accurately maps raw images captured by a certain camera to the target camera, facilitating the generalization of learnable ISPs to new unseen cameras. Our method demonstrates superior performance on real camera datasets, achieving higher accuracy compared to previous state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-04-16T16:17:48Z)
• Enabling ISP-less Low-Power Computer Vision [4.102254385058941]
  We release the raw version of a large-scale benchmark for generic high-level vision tasks. For ISP-less CV systems, training on raw images result in a 7.1% increase in test accuracy. We propose an energy-efficient form of analog in-pixel demosaicing that may be coupled with in-pixel CNN computations.
  arXiv  Detail & Related papers  (2022-10-11T13:47:30Z)
• Transform your Smartphone into a DSLR Camera: Learning the ISP in the Wild [159.71025525493354]
  We propose a trainable Image Signal Processing framework that produces DSLR quality images given RAW images captured by a smartphone. To address the color misalignments between training image pairs, we employ a color-conditional ISP network and optimize a novel parametric color mapping between each input RAW and reference DSLR image.
  arXiv  Detail & Related papers  (2022-03-20T20:13:59Z)
• Model-Based Image Signal Processors via Learnable Dictionaries [6.766416093990318]
  Digital cameras transform sensor RAW readings into RGB images by means of their Image Signal Processor (ISP) Recent approaches have attempted to bridge this gap by estimating the RGB to RAW mapping. We present a novel hybrid model-based and data-driven ISP that is both learnable and interpretable.
  arXiv  Detail & Related papers  (2022-01-10T08:36:10Z)
• Del-Net: A Single-Stage Network for Mobile Camera ISP [14.168130234198467]
  Traditional image signal processing (ISP) pipeline in a smartphone camera consists of several image processing steps performed sequentially to reconstruct a high quality sRGB image from the raw sensor data. Deep learning methods using convolutional neural networks (CNN) have become popular in solving many image-related tasks such as image denoising, contrast enhancement, super resolution, deblurring, etc. In this paper we propose DelNet - a single end-to-end deep learning model - to learn the entire ISP pipeline within reasonable complexity for smartphone deployment.
  arXiv  Detail & Related papers  (2021-08-03T16:51:11Z)
• Replacing Mobile Camera ISP with a Single Deep Learning Model [171.49776472948957]
  PyNET is a novel pyramidal CNN architecture designed for fine-grained image restoration. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera.
  arXiv  Detail & Related papers  (2020-02-13T14:22:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.