Replacing Mobile Camera ISP with a Single Deep Learning Model
- URL: http://arxiv.org/abs/2002.05509v1
- Date: Thu, 13 Feb 2020 14:22:39 GMT
- Title: Replacing Mobile Camera ISP with a Single Deep Learning Model
- Authors: Andrey Ignatov, Luc Van Gool, Radu Timofte
- Abstract summary: PyNET is a novel pyramidal CNN architecture designed for fine-grained image restoration.
The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera.
- Score: 171.49776472948957
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: As the popularity of mobile photography is growing constantly, lots of
efforts are being invested now into building complex hand-crafted camera ISP
solutions. In this work, we demonstrate that even the most sophisticated ISP
pipelines can be replaced with a single end-to-end deep learning model trained
without any prior knowledge about the sensor and optics used in a particular
device. For this, we present PyNET, a novel pyramidal CNN architecture designed
for fine-grained image restoration that implicitly learns to perform all ISP
steps such as image demosaicing, denoising, white balancing, color and contrast
correction, demoireing, etc. The model is trained to convert RAW Bayer data
obtained directly from mobile camera sensor into photos captured with a
professional high-end DSLR camera, making the solution independent of any
particular mobile ISP implementation. To validate the proposed approach on the
real data, we collected a large-scale dataset consisting of 10 thousand
full-resolution RAW-RGB image pairs captured in the wild with the Huawei P20
cameraphone (12.3 MP Sony Exmor IMX380 sensor) and Canon 5D Mark IV DSLR. The
experiments demonstrate that the proposed solution can easily get to the level
of the embedded P20's ISP pipeline that, unlike our approach, is combining the
data from two (RGB + B/W) camera sensors. The dataset, pre-trained models and
codes used in this paper are available on the project website.
Related papers
- Simple Image Signal Processing using Global Context Guidance [56.41827271721955]
Deep learning-based ISPs aim to transform RAW images into DSLR-like RGB images using deep neural networks.
We propose a novel module that can be integrated into any neural ISP to capture the global context information from the full RAW images.
Our model achieves state-of-the-art results on different benchmarks using diverse and real smartphone images.
arXiv Detail & Related papers (2024-04-17T17:11:47Z) - Rawformer: Unpaired Raw-to-Raw Translation for Learnable Camera ISPs [53.68932498994655]
This paper introduces a novel method for unpaired learning of raw-to-raw translation across diverse cameras.
It accurately maps raw images captured by a certain camera to the target camera, facilitating the generalization of learnable ISPs to new unseen cameras.
Our method demonstrates superior performance on real camera datasets, achieving higher accuracy compared to previous state-of-the-art techniques.
arXiv Detail & Related papers (2024-04-16T16:17:48Z) - PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural
Networks [115.97113917000145]
We propose a novel PyNET-V2 Mobile CNN architecture designed specifically for edge devices.
The proposed architecture is able to process RAW 12MP photos directly on mobile phones under 1.5 second.
We show that the proposed architecture is also compatible with the latest mobile AI accelerators.
arXiv Detail & Related papers (2022-11-08T17:18:01Z) - Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI &
AIM 2022 Challenge: Report [59.831324427712815]
This challenge aims to develop an efficient end-to-end AI-based image signal processing pipeline.
The models were evaluated on the Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the majority of common deep learning ops.
The proposed solutions are compatible with all recent mobile GPUs, being able to process Full HD photos in less than 20-50 milliseconds while achieving high fidelity results.
arXiv Detail & Related papers (2022-11-07T22:13:10Z) - Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021
Challenge: Report [49.643297263102845]
This challenge aims to develop an end-to-end deep learning-based image signal processing pipeline.
The proposed solutions are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results.
arXiv Detail & Related papers (2021-05-17T13:20:35Z) - AWNet: Attentive Wavelet Network for Image ISP [14.58067200317891]
We introduce a novel network that utilizes the attention mechanism and wavelet transform, dubbed AWNet, to tackle this learnable image ISP problem.
Our proposed method enables us to restore favorable image details from RAW information and achieve a larger receptive field.
Experimental results indicate the advances of our design in both qualitative and quantitative measurements.
arXiv Detail & Related papers (2020-08-20T23:28:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.