MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning
- URL: http://arxiv.org/abs/2211.06770v1
- Date: Tue, 8 Nov 2022 17:40:50 GMT
- Title: MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning
- Authors: Andrey Ignatov and Anastasia Sycheva and Radu Timofte and Yu Tseng and
Yu-Syuan Xu and Po-Hsiang Yu and Cheng-Ming Chiang and Hsien-Kai Kuo and
Min-Hung Chen and Chia-Ming Cheng and Luc Van Gool
- Abstract summary: We present a novel MicroISP model designed specifically for edge devices.
The proposed solution is capable of processing up to 32MP photos on recent smartphones using the standard mobile ML libraries.
The architecture of the model is flexible, allowing to adjust its complexity to devices of different computational power.
- Score: 114.66037224769005
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While neural networks-based photo processing solutions can provide a better
image quality compared to the traditional ISP systems, their application to
mobile devices is still very limited due to their very high computational
complexity. In this paper, we present a novel MicroISP model designed
specifically for edge devices, taking into account their computational and
memory limitations. The proposed solution is capable of processing up to 32MP
photos on recent smartphones using the standard mobile ML libraries and
requiring less than 1 second to perform the inference, while for FullHD images
it achieves real-time performance. The architecture of the model is flexible,
allowing to adjust its complexity to devices of different computational power.
To evaluate the performance of the model, we collected a novel Fujifilm
UltraISP dataset consisting of thousands of paired photos captured with a
normal mobile camera sensor and a professional 102MP medium-format FujiFilm
GFX100 camera. The experiments demonstrated that, despite its compact size, the
MicroISP model is able to provide comparable or better visual results than the
traditional mobile ISP systems, while outperforming the previously proposed
efficient deep learning based solutions. Finally, this model is also compatible
with the latest mobile AI accelerators, achieving good runtime and low power
consumption on smartphone NPUs and APUs. The code, dataset and pre-trained
models are available on the project website:
https://people.ee.ethz.ch/~ihnatova/microisp.html
Related papers
- MobileMEF: Fast and Efficient Method for Multi-Exposure Fusion [0.6261722394141346]
We propose a new method for multi-exposure fusion based on an encoder-decoder deep learning architecture.
Our model is capable of processing 4K resolution images in less than 2 seconds on mid-range smartphones.
arXiv Detail & Related papers (2024-08-15T05:03:14Z) - PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural
Networks [115.97113917000145]
We propose a novel PyNET-V2 Mobile CNN architecture designed specifically for edge devices.
The proposed architecture is able to process RAW 12MP photos directly on mobile phones under 1.5 second.
We show that the proposed architecture is also compatible with the latest mobile AI accelerators.
arXiv Detail & Related papers (2022-11-08T17:18:01Z) - Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI &
AIM 2022 Challenge: Report [59.831324427712815]
This challenge aims to develop an efficient end-to-end AI-based image signal processing pipeline.
The models were evaluated on the Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the majority of common deep learning ops.
The proposed solutions are compatible with all recent mobile GPUs, being able to process Full HD photos in less than 20-50 milliseconds while achieving high fidelity results.
arXiv Detail & Related papers (2022-11-07T22:13:10Z) - Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI
2021 Challenge: Report [67.86837649834636]
We introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image super-resolution solution.
The proposed solutions are fully compatible with all major mobile AI accelerators and are capable of reconstructing Full HD images under 40-60 ms.
arXiv Detail & Related papers (2021-05-17T13:34:15Z) - Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021
Challenge: Report [49.643297263102845]
This challenge aims to develop an end-to-end deep learning-based image signal processing pipeline.
The proposed solutions are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results.
arXiv Detail & Related papers (2021-05-17T13:20:35Z) - Replacing Mobile Camera ISP with a Single Deep Learning Model [171.49776472948957]
PyNET is a novel pyramidal CNN architecture designed for fine-grained image restoration.
The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera.
arXiv Detail & Related papers (2020-02-13T14:22:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.