PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural Networks

• URL: http://arxiv.org/abs/2211.06263v1
• Date: Tue, 8 Nov 2022 17:18:01 GMT
• Title: PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural Networks
• Authors: Andrey Ignatov and Grigory Malivenko and Radu Timofte and Yu Tseng and Yu-Syuan Xu and Po-Hsiang Yu and Cheng-Ming Chiang and Hsien-Kai Kuo and Min-Hung Chen and Chia-Ming Cheng and Luc Van Gool
• Abstract summary: We propose a novel PyNET-V2 Mobile CNN architecture designed specifically for edge devices. The proposed architecture is able to process RAW 12MP photos directly on mobile phones under 1.5 second. We show that the proposed architecture is also compatible with the latest mobile AI accelerators.
• Score: 115.97113917000145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The increased importance of mobile photography created a need for fast and performant RAW image processing pipelines capable of producing good visual results in spite of the mobile camera sensor limitations. While deep learning-based approaches can efficiently solve this problem, their computational requirements usually remain too large for high-resolution on-device image processing. To address this limitation, we propose a novel PyNET-V2 Mobile CNN architecture designed specifically for edge devices, being able to process RAW 12MP photos directly on mobile phones under 1.5 second and producing high perceptual photo quality. To train and to evaluate the performance of the proposed solution, we use the real-world Fujifilm UltraISP dataset consisting on thousands of RAW-RGB image pairs captured with a professional medium-format 102MP Fujifilm camera and a popular Sony mobile camera sensor. The results demonstrate that the PyNET-V2 Mobile model can substantially surpass the quality of tradition ISP pipelines, while outperforming the previously introduced neural network-based solutions designed for fast image processing. Furthermore, we show that the proposed architecture is also compatible with the latest mobile AI accelerators such as NPUs or APUs that can be used to further reduce the latency of the model to as little as 0.5 second. The dataset, code and pre-trained models used in this paper are available on the project website: https://github.com/gmalivenko/PyNET-v2

Related papers

• Rawformer: Unpaired Raw-to-Raw Translation for Learnable Camera ISPs [53.68932498994655]
  This paper introduces a novel method for unpaired learning of raw-to-raw translation across diverse cameras. It accurately maps raw images captured by a certain camera to the target camera, facilitating the generalization of learnable ISPs to new unseen cameras. Our method demonstrates superior performance on real camera datasets, achieving higher accuracy compared to previous state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-04-16T16:17:48Z)
• MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning [114.66037224769005]
  We present a novel MicroISP model designed specifically for edge devices. The proposed solution is capable of processing up to 32MP photos on recent smartphones using the standard mobile ML libraries. The architecture of the model is flexible, allowing to adjust its complexity to devices of different computational power.
  arXiv  Detail & Related papers  (2022-11-08T17:40:50Z)
• Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report [59.831324427712815]
  This challenge aims to develop an efficient end-to-end AI-based image signal processing pipeline. The models were evaluated on the Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the majority of common deep learning ops. The proposed solutions are compatible with all recent mobile GPUs, being able to process Full HD photos in less than 20-50 milliseconds while achieving high fidelity results.
  arXiv  Detail & Related papers  (2022-11-07T22:13:10Z)
• Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report [49.643297263102845]
  This challenge aims to develop an end-to-end deep learning-based image signal processing pipeline. The proposed solutions are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results.
  arXiv  Detail & Related papers  (2021-05-17T13:20:35Z)
• AWNet: Attentive Wavelet Network for Image ISP [14.58067200317891]
  We introduce a novel network that utilizes the attention mechanism and wavelet transform, dubbed AWNet, to tackle this learnable image ISP problem. Our proposed method enables us to restore favorable image details from RAW information and achieve a larger receptive field. Experimental results indicate the advances of our design in both qualitative and quantitative measurements.
  arXiv  Detail & Related papers  (2020-08-20T23:28:41Z)
• Replacing Mobile Camera ISP with a Single Deep Learning Model [171.49776472948957]
  PyNET is a novel pyramidal CNN architecture designed for fine-grained image restoration. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera.
  arXiv  Detail & Related papers  (2020-02-13T14:22:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.