AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

• URL: http://arxiv.org/abs/2410.22939v1
• Date: Wed, 30 Oct 2024 11:49:06 GMT
• Title: AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection
• Authors: Yujin Wang, Tianyi Xu, Fan Zhang, Tianfan Xue, Jinwei Gu,
• Abstract summary: Image Signal Processors (ISPs) convert raw sensor signals into digital images. ISPs are designed to maximize the image quality, which are sub-optimal in the performance of high-level computer vision tasks. We propose AdaptiveISP, a task-driven and scene-adaptive ISP.
• Score: 15.63212587981912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image Signal Processors (ISPs) convert raw sensor signals into digital images, which significantly influence the image quality and the performance of downstream computer vision tasks. Designing ISP pipeline and tuning ISP parameters are two key steps for building an imaging and vision system. To find optimal ISP configurations, recent works use deep neural networks as a proxy to search for ISP parameters or ISP pipelines. However, these methods are primarily designed to maximize the image quality, which are sub-optimal in the performance of high-level computer vision tasks such as detection, recognition, and tracking. Moreover, after training, the learned ISP pipelines are mostly fixed at the inference time, whose performance degrades in dynamic scenes. To jointly optimize ISP structures and parameters, we propose AdaptiveISP, a task-driven and scene-adaptive ISP. One key observation is that for the majority of input images, only a few processing modules are needed to improve the performance of downstream recognition tasks, and only a few inputs require more processing. Based on this, AdaptiveISP utilizes deep reinforcement learning to automatically generate an optimal ISP pipeline and the associated ISP parameters to maximize the detection performance. Experimental results show that AdaptiveISP not only surpasses the prior state-of-the-art methods for object detection but also dynamically manages the trade-off between detection performance and computational cost, especially suitable for scenes with large dynamic range variations. Project website: https://openimaginglab.github.io/AdaptiveISP/.

Related papers

• Beyond RGB: Adaptive Parallel Processing for RAW Object Detection [5.36869872375791]
  Raw Adaptation Module (RAM) is a module designed to replace the traditional Image Signal Processing (ISP) Our approach outperforms RGB-based methods and achieves state-of-the-art results across diverse RAW image datasets.
  arXiv  Detail & Related papers  (2025-03-17T13:36:49Z)
• Striving for Faster and Better: A One-Layer Architecture with Auto Re-parameterization for Low-Light Image Enhancement [50.93686436282772]
  We aim to delve into the limits of image enhancers both from visual quality and computational efficiency. By rethinking the task demands, we build an explicit connection, i.e., visual quality and computational efficiency are corresponding to model learning and structure design. Ultimately, this achieves efficient low-light image enhancement using only a single convolutional layer, while maintaining excellent visual quality.
  arXiv  Detail & Related papers  (2025-02-27T08:20:03Z)
• PQDynamicISP: Dynamically Controlled Image Signal Processor for Any Image Sensors Pursuing Perceptual Quality [0.5530212768657544]
  Instead of tuning the parameters of the ISP, we propose to control them dynamically for each environment and even locally. Our method can process different image sensors with a single ISP through dynamic control, whereas conventional methods require training for each sensor.
  arXiv  Detail & Related papers  (2024-03-15T08:08:24Z)
• DGNet: Dynamic Gradient-Guided Network for Water-Related Optics Image Enhancement [77.0360085530701]
  Underwater image enhancement (UIE) is a challenging task due to the complex degradation caused by underwater environments. Previous methods often idealize the degradation process, and neglect the impact of medium noise and object motion on the distribution of image features. Our approach utilizes predicted images to dynamically update pseudo-labels, adding a dynamic gradient to optimize the network's gradient space.
  arXiv  Detail & Related papers  (2023-12-12T06:07:21Z)
• Learning Degradation-Independent Representations for Camera ISP Pipelines [14.195578257521934]
  We propose a novel approach to learn degradation-independent representations (DiR) through the refinement of a self-supervised learned baseline representation. The proposed DiR learning technique has remarkable domain generalization capability and it outperforms state-of-the-art methods across various downstream tasks.
  arXiv  Detail & Related papers  (2023-07-03T05:38:28Z)
• Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning [91.5113227694443]
  We propose a novel visual. sensuous-aware fine-Tuning (SPT) scheme. SPT allocates trainable parameters to task-specific important positions. Experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods.
  arXiv  Detail & Related papers  (2023-03-15T12:34:24Z)
• DynamicISP: Dynamically Controlled Image Signal Processor for Image Recognition [0.5530212768657544]
  "DynamicISP," consists of multiple classical ISP functions and dynamically controls the parameters of each frame according to recognition result of previous frame. We show our method successfully controls the parameters of multiple ISP functions and achieves state-of-the-art accuracy with low computational cost in single and multi-category object detection tasks.
  arXiv  Detail & Related papers  (2022-11-02T14:22:50Z)
• LW-ISP: A Lightweight Model with ISP and Deep Learning [17.972611191715888]
  We show the possibility of learning-based method to achieve real-time high-performance processing in the ISP pipeline. We propose LW-ISP, a novel architecture designed to implicitly learn the image mapping from RAW data to RGB image. Experiments demonstrate that LW-ISP has achieved a 0.38 dB improvement in PSNR compared to the previous best method.
  arXiv  Detail & Related papers  (2022-10-08T04:00:03Z)
• Controllable Image Enhancement [66.18525728881711]
  We present a semiautomatic image enhancement algorithm that can generate high-quality images with multiple styles by controlling a few parameters. An encoder-decoder framework encodes the retouching skills into latent codes and decodes them into the parameters of image signal processing functions.
  arXiv  Detail & Related papers  (2022-06-16T23:54:53Z)
• An Empirical Study of Remote Sensing Pretraining [117.90699699469639]
  We conduct an empirical study of remote sensing pretraining (RSP) on aerial images. RSP can help deliver distinctive performances in scene recognition tasks. RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, but it may still suffer from task discrepancies.
  arXiv  Detail & Related papers  (2022-04-06T13:38:11Z)
• ReconfigISP: Reconfigurable Camera Image Processing Pipeline [75.46902933531247]
  Image Signal Processor (ISP) is crucial component in digital cameras that transforms sensor signals into images for us to perceive and understand. Existing ISP designs always adopt a fixed architecture, e.g., several sequential modules connected in a rigid order. In this study, we propose a novel Reconfigurable ISP (ReconfigISP) whose architecture and parameters can be automatically tailored to specific data and tasks.
  arXiv  Detail & Related papers  (2021-09-10T09:56:43Z)
• Learning Frequency-aware Dynamic Network for Efficient Super-Resolution [56.98668484450857]
  This paper explores a novel frequency-aware dynamic network for dividing the input into multiple parts according to its coefficients in the discrete cosine transform (DCT) domain. In practice, the high-frequency part will be processed using expensive operations and the lower-frequency part is assigned with cheap operations to relieve the computation burden. Experiments conducted on benchmark SISR models and datasets show that the frequency-aware dynamic network can be employed for various SISR neural architectures.
  arXiv  Detail & Related papers  (2021-03-15T12:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.