LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration

• URL: http://arxiv.org/abs/2410.15385v2
• Date: Sat, 16 Nov 2024 09:02:05 GMT
• Title: LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration
• Authors: Yuang Ai, Huaibo Huang, Ran He,
• Abstract summary: We propose LoRA-IR, a flexible framework that dynamically leverages compact low-rank experts to facilitate efficient all-in-one image restoration. LoRA-IR consists of two training stages: degradation-guided pre-training and parameter-efficient fine-tuning. Experiments demonstrate that LoRA-IR achieves SOTA performance across 14 IR tasks and 29 benchmarks, while maintaining computational efficiency.
• Score: 62.3751291442432
• License:
• Abstract: Prompt-based all-in-one image restoration (IR) frameworks have achieved remarkable performance by incorporating degradation-specific information into prompt modules. Nevertheless, handling the complex and diverse degradations encountered in real-world scenarios remains a significant challenge. To tackle this, we propose LoRA-IR, a flexible framework that dynamically leverages compact low-rank experts to facilitate efficient all-in-one image restoration. Specifically, LoRA-IR consists of two training stages: degradation-guided pre-training and parameter-efficient fine-tuning. In the pre-training stage, we enhance the pre-trained CLIP model by introducing a simple mechanism that scales it to higher resolutions, allowing us to extract robust degradation representations that adaptively guide the IR network. In the fine-tuning stage, we refine the pre-trained IR network through low-rank adaptation (LoRA). Built upon a Mixture-of-Experts (MoE) architecture, LoRA-IR dynamically integrates multiple low-rank restoration experts through a degradation-guided router. This dynamic integration mechanism significantly enhances our model's adaptability to diverse and unknown degradations in complex real-world scenarios. Extensive experiments demonstrate that LoRA-IR achieves SOTA performance across 14 IR tasks and 29 benchmarks, while maintaining computational efficiency. Code and pre-trained models will be available at: https://github.com/shallowdream204/LoRA-IR.

Related papers

• SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture [7.543093479330315]
  Training the whole model for downstream tasks is expensive, and could easily result in catastrophic forgetting. We propose a novel mixture of expert (MoE) framework based on Soft LoRA and Identity Mixture (SLIM) SLIM allows dynamic routing between LoRA adapters and skipping connection, enables the suppression of forgetting.
  arXiv  Detail & Related papers  (2024-10-10T09:16:05Z)
• UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation [50.27688690379488]
  Existing unified methods treat multi-degradation image restoration as a multi-task learning problem. We propose a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multi-degradation restoration and transfers it to specific degradation image restoration tasks.
  arXiv  Detail & Related papers  (2024-09-30T11:16:56Z)
• Any Image Restoration with Efficient Automatic Degradation Adaptation [132.81912195537433]
  We propose a unified manner to achieve joint embedding by leveraging the inherent similarities across various degradations for efficient and comprehensive restoration. Our network sets new SOTA records while reducing model complexity by approximately -82% in trainable parameters and -85% in FLOPs.
  arXiv  Detail & Related papers  (2024-07-18T10:26:53Z)
• AdaIR: Exploiting Underlying Similarities of Image Restoration Tasks with Adapters [57.62742271140852]
  AdaIR is a novel framework that enables low storage cost and efficient training without sacrificing performance. AdaIR requires solely the training of lightweight, task-specific modules, ensuring a more efficient storage and training regimen.
  arXiv  Detail & Related papers  (2024-04-17T15:31:06Z)
• Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning [45.13580581290495]
  This work introduces a novel "Low-Res Leads the Way" (LWay) training framework to enhance the adaptability of SR models to real-world images. Our approach utilizes a low-resolution (LR) reconstruction network to extract degradation embeddings from LR images, merging them with super-resolved outputs for LR reconstruction. Our training regime is universally compatible, requiring no network architecture modifications, making it a practical solution for real-world SR applications.
  arXiv  Detail & Related papers  (2024-03-05T02:29:18Z)
• ICF-SRSR: Invertible scale-Conditional Function for Self-Supervised Real-world Single Image Super-Resolution [60.90817228730133]
  Single image super-resolution (SISR) is a challenging problem that aims to up-sample a given low-resolution (LR) image to a high-resolution (HR) counterpart. Recent approaches are trained on simulated LR images degraded by simplified down-sampling operators. We propose a novel Invertible scale-Conditional Function (ICF) which can scale an input image and then restore the original input with different scale conditions.
  arXiv  Detail & Related papers  (2023-07-24T12:42:45Z)
• DRM-IR: Task-Adaptive Deep Unfolding Network for All-In-One Image Restoration [5.573836220587265]
  This work proposes an efficient Dynamic Reference Modeling paradigm (DRM-IR) DRM-IR consists of task-adaptive degradation modeling and model-based image restoring. Experiments on multiple benchmark datasets show that our DRM-IR achieves state-of-the-art in All-In-One IR.
  arXiv  Detail & Related papers  (2023-07-15T02:42:19Z)
• RBSR: Efficient and Flexible Recurrent Network for Burst Super-Resolution [57.98314517861539]
  Burst super-resolution (BurstSR) aims at reconstructing a high-resolution (HR) image from a sequence of low-resolution (LR) and noisy images. In this paper, we suggest fusing cues frame-by-frame with an efficient and flexible recurrent network.
  arXiv  Detail & Related papers  (2023-06-30T12:14:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.