Referring Flexible Image Restoration

• URL: http://arxiv.org/abs/2404.10342v1
• Date: Tue, 16 Apr 2024 07:25:17 GMT
• Title: Referring Flexible Image Restoration
• Abstract summary: We term this task Referring Flexible Image Restoration (RFIR) To address this, we first construct a large-scale synthetic dataset called RFIR, comprising 153,423 samples with the degraded image. We propose a novel transformer-based multi-task model named TransRFIR, which simultaneously perceives degradation types in the degraded image and removes specific degradation upon text prompt.
• Score: 9.664435854616633
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In reality, images often exhibit multiple degradations, such as rain and fog at night (triple degradations). However, in many cases, individuals may not want to remove all degradations, for instance, a blurry lens revealing a beautiful snowy landscape (double degradations). In such scenarios, people may only desire to deblur. These situations and requirements shed light on a new challenge in image restoration, where a model must perceive and remove specific degradation types specified by human commands in images with multiple degradations. We term this task Referring Flexible Image Restoration (RFIR). To address this, we first construct a large-scale synthetic dataset called RFIR, comprising 153,423 samples with the degraded image, text prompt for specific degradation removal and restored image. RFIR consists of five basic degradation types: blur, rain, haze, low light and snow while six main sub-categories are included for varying degrees of degradation removal. To tackle the challenge, we propose a novel transformer-based multi-task model named TransRFIR, which simultaneously perceives degradation types in the degraded image and removes specific degradation upon text prompt. TransRFIR is based on two devised attention modules, Multi-Head Agent Self-Attention (MHASA) and Multi-Head Agent Cross Attention (MHACA), where MHASA and MHACA introduce the agent token and reach the linear complexity, achieving lower computation cost than vanilla self-attention and cross-attention and obtaining competitive performances. Our TransRFIR achieves state-of-the-art performances compared with other counterparts and is proven as an effective architecture for image restoration. We release our project at https://github.com/GuanRunwei/FIR-CP.

Related papers

• PixRestore: Unified Image Restoration via Pixel Diffusion Transformer [25.58481833908051]
  Unified image restoration (UIR) aims to recover high-quality (HQ) content from low-quality (LQ) images with different degradations.<n>We present PixRestore, a VAE-free pixel-space Diffusion Transformer (DiT) for UIR, where the diffusion backbone is trained entirely from scratch.<n>PixRestore performs flow matching directly on patchified pixels, preserving fine-grained details while keeping the token sequence tractable.
  arXiv  Detail & Related papers  (2026-08-17T16:49:55Z)
• Unified Removal of Raindrops and Reflections: A New Benchmark and A Novel Pipeline [23.05612697278085]
  raindrops and reflections frequently co-occur to significantly reduce the visibility of captured images.<n>We first formally define the unified removal of raindrops and reflections (UR$3$) task for the first time.<n>We propose a novel diffusion-based framework (i.e., DiffUR$3$) with several target designs to address this challenging task.
  arXiv  Detail & Related papers  (2026-03-17T12:30:14Z)
• Learning to Restore Multi-Degraded Images via Ingredient Decoupling and Task-Aware Path Adaptation [51.10017611491389]
  Real-world images often suffer from multiple coexisting degradations, such as rain, noise, and haze coexisting in a single image.<n>We propose an adaptive multi-degradation image restoration network that reconstructs images by leveraging decoupled representations of degradation ingredients.<n>The resulting tightly integrated architecture, termed IMDNet, is extensively validated through experiments.
  arXiv  Detail & Related papers  (2025-11-07T01:50:36Z)
• UniRes: Universal Image Restoration for Complex Degradations [53.74404005987783]
  Real-world image restoration is hampered by diverse degradations stemming from varying capture conditions, capture devices and post-processing pipelines.<n>A simple yet flexible diffusionbased framework, named UniRes, is proposed to address such degradations in an end-to-end manner.<n>Our proposed method is evaluated on both complex-degradation and single-degradation image restoration datasets.
  arXiv  Detail & Related papers  (2025-06-05T21:25:39Z)
• UniCoRN: Latent Diffusion-based Unified Controllable Image Restoration Network across Multiple Degradations [4.892790389883125]
  We propose UniCoRN, a unified image restoration approach capable of handling multiple degradation types simultaneously. Specifically, we uncover the potential of low-level visual cues extracted from images in guiding a controllable diffusion model. We also introduce MetaRestore, a metalens imaging benchmark containing images with multiple degradations and artifacts.
  arXiv  Detail & Related papers  (2025-03-20T05:42:13Z)
• MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidance [11.023241681116295]
  We propose a unified framework for multi-task, multi-degradation, and language-guided image fusion. Our framework includes two key components: 1) a practical degradation pipeline that simulates real-world image degradations and generates interactive prompts to guide the model; 2) an all-in-one Transformer (DiT) operating in latent space, which fuses a clean image conditioned on both the degraded inputs and the generated prompts.
  arXiv  Detail & Related papers  (2025-03-19T07:20:02Z)
• UniUIR: Considering Underwater Image Restoration as An All-in-One Learner [49.35128836844725]
  We propose a Universal Underwater Image Restoration method, termed as UniUIR. To decouple degradation-specific issues and explore the inter-correlations among various degradations in UIR task, we designed the Mamba Mixture-of-Experts module. This module extracts degradation prior information in both spatial and frequency domains, and adaptively selects the most appropriate task-specific prompts.
  arXiv  Detail & Related papers  (2025-01-22T16:10:42Z)
• Mixed Degradation Image Restoration via Local Dynamic Optimization and Conditional Embedding [67.57487747508179]
  Multiple-in-one image restoration (IR) has made significant progress, aiming to handle all types of single degraded image restoration with a single model. In this paper, we propose a novel multiple-in-one IR model that can effectively restore images with both single and mixed degradations.
  arXiv  Detail & Related papers  (2024-11-25T09:26:34Z)
• Chain-of-Restoration: Multi-Task Image Restoration Models are Zero-Shot Step-by-Step Universal Image Restorers [53.298698981438]
  We propose Universal Image Restoration (UIR), a new task setting that requires models to be trained on a set of degradation bases and then remove any degradation that these bases can potentially compose in a zero-shot manner. Inspired by the Chain-of-Thought which prompts LLMs to address problems step-by-step, we propose the Chain-of-Restoration (CoR) CoR instructs models to step-by-step remove unknown composite degradations.
  arXiv  Detail & Related papers  (2024-10-11T10:21:42Z)
• OneRestore: A Universal Restoration Framework for Composite Degradation [33.556183375565034]
  In real-world scenarios, image impairments often manifest as composite degradations, presenting a complex interplay of elements such as low light, haze, rain, and snow. Our study proposes a versatile imaging model that consolidates four physical corruption paradigms to accurately represent complex, composite degradation scenarios. OneRestore is a novel transformer-based framework designed for adaptive, controllable scene restoration.
  arXiv  Detail & Related papers  (2024-07-05T16:27:00Z)
• Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models [14.25759541950917]
  This work leverages a capable vision-language model and a synthetic degradation pipeline to learn image restoration in the wild (wild IR) Our base diffusion model is the image restoration SDE (IR-SDE)
  arXiv  Detail & Related papers  (2024-04-15T12:34:21Z)
• Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction [62.955327005837475]
  Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. We present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types.
  arXiv  Detail & Related papers  (2023-10-29T13:58:57Z)
• DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior [70.46245698746874]
  We present DiffBIR, a general restoration pipeline that could handle different blind image restoration tasks. DiffBIR decouples blind image restoration problem into two stages: 1) degradation removal: removing image-independent content; 2) information regeneration: generating the lost image content. In the first stage, we use restoration modules to remove degradations and obtain high-fidelity restored results. For the second stage, we propose IRControlNet that leverages the generative ability of latent diffusion models to generate realistic details.
  arXiv  Detail & Related papers  (2023-08-29T07:11:52Z)
• All-in-one Multi-degradation Image Restoration Network via Hierarchical Degradation Representation [47.00239809958627]
  We propose a novel All-in-one Multi-degradation Image Restoration Network (AMIRNet) AMIRNet learns a degradation representation for unknown degraded images by progressively constructing a tree structure through clustering. This tree-structured representation explicitly reflects the consistency and discrepancy of various distortions, providing a specific clue for image restoration.
  arXiv  Detail & Related papers  (2023-08-06T04:51:41Z)
• DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration [66.01846902242355]
  Blind face restoration usually synthesizes degraded low-quality data with a pre-defined degradation model for training. It is expensive and infeasible to include every type of degradation to cover real-world cases in the training data. We propose Robust Degradation Remover (DR2) to first transform the degraded image to a coarse but degradation-invariant prediction, then employ an enhancement module to restore the coarse prediction to a high-quality image.
  arXiv  Detail & Related papers  (2023-03-13T06:05:18Z)
• Invertible Rescaling Network and Its Extensions [118.72015270085535]
  In this work, we propose a novel invertible framework to model the bidirectional degradation and restoration from a new perspective. We develop invertible models to generate valid degraded images and transform the distribution of lost contents. Then restoration is made tractable by applying the inverse transformation on the generated degraded image together with a randomly-drawn latent variable.
  arXiv  Detail & Related papers  (2022-10-09T06:58:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.