Multi-task Image Restoration Guided By Robust DINO Features
- URL: http://arxiv.org/abs/2312.01677v2
- Date: Tue, 5 Dec 2023 17:46:12 GMT
- Title: Multi-task Image Restoration Guided By Robust DINO Features
- Authors: Xin Lin, Chao Ren, Kelvin C.K. Chan, Lu Qi, Jinshan Pan, Ming-Hsuan
Yang
- Abstract summary: We introduce mboxtextbfDINO-IR, a novel multi-task image restoration approach leveraging robust features extracted from DINOv2.
Our empirical analysis shows that while shallow features of DINOv2 capture rich low-level image characteristics, the deep features ensure a robust semantic representation insensitive to degradations.
- Score: 98.7455921708419
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multi-task image restoration has gained significant interest due to its
inherent versatility and efficiency compared to its single-task counterpart.
Despite its potential, performance degradation is observed with an increase in
the number of tasks, primarily attributed to the distinct nature of each
restoration task. Addressing this challenge, we introduce
\mbox{\textbf{DINO-IR}}, a novel multi-task image restoration approach
leveraging robust features extracted from DINOv2. Our empirical analysis shows
that while shallow features of DINOv2 capture rich low-level image
characteristics, the deep features ensure a robust semantic representation
insensitive to degradations while preserving high-frequency contour details.
Building on these features, we devise specialized components, including
multi-layer semantic fusion module, DINO-Restore adaption and fusion module,
and DINO perception contrastive loss, to integrate DINOv2 features into the
restoration paradigm. Equipped with the aforementioned components, our DINO-IR
performs favorably against existing multi-task image restoration approaches in
various tasks by a large margin, indicating the superiority and necessity of
reinforcing the robust features for multi-task image restoration.
Related papers
- RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models [45.88103575837924]
We introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models.
RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration.
Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts.
arXiv Detail & Related papers (2024-07-25T13:29:37Z) - Empowering Image Recovery_ A Multi-Attention Approach [96.25892659985342]
Diverse Restormer (DART) is an image restoration method that integrates information from various sources to address restoration challenges.
DART employs customized attention mechanisms to enhance overall performance.
evaluation across five restoration tasks consistently positions DART at the forefront.
arXiv Detail & Related papers (2024-04-06T12:50:08Z) - Unified-Width Adaptive Dynamic Network for All-In-One Image Restoration [50.81374327480445]
We introduce a novel concept positing that intricate image degradation can be represented in terms of elementary degradation.
We propose the Unified-Width Adaptive Dynamic Network (U-WADN), consisting of two pivotal components: a Width Adaptive Backbone (WAB) and a Width Selector (WS)
The proposed U-WADN achieves better performance while simultaneously reducing up to 32.3% of FLOPs and providing approximately 15.7% real-time acceleration.
arXiv Detail & Related papers (2024-01-24T04:25:12Z) - SPIRE: Semantic Prompt-Driven Image Restoration [66.26165625929747]
We develop SPIRE, a Semantic and restoration Prompt-driven Image Restoration framework.
Our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength.
Our experiments demonstrate the superior restoration performance of SPIRE compared to the state of the arts.
arXiv Detail & Related papers (2023-12-18T17:02:30Z) - Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration [58.11518043688793]
MPerceiver is a novel approach to enhance adaptiveness, generalizability and fidelity for all-in-one image restoration.
MPerceiver is trained on 9 tasks for all-in-one IR and outperforms state-of-the-art task-specific methods across most tasks.
arXiv Detail & Related papers (2023-12-05T17:47:11Z) - DRM-IR: Task-Adaptive Deep Unfolding Network for All-In-One Image
Restoration [5.573836220587265]
This work proposes an efficient Dynamic Reference Modeling paradigm (DRM-IR)
DRM-IR consists of task-adaptive degradation modeling and model-based image restoring.
Experiments on multiple benchmark datasets show that our DRM-IR achieves state-of-the-art in All-In-One IR.
arXiv Detail & Related papers (2023-07-15T02:42:19Z) - Learning Enriched Features for Fast Image Restoration and Enhancement [166.17296369600774]
This paper presents a holistic goal of maintaining spatially-precise high-resolution representations through the entire network.
We learn an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.
Our approach achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement.
arXiv Detail & Related papers (2022-04-19T17:59:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.