Leveraging Multimodal Large Language Models for All-in-One Image Restoration via a Mixture of Frequency Experts

• URL: http://arxiv.org/abs/2605.11444v2
• Date: Wed, 13 May 2026 03:16:10 GMT
• Title: Leveraging Multimodal Large Language Models for All-in-One Image Restoration via a Mixture of Frequency Experts
• Abstract summary: All-in-one image restoration seeks to recover clean images from inputs affected by diverse and unknown degradations.<n>Recent methods have shown strong performance by identifying degradation characteristics to guide the restoration process.<n>We propose a multimodal large language model (MLLM)-guided image restoration framework that exploits multimodal embeddings as guidance.
• Score: 9.395990800489828
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: All-in-one image restoration seeks to recover clean images from inputs affected by diverse and unknown degradations using a unified framework. Recent methods have shown strong performance by identifying degradation characteristics to guide the restoration process. However, many of them treat degradations as discrete categories, which limits their ability to model the continuous relational structure that arises in composite degradations. To address this issue, we propose a multimodal large language model (MLLM)-guided image restoration framework that exploits multimodal embeddings as guidance for low-level restoration. Specifically, MLLM-derived features are injected into an encoder-decoder architecture through an MLLM-guided fusion block (MGFB) to enhance degradation-aware representations. In addition, we incorporate a mixture-of-frequency-experts (MoFE) module that adaptively combines frequency experts using MLLM-guided contextual cues. To further improve expert routing, we design an MLLM-guided router with a relational alignment loss that encourages routing patterns consistent with the embedding-space relationships of degraded inputs. Extensive experiments on multiple benchmarks show that the proposed method achieves strong performance across diverse restoration settings and establishes a new state of the art on the challenging CDD11 dataset, outperforming previous methods by up to 1.35 dB.

Related papers

• Universal Image Restoration via Internalized Chain-of-Thought Reasoning [52.763760468548504]
  Image restoration seeks to recover high-quality images from degraded inputs but becomes highly ill-posed under complex, mixed degradations.<n>Recent works adopt Chain-of-Thought (CoT) reasoning for multi-round restoration using specialized modules.<n>We introduce CoTIR, a universal image restoration framework that internalizes CoT reasoning within a single model.
  arXiv  Detail & Related papers  (2026-06-16T06:01:36Z)
• ClusIR: Towards Cluster-Guided All-in-One Image Restoration [72.16989784735796]
  ClusIR aims to recover high-quality images from diverse degradations within a unified framework.<n>ClusIR comprises two key components: a Probabilistic Cluster-Guided Routing Mechanism (PCGRM) and a Degradation-Aware Frequency Modulation Module (DAFMM)
  arXiv  Detail & Related papers  (2025-12-11T18:59:47Z)
• FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration [29.26604058624955]
  We propose FAPE-IR, a Frequency-Aware Planning and Execution framework for image restoration.<n>It uses a frozen Multimodal Large Language Model (MLLM) as a planner to analyze degraded images and generate concise, frequency-aware restoration plans.<n>By coupling semantic planning with frequency-based restoration, FAPE-IR offers a unified and interpretable solution for all-in-one image restoration.
  arXiv  Detail & Related papers  (2025-11-18T03:33:10Z)
• Learning to Restore Multi-Degraded Images via Ingredient Decoupling and Task-Aware Path Adaptation [51.10017611491389]
  Real-world images often suffer from multiple coexisting degradations, such as rain, noise, and haze coexisting in a single image.<n>We propose an adaptive multi-degradation image restoration network that reconstructs images by leveraging decoupled representations of degradation ingredients.<n>The resulting tightly integrated architecture, termed IMDNet, is extensively validated through experiments.
  arXiv  Detail & Related papers  (2025-11-07T01:50:36Z)
• MoA-VR: A Mixture-of-Agents System Towards All-in-One Video Restoration [62.929029990341796]
  Real-world videos often suffer from complex degradations, such as noise, compression artifacts, and low-light distortions.<n>We propose MoA-VR, which mimics the reasoning and processing procedures of human professionals through three coordinated agents.<n>Specifically, we construct a large-scale and high-resolution video degradation recognition benchmark and build a vision-language model (VLM) driven degradation identifier.
  arXiv  Detail & Related papers  (2025-10-09T17:42:51Z)
• CoLLM: A Large Language Model for Composed Image Retrieval [76.29725148964368]
  Composed Image Retrieval (CIR) is a complex task that aims to retrieve images based on a multimodal query.<n>We present CoLLM, a one-stop framework that generates triplets on-the-fly from image-caption pairs.<n>We leverage Large Language Models (LLMs) to generate joint embeddings of reference images and modification texts.
  arXiv  Detail & Related papers  (2025-03-25T17:59:50Z)
• Mixed Degradation Image Restoration via Local Dynamic Optimization and Conditional Embedding [67.57487747508179]
  Multiple-in-one image restoration (IR) has made significant progress, aiming to handle all types of single degraded image restoration with a single model. In this paper, we propose a novel multiple-in-one IR model that can effectively restore images with both single and mixed degradations.
  arXiv  Detail & Related papers  (2024-11-25T09:26:34Z)
• UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation [50.27688690379488]
  Existing unified methods treat multi-degradation image restoration as a multi-task learning problem. We propose a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multi-degradation restoration and transfers it to specific degradation image restoration tasks.
  arXiv  Detail & Related papers  (2024-09-30T11:16:56Z)
• Training-Free Large Model Priors for Multiple-in-One Image Restoration [24.230376300759573]
  Large Model Driven Image Restoration framework (LMDIR) Our architecture comprises a query-based prompt encoder, degradation-aware transformer block injecting global degradation knowledge. This design facilitates single-stage training paradigm to address various degradations while supporting both automatic and user-guided restoration.
  arXiv  Detail & Related papers  (2024-07-18T05:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.