InvFussion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems

• URL: http://arxiv.org/abs/2504.01689v1
• Date: Wed, 02 Apr 2025 12:40:57 GMT
• Title: InvFussion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems
• Authors: Noam Elata, Hyungjin Chung, Jong Chul Ye, Tomer Michaeli, Michael Elad,
• Abstract summary: This work introduces a framework that combines the strong performance of supervised approaches and the flexibility of zero-shot methods.<n>A novel architectural design seamlessly integrates the degradation operator directly into the denoiser.<n> Experimental results on the FFHQ and ImageNet datasets demonstrate state-of-the-art posterior-sampling performance.
• Score: 76.39776789410088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Models have demonstrated remarkable capabilities in handling inverse problems, offering high-quality posterior-sampling-based solutions. Despite significant advances, a fundamental trade-off persists, regarding the way the conditioned synthesis is employed: Training-based methods achieve high quality results, while zero-shot approaches trade this with flexibility. This work introduces a framework that combines the best of both worlds -- the strong performance of supervised approaches and the flexibility of zero-shot methods. This is achieved through a novel architectural design that seamlessly integrates the degradation operator directly into the denoiser. In each block, our proposed architecture applies the degradation operator on the network activations and conditions the output using the attention mechanism, enabling adaptation to diverse degradation scenarios while maintaining high performance. Our work demonstrates the versatility of the proposed architecture, operating as a general MMSE estimator, a posterior sampler, or a Neural Posterior Principal Component estimator. This flexibility enables a wide range of downstream tasks, highlighting the broad applicability of our framework. The proposed modification of the denoiser network offers a versatile, accurate, and computationally efficient solution, demonstrating the advantages of dedicated network architectures for complex inverse problems. Experimental results on the FFHQ and ImageNet datasets demonstrate state-of-the-art posterior-sampling performance, surpassing both training-based and zero-shot alternatives.

Related papers

• LIFT: Latent Implicit Functions for Task- and Data-Agnostic Encoding [4.759109475818876]
  Implicit Neural Representations (INRs) are proving to be a powerful paradigm in unifying task modeling across diverse data domains.<n>We introduce LIFT, a novel, high-performance framework that captures multiscale information through meta-learning.<n>We also introduce ReLIFT, an enhanced variant of LIFT that incorporates residual connections and expressive frequency encodings.
  arXiv  Detail & Related papers  (2025-03-19T17:00:58Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Minimizing Embedding Distortion for Robust Out-of-Distribution Performance [1.0923877073891446]
  We introduce a novel approach we call "similarity loss", which can be incorporated into the fine-tuning process of any task. We evaluate our approach on two diverse tasks: image classification on satellite imagery and face recognition.
  arXiv  Detail & Related papers  (2024-09-11T19:22:52Z)
• Fast Unsupervised Tensor Restoration via Low-rank Deconvolution [0.09208007322096533]
  Low-rank Deconvolution (LRD) has appeared as a new multi-dimensional representation model that enjoys important efficiency and flexibility properties. We ask ourselves if this analytical model can compete against Deep Learning (DL) frameworks like Deep Image Prior (DIP) or Blind-Spot Networks (BSN)
  arXiv  Detail & Related papers  (2024-06-15T16:04:49Z)
• GFN: A graph feedforward network for resolution-invariant reduced operator learning in multifidelity applications [0.0]
  This work presents a novel resolution-invariant model order reduction strategy for multifidelity applications. We base our architecture on a novel neural network layer developed in this work, the graph feedforward network. We exploit the method's capability of training and testing on different mesh sizes in an autoencoder-based reduction strategy for parametrised partial differential equations.
  arXiv  Detail & Related papers  (2024-06-05T18:31:37Z)
• Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective [33.45878576396101]
  Feature transformation aims to reconstruct an effective representation space by mathematically refining the existing features. Existing research predominantly focuses on domain knowledge-based feature engineering or learning latent representations. Our initial work took a pioneering step towards this challenge by introducing a novel self-optimizing framework.
  arXiv  Detail & Related papers  (2023-06-29T12:29:21Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
  arXiv  Detail & Related papers  (2023-01-28T01:50:42Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)
• Iterative Network for Image Super-Resolution [69.07361550998318]
  Single image super-resolution (SISR) has been greatly revitalized by the recent development of convolutional neural networks (CNN) This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization.
  arXiv  Detail & Related papers  (2020-05-20T11:11:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.