Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information

• URL: http://arxiv.org/abs/2410.05143v1
• Date: Mon, 7 Oct 2024 15:55:02 GMT
• Title: Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information
• Authors: Timofey Efimov, Harry Dong, Megna Shah, Jeff Simmons, Sean Donegan, Yuejie Chi,
• Abstract summary: We propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information.
• Score: 19.81357829037316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality.

Related papers

• Cross-Modal Guidance for Fast Diffusion-Based Computed Tomography [1.3048920509133808]
  In certain applications, such as neutron CT, it can be expensive to collect large amounts of measurements even for a single scan.<n>One strategy to mitigate this challenge is to leverage a complementary, easily available imaging modality.<n>We propose incorporating an additional modality without retraining the diffusion prior, enabling accelerated imaging of costly modalities.
  arXiv  Detail & Related papers  (2026-03-01T20:13:13Z)
• Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis [55.959002385347645]
  Latent Drifting enables diffusion models to be conditioned for medical images fitted for the complex task of counterfactual image generation. We evaluate our method on three public longitudinal benchmark datasets of brain MRI and chest X-rays for counterfactual image generation.
  arXiv  Detail & Related papers  (2024-12-30T01:59:34Z)
• DiffusionTrend: A Minimalist Approach to Virtual Fashion Try-On [103.89972383310715]
  DiffusionTrend harnesses latent information rich in prior information to capture the nuances of garment details. It delivers a visually compelling try-on experience, underscoring the potential of training-free diffusion model.
  arXiv  Detail & Related papers  (2024-12-19T02:24:35Z)
• MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling [64.09238330331195]
  We propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework. Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss. We show that MMAR demonstrates much more superior performance than other joint multi-modal models.
  arXiv  Detail & Related papers  (2024-10-14T17:57:18Z)
• Taming Diffusion Models for Image Restoration: A Review [14.25759541950917]
  Diffusion models have been applied to low-level computer vision for photo-realistic image restoration. We introduce key constructions in diffusion models and survey contemporary techniques that make use of diffusion models in solving general IR tasks.
  arXiv  Detail & Related papers  (2024-09-16T15:04:14Z)
• DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception [66.88792390480343]
  We propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data, and a smaller base model size.
  arXiv  Detail & Related papers  (2024-05-24T05:46:04Z)
• YaART: Yet Another ART Rendering Technology [119.09155882164573]
  This study introduces YaART, a novel production-grade text-to-image cascaded diffusion model aligned to human preferences. We analyze how these choices affect both the efficiency of the training process and the quality of the generated images. We demonstrate that models trained on smaller datasets of higher-quality images can successfully compete with those trained on larger datasets.
  arXiv  Detail & Related papers  (2024-04-08T16:51:19Z)
• Decoupled Data Consistency with Diffusion Purification for Image Restoration [15.043002968696978]
  We propose a novel diffusion-based image restoration solver that addresses issues by decoupling the reverse process from the data consistency steps. Our approach demonstrates versatility, making it highly adaptable for efficient problem-solving in latent space. The efficacy of our approach is validated through comprehensive experiments across various image restoration tasks, including image denoising, deblurring, inpainting, and super-resolution.
  arXiv  Detail & Related papers  (2024-03-10T00:47:05Z)
• Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction [75.91471250967703]
  We introduce a novel sampling framework called Steerable Conditional Diffusion. This framework adapts the diffusion model, concurrently with image reconstruction, based solely on the information provided by the available measurement. We achieve substantial enhancements in out-of-distribution performance across diverse imaging modalities.
  arXiv  Detail & Related papers  (2023-08-28T08:47:06Z)
• Diffusion Models for Image Restoration and Enhancement -- A Comprehensive Survey [96.99328714941657]
  We present a comprehensive review of recent diffusion model-based methods on image restoration. We classify and emphasize the innovative designs using diffusion models for both IR and blind/real-world IR. We propose five potential and challenging directions for the future research of diffusion model-based IR.
  arXiv  Detail & Related papers  (2023-08-18T08:40:38Z)
• Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
  Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance. We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T12:18:20Z)
• DDRF: Denoising Diffusion Model for Remote Sensing Image Fusion [7.06521373423708]
  Denosing diffusion model, as a generative model, has received a lot of attention in the field of image generation. We introduce diffusion model to the image fusion field, treating the image fusion task as image-to-image translation. Our method can inspire other works and gain insight into this field to better apply the diffusion model to image fusion tasks.
  arXiv  Detail & Related papers  (2023-04-10T12:28:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.