On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction

• URL: http://arxiv.org/abs/2511.19641v1
• Date: Mon, 24 Nov 2025 19:15:47 GMT
• Title: On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction
• Authors: Ruimin Feng, Xingxin He, Ronald Mercer, Zachary Stewart, Fang Liu,
• Abstract summary: We propose a semantic distribution-guided reconstruction framework that uses a pre-trained vision-language foundation model.<n>A contrastive objective aligns the reconstructed representation with the target semantic distribution.<n>Experiments on knee and brain datasets demonstrate that semantic priors from images preserve fine anatomical structures and achieve superior perceptual quality.
• Score: 5.796028565806211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Purpose: To investigate whether a vision-language foundation model can enhance undersampled MRI reconstruction by providing high-level contextual information beyond conventional priors. Methods: We proposed a semantic distribution-guided reconstruction framework that uses a pre-trained vision-language foundation model to encode both the reconstructed image and auxiliary information into high-level semantic features. A contrastive objective aligns the reconstructed representation with the target semantic distribution, ensuring consistency with high-level perceptual cues. The proposed objective works with various deep learning-based reconstruction methods and can flexibly incorporate semantic priors from multimodal sources. To test the effectiveness of these semantic priors, we evaluated reconstruction results guided by priors derived from either image-only or image-language auxiliary information. Results: Experiments on knee and brain datasets demonstrate that semantic priors from images preserve fine anatomical structures and achieve superior perceptual quality, as reflected in lower LPIPS values, higher Tenengrad scores, and improved scores in the reader study, compared with conventional regularization. The image-language information further expands the semantic distribution and enables high-level control over reconstruction attributes. Across all evaluations, the contrastive objective consistently guided the reconstructed features toward the desired semantic distributions while maintaining data fidelity, demonstrating the effectiveness of the proposed optimization framework. Conclusion: The study highlights that vision-language foundation models can improve undersampled MRI reconstruction through semantic-space optimization.

Related papers

• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• Understanding the Transfer Limits of Vision Foundation Models [38.99867932557529]
  Foundation models leverage large-scale pretraining to capture extensive knowledge, demonstrating generalization in a wide range of language tasks.<n>We postulate that this limitation arises from a mismatch between pretraining objectives and the demands of downstream vision-and-imaging tasks.<n>Pretraining strategies like masked image reconstruction or contrastive learning shape representations for tasks such as recovery of generic visual patterns or global semantic structures.<n>Our findings indicate that better alignment between pretraining and downstream tasks, measured by simple divergence metrics such as maximum-mean-discrepancy (MMD) between the same features before and after fine-tuning, correlates with greater performance improvements and
  arXiv  Detail & Related papers  (2026-01-22T12:07:56Z)
• GloTok: Global Perspective Tokenizer for Image Reconstruction and Generation [51.95701097588426]
  We introduce a Global Perspective Tokenizer (GloTok) to model a more uniform semantic distribution of tokenized features.<n>A residual learning module is proposed to recover the fine-grained details to minimize the reconstruction error caused by quantization.<n>Experiments on the standard ImageNet-1k benchmark clearly show that our proposed method achieves state-of-the-art reconstruction performance and generation quality.
  arXiv  Detail & Related papers  (2025-11-18T06:40:26Z)
• Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction [65.67001243986981]
  We propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling.<n>MindHier achieves superior semantic fidelity, 4.67x faster inference, and more deterministic results than the diffusion-based baselines.
  arXiv  Detail & Related papers  (2025-10-25T15:40:07Z)
• RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Models [0.7165255458140439]
  Vision-Language Foundation Models (VLFM) have shown a tremendous increase in performance in terms of generating high-resolution, photorealistic natural images.<n>We propose a multi-stage architecture where a pre-trained VLFM provides a cursory semantic understanding, while a reinforcement learning algorithm refines the alignment through an iterative process.<n>The reward signal is designed to align the semantic information of the text with synthesized images.
  arXiv  Detail & Related papers  (2025-03-20T01:51:05Z)
• Optimized two-stage AI-based Neural Decoding for Enhanced Visual Stimulus Reconstruction from fMRI Data [2.0851013563386247]
  This work proposes a non-linear deep network to improve fMRI latent space representation, optimizing the dimensionality alike.<n>Experiments on the Natural Scenes dataset showed that the proposed architecture improved the structural similarity of the reconstructed image by about 2% with respect to the state-of-the-art model.<n>The noise sensitivity analysis of the LDM showed that the role of the first stage was fundamental to predict the stimulus featuring high structural similarity.
  arXiv  Detail & Related papers  (2024-12-17T16:42:55Z)
• Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation [56.87049651707208]
  Few-shot Semantic has evolved into In-context tasks, morphing into a crucial element in assessing generalist segmentation models. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework.
  arXiv  Detail & Related papers  (2024-10-03T10:33:49Z)
• SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration [20.873676111265656]
  SSP-IR aims to fully exploit semantic and structure priors from low-quality images.<n>Our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-07-04T04:55:14Z)
• JoReS-Diff: Joint Retinex and Semantic Priors in Diffusion Model for Low-light Image Enhancement [69.6035373784027]
  Low-light image enhancement (LLIE) has achieved promising performance by employing conditional diffusion models. Previous methods may neglect the importance of a sufficient formulation of task-specific condition strategy. We propose JoReS-Diff, a novel approach that incorporates Retinex- and semantic-based priors as the additional pre-processing condition.
  arXiv  Detail & Related papers  (2023-12-20T08:05:57Z)
• Stable Deep MRI Reconstruction using Generative Priors [13.400444194036101]
  We propose a novel deep neural network based regularizer which is trained in a generative setting on reference magnitude images only. The results demonstrate competitive performance, on par with state-of-the-art end-to-end deep learning methods.
  arXiv  Detail & Related papers  (2022-10-25T08:34:29Z)
• Federated Learning of Generative Image Priors for MRI Reconstruction [5.3963856146595095]
  Multi-institutional efforts can facilitate training of deep MRI reconstruction models, albeit privacy risks arise during cross-site sharing of imaging data. We introduce a novel method for MRI reconstruction based on Federated learning of Generative IMage Priors (FedGIMP) FedGIMP leverages a two-stage approach: cross-site learning of a generative MRI prior, and subject-specific injection of the imaging operator.
  arXiv  Detail & Related papers  (2022-02-08T22:17:57Z)
• Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learning [62.17532253489087]
  Deep learning methods have been shown to produce superior performance on MR image reconstruction. These methods require large amounts of data which is difficult to collect and share due to the high cost of acquisition and medical data privacy regulations. We propose a federated learning (FL) based solution in which we take advantage of the MR data available at different institutions while preserving patients' privacy.
  arXiv  Detail & Related papers  (2021-03-03T03:04:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.