Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces

• URL: http://arxiv.org/abs/2608.13455v1
• Date: Thu, 13 Aug 2026 16:40:26 GMT
• Title: Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces
• Abstract summary: We show that generated representations and images faithfully inherit phenotype information when evaluated within their originating foundation models.<n>These findings demonstrate that foundation-model latent spaces provide a powerful substrate for controllable retinal synthesis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical foundation models learn latent representations of clinically meaningful phenotypes, yet their ability to support controllable image generation remains largely unexplored. We evaluate four retinal foundation models within the representation tokenizer framework and examine whether demographic and clinical information encoded in latent representations from foundation models is preserved during synthetic image generation. We show that generated representations and images faithfully inherit phenotype information when evaluated within their originating foundation models, consistently outperforming conventional latent diffusion on multiple downstream prediction tasks. However, these gains largely disappear when evaluated using classifiers trained on real images, revealing a previously uncharacterised synthetic-to-real representation gap. These findings demonstrate that foundation-model latent spaces provide a powerful substrate for controllable retinal synthesis while highlighting the need to better align synthetic representations with real-image distributions.

Related papers

• SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation [62.55421542903781]
  We present textbfSpatialReward, a verifiable reward model explicitly designed to evaluate spatial layouts in generated images.<n>Experiments on Stable Diffusion and FLUX show that incorporating SpatialReward into RL training consistently improves spatial consistency and overall generation quality.
  arXiv  Detail & Related papers  (2026-03-23T17:26:35Z)
• Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations [0.0]
  We show vision foundation models can implicitly learn to represent continuous processes from independent static observations.<n>This framework could be applied to other domains where continuous processes are observed through static snapshots.
  arXiv  Detail & Related papers  (2026-01-29T06:50:43Z)
• Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis [55.959002385347645]
  Latent Drifting enables diffusion models to be conditioned for medical images fitted for the complex task of counterfactual image generation.<n>We evaluate our method on three public longitudinal benchmark datasets of brain MRI and chest X-rays for counterfactual image generation.
  arXiv  Detail & Related papers  (2024-12-30T01:59:34Z)
• Unleashing the Potential of Synthetic Images: A Study on Histopathology Image Classification [0.12499537119440242]
  Histopathology image classification is crucial for the accurate identification and diagnosis of various diseases. We show that synthetic images can effectively augment existing datasets, ultimately improving the performance of the downstream histopathology image classification task.
  arXiv  Detail & Related papers  (2024-09-24T12:02:55Z)
• Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors [56.82596340418697]
  We propose a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.
  arXiv  Detail & Related papers  (2024-01-29T10:36:57Z)
• Learned representation-guided diffusion models for large-image generation [58.192263311786824]
  We introduce a novel approach that trains diffusion models conditioned on embeddings from self-supervised learning (SSL) Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. Augmenting real data by generating variations of real images improves downstream accuracy for patch-level and larger, image-scale classification tasks.
  arXiv  Detail & Related papers  (2023-12-12T14:45:45Z)
• Generating Realistic Counterfactuals for Retinal Fundus and OCT Images using Diffusion Models [36.81751569090276]
  Counterfactual reasoning is often used in clinical settings to explain decisions or weigh alternatives. Here, we demonstrate that using a diffusion model in combination with an adversarially robust classifier trained on retinal disease classification tasks enables the generation of highly realistic counterfactuals. In a user study, domain experts found the counterfactuals generated using our method significantly more realistic than counterfactuals generated from a previous method, and even indistinguishable from real images.
  arXiv  Detail & Related papers  (2023-11-20T09:28:04Z)
• On the Out of Distribution Robustness of Foundation Models in Medical Image Segmentation [47.95611203419802]
  Foundations for vision and language, pre-trained on extensive sets of natural image and text data, have emerged as a promising approach. We compare the generalization performance to unseen domains of various pre-trained models after being fine-tuned on the same in-distribution dataset. We further developed a new Bayesian uncertainty estimation for frozen models and used them as an indicator to characterize the model's performance on out-of-distribution data.
  arXiv  Detail & Related papers  (2023-11-18T14:52:10Z)
• Generation of Structurally Realistic Retinal Fundus Images with Diffusion Models [1.9346186297861747]
  We generate artery/vein masks to create the vascular structure, which we then condition to produce retinal fundus images. The proposed method can generate high-quality images with more realistic vascular structures.
  arXiv  Detail & Related papers  (2023-05-11T14:09:05Z)
• Trade-offs in Fine-tuned Diffusion Models Between Accuracy and Interpretability [5.865936619867771]
  We unravel a consequential trade-off between image fidelity as gauged by conventional metrics and model interpretability in generative diffusion models. We present a set of design principles for the development of truly interpretable generative models.
  arXiv  Detail & Related papers  (2023-03-31T09:11:26Z)
• Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains [3.8137985834223502]
  Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets. We explore the sub-components of the Stable Diffusion pipeline to fine-tune the model to generate medical images. Our best-performing model improves upon the stable diffusion baseline and can be conditioned to insert a realistic-looking abnormality on a synthetic radiology image.
  arXiv  Detail & Related papers  (2022-10-09T01:43:08Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.