Fundus to Fluorescein Angiography Video Generation as a Retinal Generative Foundation Model

• URL: http://arxiv.org/abs/2410.13242v2
• Date: Fri, 18 Oct 2024 15:41:44 GMT
• Title: Fundus to Fluorescein Angiography Video Generation as a Retinal Generative Foundation Model
• Authors: Weiyi Zhang, Jiancheng Yang, Ruoyu Chen, Siyu Huang, Pusheng Xu, Xiaolan Chen, Shanfu Lu, Hongyu Cao, Mingguang He, Danli Shi,
• Abstract summary: We introduce Fundus2Video, an autoregressive generative adversarial network (GAN) model that generates dynamic FFA videos from single CF images. Fundus2Video excels in video generation, achieving an FVD of 1497.12 and a PSNR of 11.77. These findings position Fundus2Video as a powerful, non-invasive alternative to FFA exams and a versatile retinal generative foundation model.
• Score: 13.378309762602095
• License:
• Abstract: Fundus fluorescein angiography (FFA) is crucial for diagnosing and monitoring retinal vascular issues but is limited by its invasive nature and restricted accessibility compared to color fundus (CF) imaging. Existing methods that convert CF images to FFA are confined to static image generation, missing the dynamic lesional changes. We introduce Fundus2Video, an autoregressive generative adversarial network (GAN) model that generates dynamic FFA videos from single CF images. Fundus2Video excels in video generation, achieving an FVD of 1497.12 and a PSNR of 11.77. Clinical experts have validated the fidelity of the generated videos. Additionally, the model's generator demonstrates remarkable downstream transferability across ten external public datasets, including blood vessel segmentation, retinal disease diagnosis, systemic disease prediction, and multimodal retrieval, showcasing impressive zero-shot and few-shot capabilities. These findings position Fundus2Video as a powerful, non-invasive alternative to FFA exams and a versatile retinal generative foundation model that captures both static and temporal retinal features, enabling the representation of complex inter-modality relationships.

Related papers

• Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis [55.959002385347645]
  Scaling by training on large datasets has been shown to enhance the quality and fidelity of image generation and manipulation with diffusion models. Latent Drifting enables diffusion models to be conditioned for medical images fitted for the complex task of counterfactual image generation. Our results demonstrate significant performance gains in various scenarios when combined with different fine-tuning schemes.
  arXiv  Detail & Related papers  (2024-12-30T01:59:34Z)
• Rethinking Diffusion-Based Image Generators for Fundus Fluorescein Angiography Synthesis on Limited Data [9.343430674144976]
  We propose a novel latent diffusion model-based framework to overcome the challenge of limited medical data. Our framework achieves state-of-the-art results compared to existing methods, offering significant potential to enhance ophthalmic diagnostics and patient care.
  arXiv  Detail & Related papers  (2024-12-17T10:37:46Z)
• Fundus2Video: Cross-Modal Angiography Video Generation from Static Fundus Photography with Clinical Knowledge Guidance [22.92060034450964]
  Fundus Fluorescein Angiography is a critical tool for assessing retinal vascular dynamics and aiding in the diagnosis of eye diseases. Current CF to FFA translation methods are limited to static generation. We introduce an autoregressive GAN for smooth, memory-saving frame-by-frame FFA synthesis.
  arXiv  Detail & Related papers  (2024-08-27T17:30:49Z)
• Let Video Teaches You More: Video-to-Image Knowledge Distillation using DEtection TRansformer for Medical Video Lesion Detection [91.97935118185]
  We propose Video-to-Image knowledge distillation for the task of medical video lesion detection. By distilling multi-frame contexts into a single frame, the proposed V2I-DETR combines the advantages of utilizing temporal contexts from video-based models and the inference speed of image-based models. V2I-DETR outperforms previous state-of-the-art methods by a large margin while achieving the real-time inference speed (30 FPS) as the image-based model.
  arXiv  Detail & Related papers  (2024-08-26T07:17:05Z)
• StealthDiffusion: Towards Evading Diffusion Forensic Detection through Diffusion Model [62.25424831998405]
  StealthDiffusion is a framework that modifies AI-generated images into high-quality, imperceptible adversarial examples. It is effective in both white-box and black-box settings, transforming AI-generated images into high-quality adversarial forgeries.
  arXiv  Detail & Related papers  (2024-08-11T01:22:29Z)
• ViT-DAE: Transformer-driven Diffusion Autoencoder for Histopathology Image Analysis [4.724009208755395]
  We present ViT-DAE, which integrates vision transformers (ViT) and diffusion autoencoders for high-quality histopathology image synthesis. Our approach outperforms recent GAN-based and vanilla DAE methods in generating realistic images.
  arXiv  Detail & Related papers  (2023-04-03T15:00:06Z)
• VIDM: Video Implicit Diffusion Models [75.90225524502759]
  Diffusion models have emerged as a powerful generative method for synthesizing high-quality and diverse set of images. We propose a video generation method based on diffusion models, where the effects of motion are modeled in an implicit condition. We improve the quality of the generated videos by proposing multiple strategies such as sampling space truncation, robustness penalty, and positional group normalization.
  arXiv  Detail & Related papers  (2022-12-01T02:58:46Z)
• VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers [0.0]
  In Fluorescein Angiography (FA), an injected dye is injected in the bloodstream to image the vascular structure of the retina. Fundus imaging is a non-invasive technique used for photographing the retina but does not have sufficient fidelity for capturing its vascular structure. We propose a novel conditional generative adversarial network (GAN) capable of simultaneously synthesizing FA images from fundus photographs while predicting retinal degeneration.
  arXiv  Detail & Related papers  (2021-04-14T10:32:36Z)
• Attention2AngioGAN: Synthesizing Fluorescein Angiography from Retinal Fundus Images using Generative Adversarial Networks [0.0]
  Fluorescein Angiography (FA) is a technique that employs the designated camera for Fundus photography incorporating excitation and barrier filters. FA also requires fluorescein dye that is injected intravenously, which might cause adverse effects ranging from nausea, vomiting to even fatal anaphylaxis. We introduce an Attention-based Generative network that can synthesize Fluorescein Angiography from Fundus images.
  arXiv  Detail & Related papers  (2020-07-17T18:58:44Z)
• Diffusion-Weighted Magnetic Resonance Brain Images Generation with Generative Adversarial Networks and Variational Autoencoders: A Comparison Study [55.78588835407174]
  We show that high quality, diverse and realistic-looking diffusion-weighted magnetic resonance images can be synthesized using deep generative models. We present two networks, the Introspective Variational Autoencoder and the Style-Based GAN, that qualify for data augmentation in the medical field.
  arXiv  Detail & Related papers  (2020-06-24T18:00:01Z)
• Modeling and Enhancing Low-quality Retinal Fundus Images [167.02325845822276]
  Low-quality fundus images increase uncertainty in clinical observation and lead to the risk of misdiagnosis. We propose a clinically oriented fundus enhancement network (cofe-Net) to suppress global degradation factors. Experiments on both synthetic and real images demonstrate that our algorithm effectively corrects low-quality fundus images without losing retinal details.
  arXiv  Detail & Related papers  (2020-05-12T08:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.