Style-Extracting Diffusion Models for Semi-Supervised Histopathology Segmentation

• URL: http://arxiv.org/abs/2403.14429v1
• Date: Thu, 21 Mar 2024 14:36:59 GMT
• Title: Style-Extracting Diffusion Models for Semi-Supervised Histopathology Segmentation
• Authors: Mathias Öttl, Frauke Wilm, Jana Steenpass, Jingna Qiu, Matthias Rübner, Arndt Hartmann, Matthias Beckmann, Peter Fasching, Andreas Maier, Ramona Erber, Bernhard Kainz, Katharina Breininger,
• Abstract summary: Style-Extracting Diffusion Models generate images with unseen characteristics beneficial for downstream tasks. In this work, we show the capability of our method on a natural image dataset as a proof-of-concept. We verify the added value of the generated images by showing improved segmentation results and lower performance variability between patients.
• Score: 6.479933058008389
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning-based image generation has seen significant advancements with diffusion models, notably improving the quality of generated images. Despite these developments, generating images with unseen characteristics beneficial for downstream tasks has received limited attention. To bridge this gap, we propose Style-Extracting Diffusion Models, featuring two conditioning mechanisms. Specifically, we utilize 1) a style conditioning mechanism which allows to inject style information of previously unseen images during image generation and 2) a content conditioning which can be targeted to a downstream task, e.g., layout for segmentation. We introduce a trainable style encoder to extract style information from images, and an aggregation block that merges style information from multiple style inputs. This architecture enables the generation of images with unseen styles in a zero-shot manner, by leveraging styles from unseen images, resulting in more diverse generations. In this work, we use the image layout as target condition and first show the capability of our method on a natural image dataset as a proof-of-concept. We further demonstrate its versatility in histopathology, where we combine prior knowledge about tissue composition and unannotated data to create diverse synthetic images with known layouts. This allows us to generate additional synthetic data to train a segmentation network in a semi-supervised fashion. We verify the added value of the generated images by showing improved segmentation results and lower performance variability between patients when synthetic images are included during segmentation training. Our code will be made publicly available at [LINK].

Related papers

• CSGO: Content-Style Composition in Text-to-Image Generation [18.737752718787196]
  We present a data construction pipeline for content-style-stylized image triplets that generates and automatically cleanses stylized data triplets. Based on this pipeline, we construct a dataset IMAGStyle, the first large-scale style transfer dataset containing 210k image triplets. We propose CSGO, a style transfer model based on end-to-end training, which explicitly decouples content and style features.
  arXiv  Detail & Related papers  (2024-08-29T17:59:30Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• Measuring Style Similarity in Diffusion Models [118.22433042873136]
  We present a framework for understanding and extracting style descriptors from images. Our framework comprises a new dataset curated using the insight that style is a subjective property of an image. We also propose a method to extract style attribute descriptors that can be used to style of a generated image to the images used in the training dataset of a text-to-image model.
  arXiv  Detail & Related papers  (2024-04-01T17:58:30Z)
• Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model [80.61157097223058]
  A prevalent strategy to bolster image classification performance is through augmenting the training set with synthetic images generated by T2I models. In this study, we scrutinize the shortcomings of both current generative and conventional data augmentation techniques. We introduce an innovative inter-class data augmentation method known as Diff-Mix, which enriches the dataset by performing image translations between classes.
  arXiv  Detail & Related papers  (2024-03-28T17:23:45Z)
• Learned representation-guided diffusion models for large-image generation [58.192263311786824]
  We introduce a novel approach that trains diffusion models conditioned on embeddings from self-supervised learning (SSL) Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. Augmenting real data by generating variations of real images improves downstream accuracy for patch-level and larger, image-scale classification tasks.
  arXiv  Detail & Related papers  (2023-12-12T14:45:45Z)
• Self-Distilled StyleGAN: Towards Generation from Internet Photos [47.28014076401117]
  We show how StyleGAN can be adapted to work on raw uncurated images collected from the Internet. We propose a StyleGAN-based self-distillation approach, which consists of two main components. The presented technique enables the generation of high-quality images, while minimizing the loss in diversity of the data.
  arXiv  Detail & Related papers  (2022-02-24T17:16:47Z)
• Unlabeled Data Guided Semi-supervised Histopathology Image Segmentation [34.45302976822067]
  Semi-supervised learning (SSL) based on generative methods has been proven to be effective in utilizing diverse image characteristics. We propose a new data guided generative method for histopathology image segmentation by leveraging the unlabeled data distributions. Our method is evaluated on glands and nuclei datasets.
  arXiv  Detail & Related papers  (2020-12-17T02:54:19Z)
• Where to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network [50.19558726384559]
  We propose a biologically inspired framework for image-based fashion product retrieval. Our proposed framework achieves satisfactory performance on three image-based fashion product retrieval benchmarks.
  arXiv  Detail & Related papers  (2020-10-26T06:01:09Z)
• Generating Annotated High-Fidelity Images Containing Multiple Coherent Objects [10.783993190686132]
  We propose a multi-object generation framework that can synthesize images with multiple objects without explicitly requiring contextual information. We demonstrate how coherency and fidelity are preserved with our method through experiments on the Multi-MNIST and CLEVR datasets.
  arXiv  Detail & Related papers  (2020-06-22T11:33:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.