Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers

• URL: http://arxiv.org/abs/2603.13386v1
• Date: Wed, 11 Mar 2026 06:14:11 GMT
• Title: Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers
• Abstract summary: Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail.<n>We propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer.<n>IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods.
• Score: 57.54843029965778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail. However, existing text-guided diffusion models offer only coarse global control and lack the ability to enforce fine-grained structural constraints. Progress is further limited by the absence of large datasets that pair patch-level spatial layouts with detailed diagnostic descriptions, since generating such annotations for gigapixel whole-slide images is prohibitively time-consuming for human experts. To overcome these challenges, we first develop a scalable multi-agent LVLM annotation framework that integrates image description, diagnostic step extraction, and automatic quality judgment into a coordinated pipeline, and we evaluate the reliability of the system through a human verification process. This framework enables efficient construction of fine-grained and clinically aligned supervision at scale. Building on the curated data, we propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer. Through hierarchical multimodal attention, IC-DiT maintains global semantic coherence while accurately preserving structural and morphological details. Extensive experiments on five histopathology datasets show that IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods. In addition, the generated images serve as effective data augmentation resources for downstream tasks such as cancer classification and survival analysis.

Related papers

• Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback [51.08692072066352]
  Text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures.<n>Structured Defect Grounding casts T2I diagnosis as structured set prediction by modeling each defect as a location, type, reason, importance.<n>Our detector outperforms leading proprietary VLMs on structured defect grounding.
  arXiv  Detail & Related papers  (2026-06-04T13:03:16Z)
• MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts [10.686804125536765]
  We propose a flexible framework for controllable volumetric image generation that supports input from radiology reports and segmentation prompts.<n>Our approach allows users to provide segmentation of a specific anatomy or abnormality without requiring full-organ annotations.
  arXiv  Detail & Related papers  (2026-05-31T02:55:05Z)
• Particle Diffusion Matching: Random Walk Correspondence Search for the Alignment of Standard and Ultra-Widefield Fundus Images [47.944896477309555]
  We propose a robust alignment technique for Standard Fundus Images (SFIs) and Ultra-Widefield Fundus Images (UWFIs)<n>Our method, termed Particle Diffusion Matching (PDM), performs alignment through an iterative Random Walk Correspondence Search (RWCS) guided by a diffusion model.<n>PDM achieves state-of-the-art performance across multiple retinal image alignment benchmarks.
  arXiv  Detail & Related papers  (2026-04-11T08:09:53Z)
• Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathology [46.83014413674925]
  STAMP is a spatial transcriptomics-augmented multimodal pathology representation learning framework.<n>Our study shows that self-supervised, gene-guided training provides a robust and task-agnostic signal for learning pathology image representations.<n>We validate STAMP across six datasets and four downstream tasks, where it consistently achieves strong performance.
  arXiv  Detail & Related papers  (2026-02-15T00:59:13Z)
• Structure-constrained Language-informed Diffusion Model for Unpaired Low-dose Computed Tomography Angiography Reconstruction [72.80209358480424]
  overdose of iodinated contrast media (ICM) can cause kidney damage and life-threatening allergic reactions.<n>Deep learning methods can generate CT images of normal-dose ICM from low-dose ICM, reducing the required dose.<n>We propose a Structure-constrained Language-informed Diffusion Model (SLDM) that integrates structural synergy and spatial intelligence.
  arXiv  Detail & Related papers  (2026-01-28T06:54:06Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• Graph Conditioned Diffusion for Controllable Histopathology Image Generation [26.102552837222103]
  We propose graph-based object-level representations for Graph-Conditioned-Diffusion.<n>Our approach generates graph nodes corresponding to each major structure in the image, encapsulating their individual features and relationships.<n>We evaluate this approach using a real-world histopathology use case, demonstrating that our generated data can reliably substitute for annotated patient data in downstream segmentation tasks.
  arXiv  Detail & Related papers  (2025-10-08T15:26:08Z)
• Causal Disentanglement for Robust Long-tail Medical Image Generation [80.15257897500578]
  We propose a novel medical image generation framework, which generates independent pathological and structural features.<n>We leverage a diffusion model guided by pathological findings to model pathological features, enabling the generation of diverse counterfactual images.
  arXiv  Detail & Related papers  (2025-04-20T01:54:18Z)
• RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Models [0.7165255458140439]
  Vision-Language Foundation Models (VLFM) have shown a tremendous increase in performance in terms of generating high-resolution, photorealistic natural images.<n>We propose a multi-stage architecture where a pre-trained VLFM provides a cursory semantic understanding, while a reinforcement learning algorithm refines the alignment through an iterative process.<n>The reward signal is designed to align the semantic information of the text with synthesized images.
  arXiv  Detail & Related papers  (2025-03-20T01:51:05Z)
• A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation [12.948027961485536]
  We propose a novel Weakly Supervised Semantic (WSSS) approach that integrates structural guidance with text-driven strategies to generate high-quality pseudo labels. Our method achieves state-of-the-art performance, highlighting its potential to improve diagnostic accuracy and efficiency in medical imaging.
  arXiv  Detail & Related papers  (2024-11-19T16:20:27Z)
• TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation [33.45471457058221]
  This paper proposes a novel Transformer based medical image semantic segmentation framework called TransAttUnet. In particular, we establish additional multi-scale skip connections between decoder blocks to aggregate the different semantic-scale upsampling features. Our method consistently outperforms the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-07-12T09:17:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.