InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentation

• URL: http://arxiv.org/abs/2609.02004v1
• Date: Wed, 02 Sep 2026 02:28:29 GMT
• Title: InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentation
• Abstract summary: InstEditSeg is a unified generative framework that reformulates medical segmentation as an instruction-driven problem.<n>Instead of emitting a mask, the model renders a color-coded overlay on the original image, conditioned on a textual instruction.
• Score: 11.04143971119602
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate segmentation of polyps and skin lesions is pivotal for clinical diagnosis, yet existing methods struggle with low contrast, ambiguous boundaries, and cross-domain distribution discrepancies. Discriminative networks and most diffusion-based segmentation approaches predict standalone binary masks, leaving the visual priors of large-scale pretrained generative models largely unexploited. We propose InstEditSeg, a unified generative framework that reformulates medical segmentation as an instruction-driven image editing problem. Instead of emitting a mask, the model renders a color-coded overlay on the original image, conditioned on a textual instruction, so that the edited output aligns with the natural image distribution learned by latent diffusion models and mitigates the domain gap between natural and medical imagery. To recover fine anatomical structures, we introduce DINOv3 as an auxiliary visual encoder and a DINO Feature Guidance Block that builds a multi-scale feature pyramid. The pyramid is fused into the diffusion U-Net by channel concatenation and zero-initialized convolution so that hierarchical discriminative priors can be injected without perturbing the pretrained weights. A dual-branch classifier-free guidance strategy requiring only two forward passes per denoising step reduces inference cost. On polyp and skin lesion benchmarks the framework achieves accuracy competitive with strong discriminative baselines, and it further demonstrates concrete advantages of the generative formulation: notably better cross-domain generalization on unseen data, more complete multi-lesion segmentation, instruction-conditioned task control, and sampling flexibility. We also analyze the strengths and limitations of the paradigm, including its color sensitivity and unsupported attribute-conditioned selection. Code is available at: https://github.com/wincharm001/InstEditSeg.

Related papers

• Unpaired Image-to-Image Translation via a Self-Supervised Semantic Bridge [59.247871132422006]
  Adversarial diffusion and diffusion-inversion methods have advanced unpaired image-to-image translation, but each faces key limitations.<n>We propose the Self-Supervised Semantic Bridge ( SSB), a versatile framework that integrates external semantic priors into diffusion bridge models.<n>Our key idea is to leverage self-supervised visual encoders to learn representations that are invariant to appearance changes but capture geometric structure.
  arXiv  Detail & Related papers  (2026-02-18T18:05:00Z)
• ProGiDiff: Prompt-Guided Diffusion-Based Medical Image Segmentation [12.964514627034122]
  We propose a novel framework called ProGiDiff that leverages existing image generation models for medical image segmentation purposes.<n> Specifically, we propose a ControlNet-style conditioning mechanism with a custom encoder, suitable for image conditioning, to steer a pre-trained diffusion model to output segmentation masks.<n>Our experiment on organ segmentation from CT images demonstrates strong performance compared to previous methods and could greatly benefit from an expert-in-the-loop setting.
  arXiv  Detail & Related papers  (2026-01-22T15:56:21Z)
• LapFM: A Laparoscopic Segmentation Foundation Model via Hierarchical Concept Evolving Pre-training [39.51756689308245]
  We present LapFM, a foundation model designed to evolve robust segmentation capabilities from massive unlabeled surgical images.<n>We propose a Confidence-driven Evolving Labeling that iteratively generates and filters pseudo-labels based on hierarchical consistency.<n>This process yields LapBench-114K, a large-scale benchmark comprising 114K image-mask pairs.
  arXiv  Detail & Related papers  (2025-12-09T10:09:31Z)
• Diffusion Model in Latent Space for Medical Image Segmentation Task [0.0]
  MedSegLatDiff is a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation.<n>It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps.
  arXiv  Detail & Related papers  (2025-12-01T05:26:43Z)
• Zero-shot Segmentation of Skin Conditions: Erythema with Edit-Friendly Inversion [0.27624021966289597]
  This study proposes a zero-shot image segmentation framework for detecting erythema (redness of the skin) using edit-friendly inversion in diffusion models.<n>The method synthesizes reference images of the same patient that are free from erythema via generative editing and then accurately aligns these references with the original images.
  arXiv  Detail & Related papers  (2025-08-02T12:00:35Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• Dual-scale Enhanced and Cross-generative Consistency Learning for Semi-supervised Medical Image Segmentation [49.57907601086494]
  Medical image segmentation plays a crucial role in computer-aided diagnosis. We propose a novel Dual-scale Enhanced and Cross-generative consistency learning framework for semi-supervised medical image (DEC-Seg)
  arXiv  Detail & Related papers  (2023-12-26T12:56:31Z)
• Introducing Shape Prior Module in Diffusion Model for Medical Image Segmentation [7.7545714516743045]
  We propose an end-to-end framework called VerseDiff-UNet, which leverages the denoising diffusion probabilistic model (DDPM) Our approach integrates the diffusion model into a standard U-shaped architecture. We evaluate our method on a single dataset of spine images acquired through X-ray imaging.
  arXiv  Detail & Related papers  (2023-09-12T03:05:00Z)
• Lesion-aware Dynamic Kernel for Polyp Segmentation [49.63274623103663]
  We propose a lesion-aware dynamic network (LDNet) for polyp segmentation. It is a traditional u-shape encoder-decoder structure incorporated with a dynamic kernel generation and updating scheme. This simple but effective scheme endows our model with powerful segmentation performance and generalization capability.
  arXiv  Detail & Related papers  (2023-01-12T09:53:57Z)
• Self-Supervised Correction Learning for Semi-Supervised Biomedical Image Segmentation [84.58210297703714]
  We propose a self-supervised correction learning paradigm for semi-supervised biomedical image segmentation. We design a dual-task network, including a shared encoder and two independent decoders for segmentation and lesion region inpainting. Experiments on three medical image segmentation datasets for different tasks demonstrate the outstanding performance of our method.
  arXiv  Detail & Related papers  (2023-01-12T08:19:46Z)
• BoxPolyp:Boost Generalized Polyp Segmentation Using Extra Coarse Bounding Box Annotations [79.17754846553866]
  We propose a boosted BoxPolyp model to make full use of both accurate mask and extra coarse box annotations. In practice, box annotations are applied to alleviate the over-fitting issue of previous polyp segmentation models. Our proposed model outperforms previous state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-12-07T07:45:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.