SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization

• URL: http://arxiv.org/abs/2605.27924v1
• Date: Wed, 27 May 2026 03:55:13 GMT
• Title: SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization
• Abstract summary: Public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks.<n>We propose SIGMA, which performs semantic-feature differencing in a vision foundation backbone.<n>It produces a 1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets.
• Score: 64.6372217552272
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-driven image editing has advanced rapidly, but reliably localizing these manipulations requires image manipulation localization (IML) models trained on large pixel-annotated datasets, and there is still no low-cost way to obtain such training data at scale. We observe that these data already exist in disguise: public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks. Recovering these masks automatically is non-trivial: pixel differencing is overwhelmed by diffusion-induced perturbations across all pixels, and instruction-only grounding localizes only what the prompt describes, missing unintended editor side-effects. We propose SIGMA (Semantic-difference Instruction-Grounding Mask Annotator), which performs semantic-feature differencing in a vision foundation backbone and injects an instruction-derived spatial prior into this visual stream via bidirectional cross-modal refinement, amplifying the difference signal at intended-edit regions when the editor faithfully realizes user intent. SIGMA is trained in two complementary stages: Stage I supervises on inpainting masks; Stage II closes the diffusion-domain shift via VAE-roundtrip noise calibration, EMA self-training, and an edit-noise disentanglement loss. SIGMA outperforms existing automatic mask generators on five benchmarks (+12.20% F1, +11.16% IoU). When applied to public editing corpora, it produces a ~1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets, turning previously unused editing data into a model-agnostic supervisory resource for IML. We'll release the full codebase as soon as the paper is accepted.

Related papers

• Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision [3.836588693383881]
  Text-to-image diffusion models can be exploited for open-world entity segmentation without retraining or supervision.<n> Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators.<n>Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000.
  arXiv  Detail & Related papers  (2026-09-06T09:07:02Z)
• GAP-SAM: A Global Artifact Prior for Generalizable AI-Generated Image Manipulation Localization [15.115841637929895]
  We construct COCO-ControlNet with source-image Canny edges and depth maps to align semantics and geometry, improving OOD performance across multiple localizers.<n>We also identify emphboundary adhesion, where fine-tuned segmentation models snap predictions to semantic object contours rather than true manipulation boundaries.<n>GAP-SAM encodes an image and its frozen VAE reconstruction into a global artifact token and injects it into SAM3's feature pyramid via zero-gated FiLM before pixel decoding.
  arXiv  Detail & Related papers  (2026-08-21T09:49:58Z)
• FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation [27.47065769615997]
  Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into pixel-level masks.<n>In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token.<n>We introduce FIRM, a Fine-grained Intra-token Representation of Masks.
  arXiv  Detail & Related papers  (2026-08-14T05:40:36Z)
• Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing [28.295031615161136]
  Large diffusion transformers (DiTs) follow global editing instructions well but consistently leak local edits into unrelated regions.<n>We introduce AdaptEdit, a co-trained, instruction- and region-aware adapter framework.<n> AdaptEdit achieves state-of-the-art results, simultaneously outperforming mask-free and oracle-mask baselines.
  arXiv  Detail & Related papers  (2026-04-26T15:28:02Z)
• From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering [66.2662973049988]
  We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task.<n>We introduce a taxonomy spanning edit primitives and their semantic class of tampered object, linking low-level changes to high-level understanding.<n>Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity.
  arXiv  Detail & Related papers  (2026-03-20T17:59:54Z)
• MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation [10.894281690166418]
  We propose MaskAttn-SDXL, a region-level gating mechanism applied to the cross-attention logits of Stable Diffusion XL(SDXL)'s UNet.<n>MaskAttn-SDXL learns a binary mask per layer, injecting it into each cross-attention logit map before softmax to sparsify token-to-latent interactions.<n>Our model improves spatial compliance and attribute binding in multi-object prompts while preserving overall image quality and diversity.
  arXiv  Detail & Related papers  (2025-09-18T18:57:47Z)
• LlamaSeg: Image Segmentation via Autoregressive Mask Generation [46.17509085054758]
  We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions.<n>We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs.
  arXiv  Detail & Related papers  (2025-05-26T02:22:41Z)
• MaskInversion: Localized Embeddings via Optimization of Explainability Maps [49.50785637749757]
  MaskInversion generates a context-aware embedding for a query image region specified by a mask at test time. It can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation.
  arXiv  Detail & Related papers  (2024-07-29T14:21:07Z)
• Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation [16.863038973001483]
  This work introduces three techniques for diffusion-synthetic semantic segmentation training. First, reliability-aware robust training, originally used in weakly supervised learning, helps segmentation with insufficient synthetic mask quality. Second, large-scale pretraining of whole segmentation models, not only backbones, on synthetic ImageNet-1k-class images with pixel-labels benefits downstream segmentation tasks. Third, we introduce prompt augmentation, data augmentation to the prompt text set to scale up and diversify training images with a limited text resources.
  arXiv  Detail & Related papers  (2023-09-04T05:34:19Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models [68.21154597227165]
  We show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model. Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image.
  arXiv  Detail & Related papers  (2023-03-21T08:43:15Z)
• StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training [64.37272287179661]
  StrucTexTv2 is an effective document image pre-training framework. It consists of two self-supervised pre-training tasks: masked image modeling and masked language modeling. It achieves competitive or even new state-of-the-art performance in various downstream tasks such as image classification, layout analysis, table structure recognition, document OCR, and information extraction.
  arXiv  Detail & Related papers  (2023-03-01T07:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.