From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering

• URL: http://arxiv.org/abs/2603.20193v1
• Date: Fri, 20 Mar 2026 17:59:54 GMT
• Title: From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering
• Abstract summary: We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task.<n>We introduce a taxonomy spanning edit primitives and their semantic class of tampered object, linking low-level changes to high-level understanding.<n>Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity.
• Score: 66.2662973049988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing tampering detection benchmarks largely rely on object masks, which severely misalign with the true edit signal: many pixels inside a mask are untouched or only trivially modified, while subtle yet consequential edits outside the mask are treated as natural. We reformulate VLM image tampering from coarse region labels to a pixel-grounded, meaning and language-aware task. First, we introduce a taxonomy spanning edit primitives (replace/remove/splice/inpaint/attribute/colorization, etc.) and their semantic class of tampered object, linking low-level changes to high-level understanding. Second, we release a new benchmark with per-pixel tamper maps and paired category supervision to evaluate detection and classification within a unified protocol. Third, we propose a training framework and evaluation metrics that quantify pixel-level correctness with localization to assess confidence or prediction on true edit intensity, and further measure tamper meaning understanding via semantics-aware classification and natural language descriptions for the predicted regions. We also re-evaluate the existing strong segmentation/localization baselines on recent strong tamper detectors and reveal substantial over- and under-scoring using mask-only metrics, and expose failure modes on micro-edits and off-mask changes. Our framework advances the field from masks to pixels, meanings and language descriptions, establishing a rigorous standard for tamper localization, semantic classification and description. Code and benchmark data are available at https://github.com/VILA-Lab/PIXAR.

Related papers

• SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization [64.6372217552272]
  Public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks.<n>We propose SIGMA, which performs semantic-feature differencing in a vision foundation backbone.<n>It produces a 1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets.
  arXiv  Detail & Related papers  (2026-05-27T03:55:13Z)
• High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation [109.19165503929992]
  We present MaskCLIP++, which uses ground-truth masks instead of generated masks to enhance the mask classification capability of CLIP.<n>After low-cost fine-tuning, MaskCLIP++ significantly improves the mask classification performance on multi-domain datasets.<n>We achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets.
  arXiv  Detail & Related papers  (2024-12-16T05:44:45Z)
• MaskCD: A Remote Sensing Change Detection Network Based on Mask Classification [29.15203530375882]
  Change (CD) from remote sensing (RS) images using deep learning has been widely investigated in the literature. We propose MaskCD to detect changed areas by adaptively generating categorized masks from input image pairs. It reconstructs the desired changed objects by decoding the pixel-wise representations into learnable mask proposals.
  arXiv  Detail & Related papers  (2024-04-18T11:05:15Z)
• Unmasking Anomalies in Road-Scene Segmentation [18.253109627901566]
  Anomaly segmentation is a critical task for driving applications. We propose a paradigm change by shifting from a per-pixel classification to a mask classification. Mask2Anomaly demonstrates the feasibility of integrating an anomaly detection method in a mask-classification architecture.
  arXiv  Detail & Related papers  (2023-07-25T08:23:10Z)
• CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation [104.5033800500497]
  Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects. We propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method.
  arXiv  Detail & Related papers  (2022-12-04T08:53:42Z)
• GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation [16.900404701997502]
  We propose a GAN-based approach that generates images conditioned on latent masks. We show that such mask-conditioned image generation can be learned faithfully when conditioning the masks in a hierarchical manner. It also lets us generate image-mask pairs for training a segmentation network, which outperforms the state-of-the-art unsupervised segmentation methods on established benchmarks.
  arXiv  Detail & Related papers  (2021-12-02T07:57:56Z)
• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling [61.03262873980619]
  Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. We propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model.
  arXiv  Detail & Related papers  (2021-11-24T18:50:47Z)
• Context-Aware Mixup for Domain Adaptive Semantic Segmentation [52.1935168534351]
  Unsupervised domain adaptation (UDA) aims to adapt a model of the labeled source domain to an unlabeled target domain. We propose end-to-end Context-Aware Mixup (CAMix) for domain adaptive semantic segmentation. Experimental results show that the proposed method outperforms the state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2021-08-08T03:00:22Z)
• Per-Pixel Classification is Not All You Need for Semantic Segmentation [184.2905747595058]
  Mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks. We propose MaskFormer, a simple mask classification model which predicts a set of binary masks. Our method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.
  arXiv  Detail & Related papers  (2021-07-13T17:59:50Z)
• Instance Semantic Segmentation Benefits from Generative Adversarial Networks [13.295723883560122]
  We define the problem of predicting masks as a GANs game framework. A segmentation network generates the masks, and a discriminator network decides on the quality of the masks. We report on cellphone recycling, autonomous driving, large-scale object detection, and medical glands.
  arXiv  Detail & Related papers  (2020-10-26T17:47:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.