T-IMPACT: A Severity-Aware Benchmark for Contextual Image-Text Manipulation

• URL: http://arxiv.org/abs/2606.22339v1
• Date: Sun, 21 Jun 2026 05:07:20 GMT
• Title: T-IMPACT: A Severity-Aware Benchmark for Contextual Image-Text Manipulation
• Abstract summary: We introduce T-context, a first-release severity-aware benchmark for manipulated news-style image-text pairs.<n>T- contains 98,786 examples spanning pristine, image-only, text-only, and joint manipulations.<n>The pipeline extracts semantic anchors, grounds them spatially, performs localized image edits and constrained caption rewrites.
• Score: 9.049034101566642
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent advances in vision-language models and generative editing systems have made it increasingly easy to produce persuasive multimodal misinformation by altering images, text, or both jointly. However, existing datasets focus mainly on authenticity, out-of-context mismatch, or manipulation type, and rarely capture how strongly an edit changes the likely interpretation of a post. We introduce T-IMPACT, a first-release severity-aware benchmark for manipulated news-style image-text pairs. T-IMPACT contains 98,786 examples spanning pristine, image-only, text-only, and joint manipulations, with a calibrated continuous severity signal, coarse low/medium/high labels, and supporting grounding metadata. Starting from a news image-text pair, the pipeline extracts semantic anchors, grounds them spatially, performs localized image edits and constrained caption rewrites, and calibrates contextual-impact scores using limited human ratings. In this release, the calibrated continuous score is the primary severity target, while the low/medium/high bands should be interpreted as coarse operating buckets rather than balanced classes. Experiments show that current models recover some authenticity signal, but severity prediction remains substantially harder and only weakly aligned with human judgment. T-IMPACT provides an initial benchmark for studying multimodal manipulation beyond binary real/fake classification toward graded contextual impact.

Related papers

• Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models [14.752088383510788]
  Vision-language models (VLMs) are often deployed on text-only inputs, although they are trained with images.<n>We find that removing the vision modality causes large drops in accuracy and severe miscalibration, and the model does not behave like its original language backbone under text-only prompting.<n>We propose the Latent Imagination Module (LIM), a lightweight cross-attention module that predicts imagined latent embeddings from textual input and feeds them into a frozen VLM backbone without pixel-level image synthesis.
  arXiv  Detail & Related papers  (2026-04-03T10:03:02Z)
• Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection [53.789057575175065]
  Region-Aware Prompt Augmentation (RAPTA) and Attention-Driven Multimodal Copy Detection (ADMCD) are presented.<n>RAPTA uses an object detector to find salient regions and turn them into semantically grounded prompt variants.<n>ADMCD aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation.<n>Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.
  arXiv  Detail & Related papers  (2026-03-13T15:16:27Z)
• TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment [68.91073792449201]
  We propose TextGuider, a training-free method that encourages accurate and complete text appearance.<n>Specifically, we analyze attention patterns in Multi-Modal Diffusion Transformer(MM-DiT) models, particularly for text-related tokens intended to be rendered in the image.<n>Our method achieves state-of-the-art performance in test-time text rendering, with significant gains in recall and strong results in OCR accuracy and CLIP score.
  arXiv  Detail & Related papers  (2025-12-10T06:18:30Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Debiasing Vison-Language Models with Text-Only Training [15.069736314663352]
  We propose a Text-Only Debiasing framework called TOD, leveraging a text-as-image training paradigm to mitigate visual biases. To address the limitations, we propose a Text-Only Debiasing framework called TOD, leveraging a text-as-image training paradigm to mitigate visual biases.
  arXiv  Detail & Related papers  (2024-10-12T04:34:46Z)
• When Text and Images Don't Mix: Bias-Correcting Language-Image Similarity Scores for Anomaly Detection [35.09035417676343]
  We show that the embeddings of text inputs unexpectedly tightly cluster together, far away from image embeddings, contrary to the model's contrastive training objective. We propose a novel methodology called BLISS which directly accounts for this similarity bias through the use of an auxiliary, external set of text inputs.
  arXiv  Detail & Related papers  (2024-07-24T08:20:02Z)
• Is it an i or an l: Test-time Adaptation of Text Line Recognition Models [9.149602257966917]
  We introduce the problem of adapting text line recognition models during test time. We propose an iterative self-training approach that uses feedback from the language model to update the optical model. Experimental results show that the proposed adaptation method offers an absolute improvement of up to 8% in character error rate.
  arXiv  Detail & Related papers  (2023-08-29T05:44:00Z)
• Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback [20.78162037954646]
  We introduce a decompositional approach towards evaluation and improvement of text-to-image alignment. Human user studies indicate that the proposed approach surpasses previous state-of-the-art by 8.7% in overall text-to-image alignment accuracy.
  arXiv  Detail & Related papers  (2023-07-10T17:54:57Z)
• Discriminative Class Tokens for Text-to-Image Diffusion Models [102.88033622546251]
  We propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text.<n>Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images.<n>We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier.
  arXiv  Detail & Related papers  (2023-03-30T05:25:20Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)
• NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media [93.51739200834837]
  We propose a dataset where both image and text are unmanipulated but mismatched. We introduce several strategies for automatic retrieval of suitable images for the given captions. Our large-scale automatically generated NewsCLIPpings dataset requires models to jointly analyze both modalities.
  arXiv  Detail & Related papers  (2021-04-13T01:53:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.