Language-Guided Invariance Probing of Vision-Language Models

• URL: http://arxiv.org/abs/2511.13494v1
• Date: Mon, 17 Nov 2025 15:35:49 GMT
• Title: Language-Guided Invariance Probing of Vision-Language Models
• Abstract summary: We introduce Language-Guided Invariance Probing (LGIP) to measure invariance to paraphrases and sensitivity to semantic flips in image-text matching.<n>We show that EVA02-CLIP and large OpenCLIP variants lie on a favorable invariance-sensitivity frontier, combining low paraphrase-induced variance with consistently higher scores for original captions.
• Score: 2.0789144738685956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent vision-language models (VLMs) such as CLIP, OpenCLIP, EVA02-CLIP and SigLIP achieve strong zero-shot performance, but it is unclear how reliably they respond to controlled linguistic perturbations. We introduce Language-Guided Invariance Probing (LGIP), a benchmark that measures (i) invariance to meaning-preserving paraphrases and (ii) sensitivity to meaning-changing semantic flips in image-text matching. Using 40k MS COCO images with five human captions each, we automatically generate paraphrases and rule-based flips that alter object category, color or count, and summarize model behavior with an invariance error, a semantic sensitivity gap and a positive-rate statistic. Across nine VLMs, EVA02-CLIP and large OpenCLIP variants lie on a favorable invariance-sensitivity frontier, combining low paraphrase-induced variance with consistently higher scores for original captions than for their flipped counterparts. In contrast, SigLIP and SigLIP2 show much larger invariance error and often prefer flipped captions to the human descriptions, especially for object and color edits. These failures are largely invisible to standard retrieval metrics, indicating that LGIP provides a model-agnostic diagnostic for the linguistic robustness of VLMs beyond conventional accuracy scores.

Related papers

• SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection [0.0]
  We introduce Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark that evaluates whether models consistently reject factual errors across languages.<n>SWORD generates syntactically well-formed but factually incorrect statements in eight widely spoken languages through controlled perturbations of Wikidata triples.<n>Our evaluation surfaces two critical insights that remain entirely obscured by conventional benchmarks.
  arXiv  Detail & Related papers  (2026-09-08T18:39:01Z)
• Evaluating LLM Personalization via Semantic Constraint Verification [25.539904066738288]
  We introduce Natural Language Inference Constraint Verification (NLICV)<n>NLICV maps sentence meanings to truth-condition sets to verify personalization constraints via a Natural Language Inference (NLI) model.<n>Experiments demonstrate that NLICV aligns closely with human annotations while drastically reducing the latency and token costs associated with LLM judges.
  arXiv  Detail & Related papers  (2026-06-15T08:04:56Z)
• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification [0.0]
  Turkish idiomatic light verb constructions (LVCs) are challenging for multiword expression processing.<n>We frame Turkish LVC detection as a binary classification task (literal meaning vs. idiomatic meaning)<n>We evaluate on a manually created controlled set (N=147) with matched negatives.
  arXiv  Detail & Related papers  (2026-06-05T17:34:07Z)
• Identifying Influential N-grams in Confidence Calibration via Regression Analysis [53.95767806659635]
  We identify what linguistic expressions are related to confidence by applying the regression method.<n>We show that large language models (LLMs) remain overconfident when reasoning is involved.<n>We reveal that confidence calibration is possible by simply suppressing those overconfident expressions without drops in performance.
  arXiv  Detail & Related papers  (2026-04-07T11:58:49Z)
• Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity [6.285055089721361]
  We introduce a principled interpretability framework based on token-level perplexity to test whether models rely on linguistically relevant cues.<n>Our method enables precise, hypothesis-driven analysis without relying on unstable feature-attribution techniques.
  arXiv  Detail & Related papers  (2026-03-31T08:00:55Z)
• LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models [15.955490895662384]
  We introduce LIBERO-Para, a benchmark for fine-grained analysis of linguistic generalization.<n>We observe consistent performance degradation of 22-52 pp under paraphrasing.<n>We propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors.
  arXiv  Detail & Related papers  (2026-03-30T11:27:34Z)
• Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models [7.5752750293638735]
  We introduce a training-free, post-hoc uncertainty estimation method for contrastive vision-language models.<n>Our method is VLM-agnostic, requires no fine-tuning, demonstrates robustness to distribution shift, and works effectively with as few as 10 training images per class.
  arXiv  Detail & Related papers  (2025-11-27T01:48:27Z)
• Contrastive vision-language learning with paraphrasing and negation [0.018665975431697426]
  Contrastive language-Image Pre-training (CLIP) trains two neural networks in contrastive manner to align their image and text embeddings in a shared latent space.<n>Results evaluating CLIP on negated or paraphrased text have shown mixed performance because negation changes meaning radically with minimal lexical changes, while paraphrasing can create very different textual expressions with the same intended meaning.<n>This paper proposes a new CLIP contrastive loss function accounting for both paraphrasing and negation, and applies LLM-generated training triples consisting of original, paraphrased and negated textual captions to CLIP-like training models.
  arXiv  Detail & Related papers  (2025-11-20T16:41:36Z)
• On the Brittleness of CLIP Text Encoders [3.482579981426874]
  We present a systematic analysis of the effect of multiple classes of non-semantic query perturbations in a multimedia information retrieval scenario.<n>We find that syntactic and semantic perturbations drive the largest instabilities, while brittleness is concentrated in trivial surface edits such as punctuation and case.
  arXiv  Detail & Related papers  (2025-11-06T10:33:55Z)
• TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptation [9.906359339999039]
  We introduce a novel UDA approach that exploits the robustness of the language modality to guide the adaptation of a vision model.<n>We propose a multimodal soft-contrastive learning loss that aligns the vision and language feature spaces.<n>Our approach outperforms previous methods, setting the new state-of-the-art on classical (DomainNet) and complex (GeoNet) domain shifts.
  arXiv  Detail & Related papers  (2025-08-08T16:51:44Z)
• VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions [16.90061119174727]
  We present CLIP-IN, a novel framework that bolsters CLIP's fine-grained perception through two core innovations.<n> Firstly, we leverage instruction-editing datasets, originally designed for image manipulation, as a unique source of hard negative image-text pairs.<n> Secondly, CLIP-IN incorporates long captions, utilizing rotary positional encodings to capture rich semantic context often missed by standard CLIP.
  arXiv  Detail & Related papers  (2025-08-04T11:57:10Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models [58.936893810674896]
  Face Anti-Spoofing (FAS) is essential for ensuring the security and reliability of facial recognition systems.<n>We introduce a multimodal large language model framework for FAS, termed Interpretable Face Anti-Spoofing (I-FAS)<n>We propose a Spoof-aware Captioning and Filtering (SCF) strategy to generate high-quality captions for FAS images.
  arXiv  Detail & Related papers  (2025-01-03T09:25:04Z)
• VladVA: Discriminative Fine-tuning of LVLMs [67.14293827774827]
  Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the de facto approach for discriminative vision-language representation learning.<n>We propose to combine "the best of both worlds": a new training approach for discriminative fine-tuning of LVLMs.
  arXiv  Detail & Related papers  (2024-12-05T17:54:27Z)
• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis.<n>Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration [60.95748658638956]
  This paper introduces the Multi-Label Confidence task, aiming to provide well-calibrated confidence scores in multi-label scenarios. Existing single-label calibration methods fail to account for category correlations, which are crucial for addressing semantic confusion. We propose the Dynamic Correlation Learning and Regularization algorithm, which leverages multi-grained semantic correlations to better model semantic confusion.
  arXiv  Detail & Related papers  (2024-07-09T13:26:21Z)
• Fine-tuning CLIP Text Encoders with Two-step Paraphrasing [83.3736789315201]
  We introduce a straightforward fine-tuning approach to enhance the representations of CLIP models for paraphrases. Our model, which we call ParaCLIP, exhibits significant improvements over baseline CLIP models across various tasks.
  arXiv  Detail & Related papers  (2024-02-23T06:11:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.