Semantic Robustness Certification for Vision-Language Models

• URL: http://arxiv.org/abs/2606.18839v1
• Date: Wed, 17 Jun 2026 09:15:50 GMT
• Title: Semantic Robustness Certification for Vision-Language Models
• Abstract summary: Robustness certification determines if a model's prediction changes when transformations are applied to its input.<n>This work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations.
• Score: 23.329728009549388
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model's prediction changes when transformations are applied to its input. While most certification frameworks study geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.

Related papers

• Semantic Browsing: Controllable Diversity for Image Generation [51.503726779537]
  We introduce a method for controlled diversity that enables Semantic Browsing.<n>We exploit the fact that recent text-to-image models are trained on elaborated captions.<n>This enables a paradigm shift: instead of relying on variation within the text-to-image model, we induce diversity directly at the text level.
  arXiv  Detail & Related papers  (2026-06-22T17:59:17Z)
• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Topology-Aware Layer Pruning for Large Vision-Language Models [21.06771347736129]
  Large Vision-Language Models (LVLMs) incur substantial computational and memory costs.<n>Existing layer pruning methods typically rely on local similarity metrics or static proxy signals.<n>We propose a topology-aware layer pruning framework for LVLMs.
  arXiv  Detail & Related papers  (2026-04-14T14:36:53Z)
• Unified modality separation: A vision-language framework for unsupervised domain adaptation [60.8391821117794]
  Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains.<n>We propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components.<n>Our methods achieve up to 9% performance gain with 9 times of computational efficiencies.
  arXiv  Detail & Related papers  (2025-08-07T02:51:10Z)
• An Empirical Study of Federated Prompt Learning for Vision Language Model [89.2963764404892]
  This paper systematically investigates the behavioral differences between language prompt learning (VPT) and vision prompt learning (VLM)<n>We evaluate the impact of various FL and prompt configurations, such as client scale, aggregation strategies, and prompt length, to assess the robustness of Federated Prompt Learning (FPL)
  arXiv  Detail & Related papers  (2025-05-29T03:09:15Z)
• Analyzing Finetuning Representation Shift for Multimodal LLMs Steering [56.710375516257876]
  We propose to map hidden states to interpretable visual and textual concepts.<n>This enables us to more efficiently compare certain semantic dynamics, such as the shift from an original and fine-tuned model.<n>We also demonstrate the use of shift vectors to capture these concepts changes.
  arXiv  Detail & Related papers  (2025-01-06T13:37:13Z)
• Semantic-Aware Implicit Template Learning via Part Deformation Consistency [18.63665468429503]
  We propose a semantic-aware implicit template learning framework to enable semantically plausible deformation. By leveraging semantic prior from a self-supervised feature extractor, we suggest local conditioning with novel semantic-aware deformation code. Our experiments demonstrate the superiority of the proposed method over baselines in various tasks.
  arXiv  Detail & Related papers  (2023-08-23T05:02:17Z)
• Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning [74.48337375174297]
  Generalized Zero-Shot Learning (GZSL) identifies unseen categories by knowledge transferred from the seen domain. We deploy the dual semantic-visual transformer module (DSVTM) to progressively model the correspondences between prototypes and visual features. DSVTM devises an instance-motivated semantic encoder that learns instance-centric prototypes to adapt to different images, enabling the recast of the unmatched semantic-visual pair into the matched one.
  arXiv  Detail & Related papers  (2023-03-27T15:21:43Z)
• VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding [5.1282202633907]
  We propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.
  arXiv  Detail & Related papers  (2022-11-01T12:51:51Z)
• Latent Covariate Shift: Unlocking Partial Identifiability for Multi-Source Domain Adaptation [82.14087963690561]
  Multi-source domain adaptation (MSDA) addresses the challenge of learning a label prediction function for an unlabeled target domain.<n>We present an intricate causal generative model by introducing latent noises across domains, along with a latent content variable and a latent style variable.<n>The proposed approach showcases exceptional performance and efficacy on both simulated and real-world datasets.
  arXiv  Detail & Related papers  (2022-08-30T11:25:15Z)
• GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing [40.38555458216436]
  We propose a unified theoretical framework for certifying robustness against general semantic transformations. Under the GSmooth framework, we present a scalable algorithm that uses a surrogate image-to-image network to approximate the complex transformation.
  arXiv  Detail & Related papers  (2022-06-09T07:12:17Z)
• Point Cloud Learning with Transformer [2.3204178451683264]
  We introduce a novel framework, called Multi-level Multi-scale Point Transformer (MLMSPT) Specifically, a point pyramid transformer is investigated to model features with diverse resolutions or scales. A multi-level transformer module is designed to aggregate contextual information from different levels of each scale and enhance their interactions.
  arXiv  Detail & Related papers  (2021-04-28T08:39:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.