Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification

• URL: http://arxiv.org/abs/2605.00326v1
• Date: Fri, 01 May 2026 01:06:30 GMT
• Title: Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification
• Abstract summary: Cross-prompt variance is strongly associated with prompt-level disagreement and higher error.<n>A training-free mean ensemble improves NLL on all 14 dataset-model evaluation pairs.<n>We frame this as a reliability stress test for zero-shot VLM first-token safety scores.
• Score: 43.86209535250233
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Single-prompt first-token probabilities from zero-shot vision-language model (VLM) safety classifiers are treated as decision scores, but we show they are unreliable under semantically equivalent prompt reformulation: even when the binary label is constrained to a fixed output position, equivalent prompts can induce materially different unsafe probabilities for the same sample. Across multimodal safety benchmarks and multiple VLM families, cross-prompt variance is strongly associated with prompt-level disagreement and higher error, making it a useful fragility diagnostic. A training-free mean ensemble improves NLL on all 14 dataset-model evaluation pairs and ECE on 12/14 relative to a train-selected single-prompt baseline, and wins more head-to-head NLL comparisons than labeled temperature scaling, Platt scaling, and isotonic regression applied to the same prompt. Ranking gains are consistent against the train-selected baseline on both AUROC and AUPRC, and against the full 15-prompt distribution remain consistent on AUPRC while softening on AUROC. Labeled calibration on top of the mean provides further gains when labels are available, identifying prompt averaging as a strong label-free first stage rather than a replacement for calibration. We frame this as a reliability stress test for zero-shot VLM first-token safety scores and recommend prompt-family evaluation with mean aggregation as a standard label-free reliability baseline.

Related papers

• Symmetry-Aware Likelihood-Orbit Aggregation for Selective Left-Right Claim Verification [14.108900142001628]
  We introduce Relation-Orbit, a closed-form contrast that assigns eight normalized likelihoods to query-supporting and counterfactual roles.<n>On VSR and GQA across four frozen vision-language models, Relation-Orbit yields higher mean test coverage at a 10% selective-risk calibration target.
  arXiv  Detail & Related papers  (2026-09-15T11:19:03Z)
• Multi-Label Test-Time Adaptation with Bayesian Conditional Priors [15.878958356965159]
  Conditional Priors Estimation (BCP) is a gradient-free test-time adaptation method that injects label dependency without tuning the backbone.<n>BCP operates without target annotations by estimating anchor-conditioned priors online from the unlabeled test stream.<n>BCP consistently outperforms strong TTA baselines across standard multi-label benchmarks and multiple CLIP backbones.
  arXiv  Detail & Related papers  (2026-06-11T05:29:00Z)
• LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs [61.06744611795341]
  Medical vision-language models (VLMs) are strong zero-shot recognizers for medical imaging.<n>We propose texttttextbfLATA (Laplacian-Assisted Transductive Adaptation), a textittraining- and label-free refinement.<n>texttttextbfLATA sharpens zero-shot predictions without compromising exchangeability.
  arXiv  Detail & Related papers  (2026-02-19T16:45:38Z)
• SCOPE: Selective Conformal Optimized Pairwise LLM Judging [16.73004625486562]
  Large language models (LLMs) are increasingly used as judges to replace costly human preference labels in pairwise evaluation.<n>Despite their practicality, LLM judges remain prone to miscalibration and systematic biases.<n>This paper proposes SCOPE, a framework for selective pairwise judging with finite-sample statistical guarantees.
  arXiv  Detail & Related papers  (2026-02-13T17:10:43Z)
• Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning [53.9713678229744]
  Multi-instance partial-label learning (MIPL) is a weakly supervised framework that addresses the challenges of inexact supervision in both instance and label spaces.<n>Existing MIPL approaches often suffer from poor calibration, undermining reliability.<n>We propose a plug-and-play calibratable disambiguation loss (CDL) that simultaneously improves classification accuracy and calibration performance.
  arXiv  Detail & Related papers  (2025-12-19T16:58:31Z)
• Learning Robust Representations for Malicious Content Detection via Contrastive Sampling and Uncertainty Estimation [0.0]
  Uncertainty Contrastive Framework (UCF) integrates uncertainty-aware contrastive loss, adaptive temperature scaling, and a self-attention-guided LSTM encoder to improve classification under noisy and imbalanced conditions.<n>UCF dynamically adjusts contrastive weighting based on sample confidence, stabilizes training using positive anchors, and adapts temperature parameters to batch-level variability.
  arXiv  Detail & Related papers  (2025-12-01T22:06:06Z)
• Reliable Active Learning from Unreliable Labels via Neural Collapse Geometry [5.1511135538176]
  Active Learning (AL) promises to reduce annotation cost by prioritizing informative samples, yet its reliability is undermined when labels are noisy or when the data distribution shifts.<n>We propose Active Learning via Neural Collapse Geometry (NCAL-R), a framework that leverages the emergent geometric regularities of deep networks to counteract unreliable supervision.
  arXiv  Detail & Related papers  (2025-10-10T17:50:31Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.