Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

• URL: http://arxiv.org/abs/2604.12119v1
• Date: Mon, 13 Apr 2026 22:53:39 GMT
• Title: Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models
• Abstract summary: Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures.<n>We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping.<n>Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap.
• Score: 1.7767466724342065
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap. Prompt interventions support this mechanism: neutral alias prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and dataset available at https://maveryn.github.io/vlm-fix/.

Related papers

• Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detection [7.184018933441995]
  Vision-language models offer a promising approach for zero-shot anomaly detection (ZSAD)<n>Normal and anomalous text prototypes exhibit a high semantic overlap.<n>We propose Proximity-CLIP, a framework that visually calibrates the semantic margin to guide visual adaptation.
  arXiv  Detail & Related papers  (2026-09-07T08:48:27Z)
• Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations [42.90321348046055]
  We propose an annotation-free attribution regularization framework based on submodular search over image regions.<n>By measuring how candidate subsets affect model outputs, the search extracts compact, class-discriminative evidence as search-derived supervision.<n> Experiments on ImageNet-100 show that our method substantially improves attribution stability, Insertion, and Deletion on ViT-B/16 with only a 0.28-point accuracy drop.
  arXiv  Detail & Related papers  (2026-07-26T20:41:38Z)
• FlowBender: Feedback-Aware Training for Self-Correcting Conditional Flows [18.746098727212544]
  FlowBender is a new diffusion and flow model for 3D reconstruction.<n>We show that FlowBender consistently outperforms standard supervised alignment training.<n>We propose several variants of FlowBender for differentiable operators.
  arXiv  Detail & Related papers  (2026-06-18T15:56:07Z)
• The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment [53.510840985295154]
  We show that finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains.<n>We propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries.<n>We propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM.
  arXiv  Detail & Related papers  (2026-06-04T19:32:00Z)
• Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shift [17.611062308867275]
  We show that distribution shift rotates the subspace that the model actively uses, misaligning the explainer's dictionary trained on in-distribution (ID) activations.<n>We formalize this misalignment as the faithfulness gap, a geometric distance between the ID dictionary and the OOD-active subspace.<n>We propose the Geometry-Adaptive Explainer (GAE), which realigns the explainer's dictionary with the OOD-active subspace while preserving the original feature structure.
  arXiv  Detail & Related papers  (2026-05-21T00:46:01Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models [15.955490895662384]
  We introduce LIBERO-Para, a benchmark for fine-grained analysis of linguistic generalization.<n>We observe consistent performance degradation of 22-52 pp under paraphrasing.<n>We propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors.
  arXiv  Detail & Related papers  (2026-03-30T11:27:34Z)
• VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer [18.348454274148185]
  Zero-shot anomaly detection (ZSAD) requires detecting and localizing anomalies without access to target-class anomaly samples.<n>This work revisits the necessity of the text branch in ZSAD and presents VisualAD, a purely visual framework built on Vision Transformers.<n>VisualAD achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains.
  arXiv  Detail & Related papers  (2026-03-09T04:33:56Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics [25.374192139098284]
  We study prototypicality bias as a systematic failure mode in multimodal evaluation.<n>We introduce a controlled contrastive benchmark ProtoBias, spanning Animals, Objects, and Demography images.<n>Our results show that widely used metrics, including CLIPScore, PickScore, and VQA-based scores, frequently misrank these pairs.<n>We propose ProtoScore, a robust 7B- parameter metric that substantially reduces failure rates and suppresses misranking.
  arXiv  Detail & Related papers  (2026-01-08T13:49:14Z)
• Explaining Time Series Classifiers with PHAR: Rule Extraction and Fusion from Post-hoc Attributions [7.51289645756884]
  PHAR is a framework that transforms numeric feature attributions into structured, human-readable rules.<n>A dedicated rule fusion step consolidates rule sets using strategies like weighted selection and lasso-based refinement.<n>Experiments on UCR/UEA Time Series Classification Archive demonstrate that PHAR may improve interpretability, decision transparency, and practical applicability for TS classification tasks.
  arXiv  Detail & Related papers  (2025-08-03T09:45:40Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• RESTORE: Towards Feature Shift for Vision-Language Prompt Learning [33.13407089704543]
  We show that prompt tuning along only one branch of CLIP is the reason why the misalignment occurs. Without proper regularization across the learnable parameters in different modalities, prompt learning violates the original pre-training constraints. We propose RESTORE, a multi-modal prompt learning method that exerts explicit constraints on cross-modal consistency.
  arXiv  Detail & Related papers  (2024-03-10T08:52:48Z)
• Contrastive Instruction Tuning [61.97704869248903]
  We propose Contrastive Instruction Tuning to maximize the similarity between semantically equivalent instruction-instance pairs. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy.
  arXiv  Detail & Related papers  (2024-02-17T00:09:32Z)
• ADEPT: A DEbiasing PrompT Framework [64.54665501064659]
  Finetuning is an applicable approach for debiasing contextualized word embeddings.<n> discrete prompts with semantic meanings have shown to be effective in debiasing tasks.<n>We propose ADEPT, a method to debias PLMs using prompt tuning while maintaining the delicate balance between removing biases and ensuring representation ability.
  arXiv  Detail & Related papers  (2022-11-10T08:41:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.