Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations

• URL: http://arxiv.org/abs/2605.09090v1
• Date: Sat, 09 May 2026 17:54:31 GMT
• Title: Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations
• Abstract summary: Grounding models are rarely evaluated under semantically mismatched captions.<n>This work examines whether embedding anisotropy contributes to counterfactual failures.
• Score: 1.423280626666929
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual Grounding benchmarks assume that the object described by a referring expression is always present in the image, and grounding models are therefore rarely evaluated under semantically mismatched captions. In such cases, models frequently exhibit approximation behavior, producing a plausible bounding box that satisfies only part of the expression (\eg, preserving the original object while ignoring modified contextual cues). Because mismatched captions represent realistic edge cases, this behavior compromises reliability and raises concerns from an explainability perspective. Identifying its underlying causes is thus essential for improving model faithfulness and interpretability. Adopting a mechanistic interpretability viewpoint, this work examines whether embedding anisotropy contributes to counterfactual failures. A similarity-controlled counterfactual caption generation protocol is introduced to systematically perturb object or contextual components within predefined embedding similarity intervals, enabling a fine-grained analysis of grounding behavior as a function of alignment. Experiments on two Transformer-based models with markedly different embedding geometries (BERT-based TransVG and CLIP-based SwimVG) reveal no meaningful correlation between cosine similarity and approximation. These findings suggest that anisotropy alone does not account for counterfactual errors, and that robustness requires investigating finer-grained geometric properties of the embedding space.

Related papers

• Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detection [7.184018933441995]
  Vision-language models offer a promising approach for zero-shot anomaly detection (ZSAD)<n>Normal and anomalous text prototypes exhibit a high semantic overlap.<n>We propose Proximity-CLIP, a framework that visually calibrates the semantic margin to guide visual adaptation.
  arXiv  Detail & Related papers  (2026-09-07T08:48:27Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Geo-ID: Test-Time Geometric Consensus for Cross-View Consistent Intrinsics [37.614964138575935]
  Intrinsic image decomposition aims to estimate physically based rendering parameters such as albedo, roughness, and metallicity from images.<n>Video-based models can improve cross-frame consistency but require dense, ordered sequences and substantial compute.<n>We propose Geo-ID, a novel test-time framework that repurposes pretrained single-view predictors intrinsic to produce cross-view consistent decompositions.
  arXiv  Detail & Related papers  (2026-03-14T09:36:27Z)
• OrthoFormer: Instrumental Variable Estimation in Transformer Hidden States via Neural Control Functions [0.0]
  Transformer architectures excel at sequential modeling yet remain fundamentally limited by correlational learning.<n>We propose OrthoFormer, a causally grounded architecture that embeds instrumental variable estimation directly into Transformer blocks via neural control functions.
  arXiv  Detail & Related papers  (2026-03-08T03:05:16Z)
• On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformation [51.56484100374058]
  We formalise such information-conditioned interaction patterns as behavioural dependency.<n>This induces a probe-relative notion of $$-behavioural equivalence and a within-policy behavioural distance.<n>Results identify structural conditions under which probe-conditioned behavioural separation is not preserved under common policy transformations.
  arXiv  Detail & Related papers  (2026-02-24T22:55:21Z)
• Simulated Adoption: Decoupling Magnitude and Direction in LLM In-Context Conflict Resolution [3.0242762196828448]
  Large Language Models (LLMs) frequently prioritize conflicting in-context information over pre-existing parametric memory.<n>We show that models do not "unlearn" or suppress the magnitude of internal truths but rather employ a mechanism of geometric displacement.
  arXiv  Detail & Related papers  (2026-02-04T06:13:11Z)
• Unlearning-based Neural Interpretations [51.99182464831169]
  We show that current baselines defined using static functions are biased, fragile and manipulable.<n>We propose UNI to compute an (un)learnable, debiased and adaptive baseline by perturbing the input towards an unlearning direction of steepest ascent.
  arXiv  Detail & Related papers  (2024-10-10T16:02:39Z)
• Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models [44.56781176879151]
  State-of-the-art Natural Language Inference (NLI) models are sensitive towards minor semantics preserving surface-form variations. We show that semantic sensitivity causes performance degradations of $12.92%$ and $23.71%$ average over $textbfin-$ and $textbfout-of-$ domain settings.
  arXiv  Detail & Related papers  (2024-01-25T14:47:05Z)
• An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation [83.93224401261068]
  Source model is motivated by the recent surge of interest in the semantic aspect of information. intrinsic state corresponds to the semantic feature of the source, which in general is not observable. Rate-distortion function is the semantic rate-distortion function of the source.
  arXiv  Detail & Related papers  (2022-01-29T02:14:24Z)
• Where and What? Examining Interpretable Disentangled Representations [96.32813624341833]
  Capturing interpretable variations has long been one of the goals in disentanglement learning. Unlike the independence assumption, interpretability has rarely been exploited to encourage disentanglement in the unsupervised setting. In this paper, we examine the interpretability of disentangled representations by investigating two questions: where to be interpreted and what to be interpreted.
  arXiv  Detail & Related papers  (2021-04-07T11:22:02Z)
• Structural Causal Models Are (Solvable by) Credal Networks [70.45873402967297]
  Causal inferences can be obtained by standard algorithms for the updating of credal nets. This contribution should be regarded as a systematic approach to represent structural causal models by credal networks. Experiments show that approximate algorithms for credal networks can immediately be used to do causal inference in real-size problems.
  arXiv  Detail & Related papers  (2020-08-02T11:19:36Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.