Rethinking Feature Reliance Evaluation with Semantically Matched Suppression

• URL: http://arxiv.org/abs/2607.16298v1
• Date: Mon, 13 Jul 2026 13:02:07 GMT
• Title: Rethinking Feature Reliance Evaluation with Semantically Matched Suppression
• Abstract summary: ImageNet-trained CNNs show stronger degradation under texture suppression than under shape suppression.<n>Vision Transformers retain higher accuracy than CNNs under both shape and texture suppression.
• Score: 5.842411095887196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding whether visual recognition models rely on shape, texture, or color is central to interpreting their behavior. Prior cue-conflict studies have strongly influenced the view that CNNs are texture-biased, yet such tests measure cue preference under artificial conflicts rather than feature reliance during natural recognition. We revisit this question through controlled feature suppression and show that performance drops are difficult to interpret unless different suppression operations impose comparable category-level damage. We introduce a semantically matched evaluation framework that compares shape and texture suppression at matched levels of category separability loss. Under this framework, ImageNet-trained CNNs show stronger degradation under texture suppression than under shape suppression, revealing greater texture reliance than suggested by unmatched suppression analyses. Extending the comparison across architectures, we find that Vision Transformers retain higher accuracy than CNNs under both shape and texture suppression. Brain encoding further shows that ViT representations exhibit smaller suppression-induced decreases in neural prediction performance under the tested suppression settings. These findings indicate that semantic comparability is essential for interpreting feature reliance from suppression experiments, and suggest that the robustness advantage of ViTs may be related to representations more compatible with human visual cortex.

Related papers

• SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation [52.62632888060867]
  Visible-to-infrared image translation provides a practical way to expand infrared training data using abundant visible images.<n>Existing diffusion-based methods typically use semantic priors only as external conditions.<n>We propose SC-Diff, a semantically latent diffusion framework that uses semantic priors for both conditional guidance and internal self-attention calibration.
  arXiv  Detail & Related papers  (2026-08-09T07:57:40Z)
• Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? [10.315515647818009]
  Benchmark accuracy is often assumed to reflect grounded visual understanding in vision-language models.<n>We systematically investigate this mismatch in a set of open-source vision-language models.<n>We find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence.
  arXiv  Detail & Related papers  (2026-05-21T17:35:04Z)
• A Causally Grounded Taxonomy for Image Degradation Robustness Evaluation [3.8265249634979734]
  We propose a causally grounded framework for organizing and interpreting image degradations.<n>Instead of introducing new degradations or redefining existing benchmarks, we provide an interpretive representation and measurement layer.<n>We demonstrate the framework through COCO Degradation, a taxonomy aligned benchmark for evaluating object detector robustness under diverse imaging conditions.
  arXiv  Detail & Related papers  (2026-05-15T12:44:02Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression [10.20881592893918]
  We revisit the hypothesis that Convolutional Neural Networks (CNNs) are inherently texture-biased.<n>We propose a domain-agnostic framework that quantifies feature reliance through systematic suppression of shape, texture, and color cues.<n>We extend the analysis across computer vision, medical imaging, and remote sensing, revealing that reliance patterns differ systematically.
  arXiv  Detail & Related papers  (2025-09-24T15:24:43Z)
• Do Sparse Subnetworks Exhibit Cognitively Aligned Attention? Effects of Pruning on Saliency Map Fidelity, Sparsity, and Concept Coherence [2.7415651415305597]
  We investigate how magnitude-based pruning followed by fine-tuning affects both low-level saliency maps and high-level concept representations.<n>Our results show that light-to-moderate pruning improves saliency-map focus and faithfulness while retaining distinct, semantically meaningful concepts.<n>In contrast, aggressive pruning merges heterogeneous features, reducing saliency map sparsity and concept coherence despite maintaining accuracy.
  arXiv  Detail & Related papers  (2025-09-23T20:10:23Z)
• Improving Network Interpretability via Explanation Consistency Evaluation [56.14036428778861]
  We propose a framework that acquires more explainable activation heatmaps and simultaneously increase the model performance. Specifically, our framework introduces a new metric, i.e., explanation consistency, to reweight the training samples adaptively in model learning. Our framework then promotes the model learning by paying closer attention to those training samples with a high difference in explanations.
  arXiv  Detail & Related papers  (2024-08-08T17:20:08Z)
• Semantic Ensemble Loss and Latent Refinement for High-Fidelity Neural Image Compression [58.618625678054826]
  This study presents an enhanced neural compression method designed for optimal visual fidelity. We have trained our model with a sophisticated semantic ensemble loss, integrating Charbonnier loss, perceptual loss, style loss, and a non-binary adversarial loss. Our empirical findings demonstrate that this approach significantly improves the statistical fidelity of neural image compression.
  arXiv  Detail & Related papers  (2024-01-25T08:11:27Z)
• Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks [3.3504365823045044]
  We show that adversarially robust representations capture peripheral computation better than non-robust representations. Our findings support the idea that localized texture summary statistic representations may drive human in robustness to adversarials.
  arXiv  Detail & Related papers  (2022-02-02T01:19:40Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Stereopagnosia: Fooling Stereo Networks with Adversarial Perturbations [71.00754846434744]
  We show that imperceptible additive perturbations can significantly alter the disparity map. We show that, when used for adversarial data augmentation, our perturbations result in trained models that are more robust.
  arXiv  Detail & Related papers  (2020-09-21T19:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.