From Local to Global to Mechanistic: An iERF-Centered Unified Framework for Interpreting Vision Models

• URL: http://arxiv.org/abs/2605.00474v1
• Date: Fri, 01 May 2026 07:25:49 GMT
• Title: From Local to Global to Mechanistic: An iERF-Centered Unified Framework for Interpreting Vision Models
• Abstract summary: We introduce an iERF-centric framework that unifies local, global, and mechanistic interpretability around a single analysis unit.<n>On the local side, Sharing Ratio Decomposition (SRD) expresses each PFV as a mixture of upstream PFVs via sharing ratios and propagates iERFs to construct class-discriminative saliency maps.<n>For the global view, we introduce Concept-Anchored Feature Explanation (CAFE), which utilizes the iERF as a semantic label.
• Score: 29.908678230814015
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern vision models achieve remarkable accuracy, but explaining where evidence arises, what the model encodes, and how internal computations assemble that evidence remains fragmented. We introduce an iERF-centric framework that unifies local, global, and mechanistic interpretability around a single analysis unit: the pointwise feature vector (PFV) paired with its instance-specific Effective Receptive Field (iERF). On the local side, Sharing Ratio Decomposition (SRD) expresses each PFV as a mixture of upstream PFVs via sharing ratios and propagates iERFs to construct class-discriminative saliency maps. SRD yields high-resolution, activation-faithful explanations, is robust to targeted manipulation and noise, and remains activation-agnostic across common nonlinearities. For the global view, we introduce Concept-Anchored Feature Explanation (CAFE), which utilizes the iERF as a semantic label, grounding abstract latent vectors in verifiable pixel-level evidence. With CAFE, we address the challenge of non-localized sparse autoencoder latents--especially in Transformers, where early self-attention mixes distant context. To answer how representations are composed through depth, we propose the Interlayer Concept Graph with Interlayer Concept Attribution (ICAT), which quantifies concept-to-concept influence while isolating layer pairs; an interlayer insertion, deletion protocol identifies Integrated Gradients as the most faithful instantiation. Empirically, across ResNet50, VGG16, and ViTs, our framework outperforms baselines in both fidelity and robustness, successfully interprets dispersed SAE features, and exposes dominant concept routes in correct, misclassified, and adversarial cases. Grounded in iERFs, our approach provides a coherent, evidence-backed map from pixels to concepts to decisions.

Related papers

• FedVAR: Prototype-Aligned Federated Framework for Video Anomaly Recognition [3.940235853574397]
  Federated Learning offers a promising decentralized intelligence paradigm for Video Anomaly Recognition.<n>In this paper, we introduce Fedvar, a weakly-supervised FL framework explicitly designed for Video Anomaly Recognition.<n>We conduct extensive experiments on challenging benchmarks under various non-IID data partitioning schemes, unseen domains, and novel anomaly classes.<n>The results demonstrate that Fedvar consistently outperforms state-of-the-art federated baselines, establishing a robust framework for distributed intelligence in video-based CPS.
  arXiv  Detail & Related papers  (2026-08-07T07:01:42Z)
• Topology-Aware Neighborhood Learning for Source-Free Cross-Scene Hyperspectral Image Classification [11.354716471312239]
  Domain adaptation has advanced cross-scene hyperspectral image classification, significantly improving discriminative capability in complex scenarios.<n>Conventional domain adaptation methods become impractical, severely restricting their utility in realistic remote sensing scenarios.<n>We propose a topology-aware source-free learning framework to achieve robust and accurate classification without source data.
  arXiv  Detail & Related papers  (2026-08-06T12:40:06Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method [59.30562121800656]
  Open-Vocabulary Object Detection has achieved remarkable success in generalizing to novel categories.<n>We provide a principled revisit of the OVOD paradigm, uncovering a fundamental vulnerability.<n>We propose Progressive Domain-invariant Cross-modal Alignment (PICA)
  arXiv  Detail & Related papers  (2026-03-29T07:39:31Z)
• MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation [1.1470070927586016]
  Contextualization layer generalizes standard multi-head attention.<n>ManAR Conceptual addresses by synthesizing representations that provably lie outside input tokens.
  arXiv  Detail & Related papers  (2026-03-19T09:37:18Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• UAGLNet: Uncertainty-Aggregated Global-Local Fusion Network with Cooperative CNN-Transformer for Building Extraction [83.48950950780554]
  Building extraction from remote sensing images is a challenging task due to the complex structure variations of buildings.<n>Existing methods employ convolutional or self-attention blocks to capture the multi-scale features in the segmentation models.<n>We present an Uncertainty-Aggregated Global-Local Fusion Network (UAGLNet) to exploit high-quality global-local visual semantics.
  arXiv  Detail & Related papers  (2025-12-15T02:59:16Z)
• Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm [17.63632082331749]
  Large-scale visual foundation models (VFMs) exhibit strong generalization across diverse visual domains, but their potential for single-frame infrared small target (SIRST) detection remains largely unexplored.<n>We propose a Foundation-Driven Efficient Paradigm (FDEP) which can seamlessly adapt to existing encoder-decoder-based methods and significantly improve accuracy without additional inference overhead.
  arXiv  Detail & Related papers  (2025-12-05T08:12:35Z)
• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation [27.62798534410331]
  We introduce Fine-grained Cross-modal Causal Tracing (FCCT) framework, which systematically quantifies the causal effects on visual object perception.<n>FCCT conducts fine-grained analysis covering the full range of visual and textual tokens, three core model components including multi-head self-attention (MHSA), feed-forward networks (FFNs), and hidden states, across all decoder layers.<n>Our analysis is the first to demonstrate that MHSAs of the last token in middle layers play a critical role in aggregating cross-modal information, while FFNs exhibit a three-stage hierarchical progression for the storage
  arXiv  Detail & Related papers  (2025-11-08T08:37:26Z)
• OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triad [22.358084538002327]
  We propose a novel framework, the Object-Concept-Relation Triad (OCRT), that enables FMs to extract sparse, high-level concepts and intricate relational structures from raw visual inputs.<n>To be specific, we project the object-centric representations onto a semantic concept space that the model can readily interpret and estimate their importance to filter out irrelevant elements.<n>To be specific, OCRT can substantially boost the generalizability and robustness of SAM and CLIP across multiple downstream tasks.
  arXiv  Detail & Related papers  (2025-03-24T14:04:17Z)
• You Only Train Once: A Unified Framework for Both Full-Reference and No-Reference Image Quality Assessment [45.62136459502005]
  We propose a network to perform full reference (FR) and no reference (NR) IQA. We first employ an encoder to extract multi-level features from input images. A Hierarchical Attention (HA) module is proposed as a universal adapter for both FR and NR inputs. A Semantic Distortion Aware (SDA) module is proposed to examine feature correlations between shallow and deep layers of the encoder.
  arXiv  Detail & Related papers  (2023-10-14T11:03:04Z)
• Centralized Feature Pyramid for Object Detection [53.501796194901964]
  Visual feature pyramid has shown its superiority in both effectiveness and efficiency in a wide range of applications. In this paper, we propose a OLO Feature Pyramid for object detection, which is based on a globally explicit centralized feature regulation.
  arXiv  Detail & Related papers  (2022-10-05T08:32:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.