AIM: Amending Inherent Interpretability via Self-Supervised Masking

• URL: http://arxiv.org/abs/2508.11502v1
• Date: Fri, 15 Aug 2025 14:29:59 GMT
• Title: AIM: Amending Inherent Interpretability via Self-Supervised Masking
• Abstract summary: We propose "Amending Inherent Interpretability via Self-Supervised Masking" (AIM)<n>AIM promotes the network's utilization of genuine features over spurious alternatives without requiring additional annotations.<n>We validate AIM across a range of challenging datasets that test both out-of-distribution generalization and fine-grained visual understanding.
• Score: 57.17600766859953
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It has been observed that deep neural networks (DNNs) often use both genuine as well as spurious features. In this work, we propose "Amending Inherent Interpretability via Self-Supervised Masking" (AIM), a simple yet interestingly effective method that promotes the network's utilization of genuine features over spurious alternatives without requiring additional annotations. In particular, AIM uses features at multiple encoding stages to guide a self-supervised, sample-specific feature-masking process. As a result, AIM enables the training of well-performing and inherently interpretable models that faithfully summarize the decision process. We validate AIM across a diverse range of challenging datasets that test both out-of-distribution generalization and fine-grained visual understanding. These include general-purpose classification benchmarks such as ImageNet100, HardImageNet, and ImageWoof, as well as fine-grained classification datasets such as Waterbirds, TravelingBirds, and CUB-200. AIM demonstrates significant dual benefits: interpretability improvements, as measured by the Energy Pointing Game (EPG) score, and accuracy gains over strong baselines. These consistent gains across domains and architectures provide compelling evidence that AIM promotes the use of genuine and meaningful features that directly contribute to improved generalization and human-aligned interpretability.

Related papers

• AIM: Adversarial Information Masking for Faithfulness Evaluation of Saliency Maps [0.9940728137241214]
  Post-hoc saliency methods are widely used to interpret deep neural networks.<n> Existing evaluations mask features according to saliency-induced feature ordering and measure performance degradation.<n>We propose Adrial Information Masking (AIM), a saliency-guided adversarial feature replacement framework for evaluating both saliency-map faithfulness and masking-operator reliability.
  arXiv  Detail & Related papers  (2026-05-16T09:36:58Z)
• AIMing for Standardised Explainability Evaluation in GNNs: A Framework and Case Study on Graph Kernel Networks [7.535219325248995]
  Graph Neural Networks (GNNs) have advanced significantly in handling graph-structured data, but a comprehensive framework for evaluating explainability remains lacking.<n>We introduce AIM, a comprehensive framework that addresses these limitations by measuring Accuracy, Instance-level explanations, and Model-level explanations.<n>We show how the insights obtained from AIM can be used to develop an updated model, xGKN, that maintains high accuracy while demonstrating improved explainability.
  arXiv  Detail & Related papers  (2026-05-14T14:28:24Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• AIM-Bench: Benchmarking and Improving Affective Image Manipulation via Fine-Grained Hierarchical Control [30.172679310027533]
  Affective Image Manipulation (AIM) aims to evoke specific emotions through targeted editing.<n>We introduce the first benchmark designed for AIM termed AIM-Bench.<n>This benchmark is built upon a dual-path affective modeling scheme that integrates the Mikels emotion taxonomy with the Valence-Arousal-Dominance framework.
  arXiv  Detail & Related papers  (2026-04-12T04:32:30Z)
• Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detection [59.04089915447622]
  ForenAgent is an interactive IFD framework that enables MLLMs to autonomously generate, execute, and refine Python-based low-level tools around the detection objective.<n>Inspired by human reasoning, we design a dynamic reasoning loop comprising global perception, local focusing, iterative probing, and holistic adjudication.<n>Experiments show that ForenAgent exhibits emergent tool-use competence and reflective reasoning on challenging IFD tasks.
  arXiv  Detail & Related papers  (2025-12-18T08:38:44Z)
• From Attribution to Action: Jointly ALIGNing Predictions and Explanations [7.1383591932321115]
  We propose ALIGN, a novel framework that jointly trains a classifier and a masker in an iterative manner.<n>By leveraging high-quality masks as guidance, ALIGN improves both interpretability and generalizability, showing its superiority across various settings.
  arXiv  Detail & Related papers  (2025-11-10T10:52:17Z)
• HAMLET-FFD: Hierarchical Adaptive Multi-modal Learning Embeddings Transformation for Face Forgery Detection [6.060036926093259]
  HAMLET-FFD is a cross-domain generalization framework for face forgery detection.<n>It integrates visual evidence with conceptual cues, emulating expert forensic analysis.<n>By design, HAMLET-FFD freezes all pretrained parameters, serving as an external plugin.
  arXiv  Detail & Related papers  (2025-07-28T15:09:52Z)
• ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion [25.1725138364452]
  We introduce an Attention-based Diffusion model for Missing Modalities feature Completion (ADMC)<n>Our framework independently trains feature extraction networks for each modality, preserving their unique characteristics and avoiding over-coupling.<n>Our approach achieves state-of-the-art results on the IEMOCAP and MIntRec benchmarks, demonstrating its effectiveness in both missing and complete modality scenarios.
  arXiv  Detail & Related papers  (2025-07-08T03:08:52Z)
• LSM-2: Learning from Incomplete Wearable Sensor Data [65.58595667477505]
  This paper introduces the second generation of Large Sensor Model (LSM-2) with Adaptive and Inherited Masking (AIM)<n>AIM learns robust representations directly from incomplete data without requiring explicit imputation.<n>Our LSM-2 with AIM achieves the best performance across a diverse range of tasks, including classification, regression and generative modeling.
  arXiv  Detail & Related papers  (2025-06-05T17:57:11Z)
• MetaGMT: Improving Actionable Interpretability of Graph Multilinear Networks via Meta-Learning Filtration [6.102559098873098]
  We present MetaGMT, a meta-learning framework that enhances explanation fidelity through a novel bi-level optimization approach.<n>We demonstrate that MetaGMT significantly improves both explanation quality (AUC-ROC, Precision@K) and robustness to spurious patterns.<n>Our work contributes to building more trustworthy and actionable GNN systems for real-world applications.
  arXiv  Detail & Related papers  (2025-05-26T03:07:58Z)
• Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learning [2.713322720372114]
  Current approaches in Explainable Deep Reinforcement Learning have limitations in which the attention mask has a displacement with the objects in visual input.<n>We propose the Interpretable Feature Extractor architecture, aimed at generating an accurate attention mask to illustrate both "what" and "where" the agent concentrates on in the spatial domain.<n>The resulting attention mask is consistent, highly understandable by humans, accurate in spatial dimension, and effectively highlights important objects or locations in visual input.
  arXiv  Detail & Related papers  (2025-04-14T10:18:34Z)
• Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond [52.486290612938895]
  We propose a novel method that leverages the semantic knowledge from the Segment Anything Model (SAM) to Grow the quality of fusion results and Enable downstream task adaptability.<n> Specifically, we design a Semantic Persistent Attention (SPA) Module that efficiently maintains source information via the persistent repository while extracting high-level semantic priors from SAM.<n>Our method achieves a balance between high-quality visual results and downstream task adaptability while maintaining practical deployment efficiency.
  arXiv  Detail & Related papers  (2025-03-03T06:16:31Z)
• Self-supervised Gait-based Emotion Representation Learning from Selective Strongly Augmented Skeleton Sequences [4.740624855896404]
  We propose a contrastive learning framework utilizing selective strong augmentation for self-supervised gait-based emotion representation. Our approach is validated on the Emotion-Gait (E-Gait) and Emilya datasets and outperforms the state-of-the-art methods under different evaluation protocols.
  arXiv  Detail & Related papers  (2024-05-08T09:13:10Z)
• Manipulating Feature Visualizations with Gradient Slingshots [53.94925202421929]
  Feature Visualization (FV) is a widely used technique for interpreting the concepts learned by Deep Neural Networks (DNNs)<n>We introduce a novel method, Gradient Slingshots, that enables manipulation of FV without modifying the model architecture or significantly degrading its performance.
  arXiv  Detail & Related papers  (2024-01-11T18:57:17Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.