Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations
Abstract Overview
This paper proposes an annotation-free attribution regularization framework for image recognition that aims to enforce consistent evidence use under label-preserving geometric transformations. Instead of aligning gradient-based saliency maps, the method derives supervision from a submodular search over image regions that measures how candidate subsets affect model outputs, producing compact and class-discriminative evidence sequences. To make this discrete search process trainable, the authors introduce a Submodular Ranking Loss with path-consistency and terminal stopping components that align the transformed image's evidence-selection trajectory with the original query sequence. The training pipeline further filters for correctly classified, high-confidence, compact attributions so that the regularization signal is tied to more reliable model behavior.
Novelty
The main novelty is using search-based, decision-linked attribution as self-supervision for attribution regularization, rather than enforcing consistency on gradient-derived heatmaps whose faithfulness may be limited. A second distinctive contribution is the differentiable Submodular Ranking Loss, which regularizes not only the final attribution set but also the intermediate region-selection path and stopping criterion under geometric transformations.
Results
On ImageNet-100, the method improves attribution stability on ViT-B/16 from 0.14 to 0.27, raises Insertion from 0.4496 to 0.6840, and lowers Deletion from 0.1363 to 0.0515, with only a 0.28-percentage-point accuracy drop; similar attribution gains are achieved with ViT-L/16. On ImageNet-1K, it improves mean transformed-input accuracy by 0.55 points on ResNet-50 and 0.33 points on ConvNeXt-B while limiting clean-accuracy drops to 0.12 and 0.30 points, respectively. The ViT-B experiments also show competitive or best performance across in-distribution, corruption, ObjectNet, and several OOD benchmarks.
Key Points
- The framework builds attribution supervision from submodular region search that explicitly probes model output changes, aiming for more faithful evidence than gradient-based saliency alignment.
- The proposed Submodular Ranking Loss aligns spatially corresponding region-selection trajectories and enforces matched stopping behavior after geometric transformations.
- Experiments indicate substantial improvements in attribution stability, faithfulness, and compactness, alongside stronger robustness and OOD performance with only minor clean-accuracy trade-offs.