DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification

• URL: http://arxiv.org/abs/2604.07166v1
• Date: Wed, 08 Apr 2026 14:55:36 GMT
• Title: DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification
• Abstract summary: This work proposes DINO-QPM, which converts powerful but entangled features into contrastive, class-independent representations.<n>DINO-QPM is a lightweight interpretability adapter that pursues globally interpretable image classification.
• Score: 26.47045755286244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although visual foundation models like DINOv2 provide state-of-the-art performance as feature extractors, their complex, high-dimensional representations create substantial hurdles for interpretability. This work proposes DINO-QPM, which converts these powerful but entangled features into contrastive, class-independent representations that are interpretable by humans. DINO-QPM is a lightweight interpretability adapter that pursues globally interpretable image classification, adapting the Quadratic Programming Enhanced Model (QPM) to operate on strictly frozen DINO backbones. While classification with visual foundation models typically relies on the \texttt{CLS} token, we deliberately diverge from this standard. By leveraging average-pooling, we directly connect the patch embeddings to the model's features and therefore enable spatial localisation of DINO-QPM's globally interpretable features within the input space. Furthermore, we apply a sparsity loss to minimise spatial scatter and background noise, ensuring that explanations are grounded in relevant object parts. With DINO-QPM we make the level of interpretability of QPM available as an adapter while exceeding the accuracy of DINOv2 linear probe. Evaluated through an introduced Plausibility metric and other interpretability metrics, extensive experiments demonstrate that DINO-QPM is superior to other applicable methods for frozen visual foundation models in both classification accuracy and explanation quality.

Related papers

• Beyond Weight Adaptation: Feature-Space Domain Injection for Cross-Modal Ship Re-Identification [3.6907522136316975]
  Cross-Modality Ship Re-Identification (CMS Re-ID) is critical for achieving all-day and all-weather maritime target tracking.<n>We explore the potential of Vision Foundation Models (VFMs) in bridging modality gaps.<n>We propose a novel PEFT strategy termed Domain Representation Injection (DRI)
  arXiv  Detail & Related papers  (2025-12-24T02:30:23Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• Beyond Fully Supervised Pixel Annotations: Scribble-Driven Weakly-Supervised Framework for Image Manipulation Localization [11.10178274806454]
  We propose a form of weak supervision that improves the annotation efficiency and detection performance.<n>We re-annotated mainstream IML datasets with scribble labels and propose the first scribble-based IML dataset.<n>We employ self-supervised training with a structural consistency loss to encourage the model to produce consistent predictions.
  arXiv  Detail & Related papers  (2025-07-17T11:45:27Z)
• Empowering Sparse-Input Neural Radiance Fields with Dual-Level Semantic Guidance from Dense Novel Views [66.1245505423179]
  We show that rendered semantics can be treated as a more robust form of augmented data than rendered RGB.<n>Our method enhances NeRF's performance by incorporating guidance derived from the rendered semantics.
  arXiv  Detail & Related papers  (2025-03-04T03:13:44Z)
• QPM: Discrete Optimization for Globally Interpretable Image Classification [17.460420995034216]
  We introduce the Quadratic Programming Enhanced Model (QPM), which learns globally interpretable class representations.<n>QPM represents every class with a binary assignment of very few, typically 5, features, that are also assigned to other classes.<n>The resulting optimal assignment is used to fine-tune the diverse features, so that each of them becomes the shared general concept.
  arXiv  Detail & Related papers  (2025-02-27T14:25:36Z)
• Predicting the Performance of Black-box LLMs through Self-Queries [60.87193950962585]
  Large language models (LLMs) are increasingly relied on in AI systems, predicting when they make mistakes is crucial.<n>In this paper, we extract features of LLMs in a black-box manner by using follow-up prompts and taking the probabilities of different responses as representations.<n>We demonstrate that training a linear model on these low-dimensional representations produces reliable predictors of model performance at the instance level.
  arXiv  Detail & Related papers  (2025-01-02T22:26:54Z)
• Rethinking interpretation: Input-agnostic saliency mapping of deep visual classifiers [28.28834523468462]
  Saliency methods provide post-hoc model interpretation by attributing input features to the model outputs. We show that input-specific saliency mapping is intrinsically susceptible to misleading feature attribution. We introduce a new perspective of input-agnostic saliency mapping that computationally estimates the high-level features attributed by the model to its outputs.
  arXiv  Detail & Related papers  (2023-03-31T06:58:45Z)
• Robust Face Anti-Spoofing with Dual Probabilistic Modeling [49.14353429234298]
  We propose a unified framework called Dual Probabilistic Modeling (DPM), with two dedicated modules, DPM-LQ (Label Quality aware learning) and DPM-DQ (Data Quality aware learning) DPM-LQ is able to produce robust feature representations without overfitting to the distribution of noisy semantic labels. DPM-DQ can eliminate data noise from False Reject' and False Accept' during inference by correcting the prediction confidence of noisy data based on its quality distribution.
  arXiv  Detail & Related papers  (2022-04-27T03:44:18Z)
• Layout-to-Image Translation with Double Pooling Generative Adversarial Networks [76.83075646527521]
  We propose a novel Double Pooing GAN (DPGAN) for generating photo-realistic and semantically-consistent results from the input layout. We also propose a novel Double Pooling Module (DPM), which consists of the Square-shape Pooling Module (SPM) and the Rectangle-shape Pooling Module ( RPM)
  arXiv  Detail & Related papers  (2021-08-29T19:55:14Z)
• A Framework to Learn with Interpretation [2.3741312212138896]
  We present a novel framework to jointly learn a predictive model and its associated interpretation model. We seek for a small-size dictionary of high level attribute functions that take as inputs the outputs of selected hidden layers. A detailed pipeline to visualize the learnt features is also developed.
  arXiv  Detail & Related papers  (2020-10-19T09:26:28Z)
• Interpretable Learning-to-Rank with Generalized Additive Models [78.42800966500374]
  Interpretability of learning-to-rank models is a crucial yet relatively under-examined research area. Recent progress on interpretable ranking models largely focuses on generating post-hoc explanations for existing black-box ranking models. We lay the groundwork for intrinsically interpretable learning-to-rank by introducing generalized additive models (GAMs) into ranking tasks.
  arXiv  Detail & Related papers  (2020-05-06T01:51:30Z)
• Feature Quantization Improves GAN Training [126.02828112121874]
  Feature Quantization (FQ) for the discriminator embeds both true and fake data samples into a shared discrete space. Our method can be easily plugged into existing GAN models, with little computational overhead in training.
  arXiv  Detail & Related papers  (2020-04-05T04:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.