Distilling BlackBox to Interpretable models for Efficient Transfer Learning

• URL: http://arxiv.org/abs/2305.17303v7
• Date: Fri, 7 Jul 2023 21:30:01 GMT
• Title: Distilling BlackBox to Interpretable models for Efficient Transfer Learning
• Authors: Shantanu Ghosh, Ke Yu, Kayhan Batmanghelich
• Abstract summary: Building generalizable AI models is one of the primary challenges in the healthcare domain. Fine-tuning a model to transfer knowledge from one domain to another requires a significant amount of labeled data in the target domain. We develop an interpretable model that can be efficiently fine-tuned to an unseen target domain with minimal computational cost.
• Score: 19.40897632956169
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building generalizable AI models is one of the primary challenges in the healthcare domain. While radiologists rely on generalizable descriptive rules of abnormality, Neural Network (NN) models suffer even with a slight shift in input distribution (e.g., scanner type). Fine-tuning a model to transfer knowledge from one domain to another requires a significant amount of labeled data in the target domain. In this paper, we develop an interpretable model that can be efficiently fine-tuned to an unseen target domain with minimal computational cost. We assume the interpretable component of NN to be approximately domain-invariant. However, interpretable models typically underperform compared to their Blackbox (BB) variants. We start with a BB in the source domain and distill it into a \emph{mixture} of shallow interpretable models using human-understandable concepts. As each interpretable model covers a subset of data, a mixture of interpretable models achieves comparable performance as BB. Further, we use the pseudo-labeling technique from semi-supervised learning (SSL) to learn the concept classifier in the target domain, followed by fine-tuning the interpretable models in the target domain. We evaluate our model using a real-life large-scale chest-X-ray (CXR) classification dataset. The code is available at: \url{https://github.com/batmanlab/MICCAI-2023-Route-interpret-repeat-CXRs}.

Related papers

• Universality in Transfer Learning for Linear Models [18.427215139020625]
  We study the problem of transfer learning in linear models for both regression and binary classification. We provide an exact and rigorous analysis and relate generalization errors (in regression) and classification errors (in binary classification) for the pretrained and fine-tuned models.
  arXiv  Detail & Related papers  (2024-10-03T03:09:09Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Explainable AI for Comparative Analysis of Intrusion Detection Models [20.683181384051395]
  This research analyzes various machine learning models to the tasks of binary and multi-class classification for intrusion detection from network traffic. We trained all models to the accuracy of 90% on the UNSW-NB15 dataset. We also discover that Random Forest provides the best performance in terms of accuracy, time efficiency and robustness.
  arXiv  Detail & Related papers  (2024-06-14T03:11:01Z)
• Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation [10.793646707711442]
  We present a framework for training generative models for density estimation. We use the score-based diffusion model to generate labeled data. Once the labeled data are generated, we can train a simple fully connected neural network to learn the generative model in the supervised manner.
  arXiv  Detail & Related papers  (2023-10-22T23:56:19Z)
• DREAM: Domain-free Reverse Engineering Attributes of Black-box Model [51.37041886352823]
  We propose a new problem of Domain-agnostic Reverse Engineering the Attributes of a black-box target model. We learn a domain-agnostic model to infer the attributes of a target black-box model with unknown training data.
  arXiv  Detail & Related papers  (2023-07-20T16:25:58Z)
• Uncertainty-Aware Semi-Supervised Learning for Prostate MRI Zonal Segmentation [0.9176056742068814]
  We propose a novel semi-supervised learning (SSL) approach that requires only a relatively small number of annotations. Our method uses a pseudo-labeling technique that employs recent deep learning uncertainty estimation models. Our proposed model outperformed the semi-supervised model in experiments with the ProstateX dataset and an external test set.
  arXiv  Detail & Related papers  (2023-05-10T08:50:04Z)
• Learning from aggregated data with a maximum entropy model [73.63512438583375]
  We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis. We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
  arXiv  Detail & Related papers  (2022-10-05T09:17:27Z)
• Adaptive Convolutional Dictionary Network for CT Metal Artifact Reduction [62.691996239590125]
  We propose an adaptive convolutional dictionary network (ACDNet) for metal artifact reduction. Our ACDNet can automatically learn the prior for artifact-free CT images via training data and adaptively adjust the representation kernels for each input CT image. Our method inherits the clear interpretability of model-based methods and maintains the powerful representation ability of learning-based methods.
  arXiv  Detail & Related papers  (2022-05-16T06:49:36Z)
• Combining Discrete Choice Models and Neural Networks through Embeddings: Formulation, Interpretability and Performance [10.57079240576682]
  This study proposes a novel approach that combines theory and data-driven choice models using Artificial Neural Networks (ANNs) In particular, we use continuous vector representations, called embeddings, for encoding categorical or discrete explanatory variables. Our models deliver state-of-the-art predictive performance, outperforming existing ANN-based models while drastically reducing the number of required network parameters.
  arXiv  Detail & Related papers  (2021-09-24T15:55:31Z)
• Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
  Human-readability is an important and desirable standard for machine-learned model interpretability. We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code. We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
  arXiv  Detail & Related papers  (2021-01-21T01:46:36Z)
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking [63.49779304362376]
  Graph neural networks (GNNs) have become a popular approach to integrating structural inductive biases into NLP models. We introduce a post-hoc method for interpreting the predictions of GNNs which identifies unnecessary edges. We show that we can drop a large proportion of edges without deteriorating the performance of the model.
  arXiv  Detail & Related papers  (2020-10-01T17:51:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.