Joint learning of interpretation and distillation

• URL: http://arxiv.org/abs/2005.11638v1
• Date: Sun, 24 May 2020 02:01:22 GMT
• Title: Joint learning of interpretation and distillation
• Authors: Jinchao Huang, Guofu Li, Zhicong Yan, Fucai Luo, Shenghong Li
• Abstract summary: This paper conducts an empirical study on the new approach to explaining each prediction of GBDT2NN. Experiments on several benchmarks show that the proposed methods achieve better performance on both explanations and predictions.
• Score: 7.412850488684037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The extra trust brought by the model interpretation has made it an indispensable part of machine learning systems. But to explain a distilled model's prediction, one may either work with the student model itself, or turn to its teacher model. This leads to a more fundamental question: if a distilled model should give a similar prediction for a similar reason as its teacher model on the same input? This question becomes even more crucial when the two models have dramatically different structure, taking GBDT2NN for example. This paper conducts an empirical study on the new approach to explaining each prediction of GBDT2NN, and how imitating the explanation can further improve the distillation process as an auxiliary learning task. Experiments on several benchmarks show that the proposed methods achieve better performance on both explanations and predictions.

Related papers

• Explainability for Machine Learning Models: From Data Adaptability to User Perception [0.8702432681310401]
  This thesis explores the generation of local explanations for already deployed machine learning models. It aims to identify optimal conditions for producing meaningful explanations considering both data and user requirements.
  arXiv  Detail & Related papers  (2024-02-16T18:44:37Z)
• Self-Supervised Monocular Depth Estimation with Self-Reference Distillation and Disparity Offset Refinement [15.012694052674899]
  We propose two novel ideas to improve self-supervised monocular depth estimation. We use a parameter-optimized model as the teacher updated as the training epochs to provide additional supervision. We leverage the contextual consistency between high-scale and low-scale features to obtain multiscale disparity offsets.
  arXiv  Detail & Related papers  (2023-02-20T06:28:52Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Learning Interpretation with Explainable Knowledge Distillation [28.00216413365036]
  Knowledge Distillation (KD) has been considered as a key solution in model compression and acceleration in recent years. We propose a novel explainable knowledge distillation model, called XDistillation, through which both the performance the explanations' information are transferred from the teacher model to the student model. Our experiments shows that models trained by XDistillation outperform those trained by conventional KD methods in term of predictive accuracy and also faithfulness to the teacher models.
  arXiv  Detail & Related papers  (2021-11-12T21:18:06Z)
• Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models [62.28551903638434]
  We measure the impact of three different adaptation methods on the generalization and accuracy of models. Experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers. We observe that alternative adaptation methods like prefix-tuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.
  arXiv  Detail & Related papers  (2021-09-07T03:13:06Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction [49.254162397086006]
  We study explanations based on visual saliency in an image-based age prediction task. We find that presenting model predictions improves human accuracy. However, explanations of various kinds fail to significantly alter human accuracy or trust in the model.
  arXiv  Detail & Related papers  (2020-07-23T20:39:40Z)
• In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction [19.346391120556884]
  This study trains interpretable models that output probabilities rather than binary predictions, and uses quantitative fairness definitions to assess the models. We generated black-box and interpretable ML models on two different criminal recidivism datasets from Florida and Kentucky. Several interpretable ML models can predict recidivism as well as black-box ML models and are more accurate than COMPAS or the Arnold PSA.
  arXiv  Detail & Related papers  (2020-05-08T17:16:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.