Interpretable AI with Local Distillation

• URL: http://arxiv.org/abs/2608.23538v1
• Date: Mon, 24 Aug 2026 17:43:07 GMT
• Title: Interpretable AI with Local Distillation
• Abstract summary: High-stakes decisions call for models that are both accurate and interpretable as built.<n>We propose local distillation, in which a black-box "teacher" guides a regularized linear "student" model at each query point.<n>Across 17 benchmark datasets, local distillation nearly matches its AI teacher's accuracy while producing a sparse linear model at each test point.
• Score: 4.2444498670665665
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern AI models such as tabular foundation models and gradient-boosted ensembles can outpredict classical methods, but provide little basis for reasoning about their predictions. High-stakes decisions call for models that are both accurate and interpretable as built. Local linear modeling offers a path forward: a smooth regression function is locally well approximated by a linear one, allowing a linear fit near each query point to achieve high accuracy without sacrificing transparency. The challenges lie in learning what is "local" and developing statistical tools for interpretation. Here, we propose local distillation, in which a black-box "teacher" guides a regularized linear "student" model at each query point. The teacher (1) defines locality by upweighting training observations with similar predicted outcomes, and (2) anchors the fit with its prediction at the query point, included as a pseudo-observation whose weight is estimated from the data. For interpretation, we add a small amount of Gaussian randomization to the local objective and use refits to assess stability: selection frequencies identify reliable features at a query point, and clustering the randomized fits identifies stable subgroups across the data. Under the lasso penalty, we prove that this randomization yields feature-selection probabilities that are stable under small perturbations of the training responses. Across 17 benchmark datasets, local distillation nearly matches its AI teacher's accuracy while producing a sparse linear model at each test point. In a high-dimensional cancer gene expression example, the framework identifies patient subgroups whose local models use different genes; this heterogeneity is invisible to a global linear model, and difficult to surface in a black-box model.

Related papers

• Supervised learning pays attention [42.97070083645048]
  In-context learning with attention enables large neural networks to make context-specific predictions by selectively focusing on relevant examples.<n>We show how to flexibly fit personalized models for each prediction point and (2) model retain simplicity and interpretability.<n>Our method fits a local model for each test observation by weighting the training data according to attention, a supervised similarity measure.
  arXiv  Detail & Related papers  (2025-12-10T18:43:46Z)
• Distributed Gradient Descent with Many Local Steps in Overparameterized Models [20.560882414631784]
  In distributed training of machine learning models, gradient descent with local iterative steps is a popular method.<n>We try to explain this good performance from a viewpoint of implicit bias in Local Gradient Descent (Local-GD) with a large number of local steps.
  arXiv  Detail & Related papers  (2024-12-10T23:19:40Z)
• Measuring training variability from stochastic optimization using robust nonparametric testing [5.519968037738177]
  We propose a robust hypothesis testing framework and a novel summary statistic, the $alpha$-trimming level, to measure model similarity.<n>Applying hypothesis testing directly with the $alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis.<n>We show how to use the $alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics.
  arXiv  Detail & Related papers  (2024-06-12T15:08:15Z)
• Variational Classification [51.2541371924591]
  We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency. We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
  arXiv  Detail & Related papers  (2023-05-17T17:47:19Z)
• Performative Prediction with Neural Networks [24.880495520422]
  performative prediction is a framework for learning models that influence the data they intend to predict.<n>Standard convergence results for finding a performatively stable classifier with the method of repeated risk minimization assume that the data distribution is Lipschitz continuous to the model's parameters.<n>In this work, we instead assume that the data distribution is Lipschitz continuous with respect to the model's predictions, a more natural assumption for performative systems.
  arXiv  Detail & Related papers  (2023-04-14T01:12:48Z)
• Personalized Interpretable Classification [6.694135469324352]
  We make a first step towards formally introducing personalized interpretable classification as a new data mining problem.<n>We present a greedy algorithm called PIC to identify a personalized rule for each individual test sample.<n>Our algorithms can achieve the same-level predictive accuracy as those state-of-the-art (SOTA) interpretables.
  arXiv  Detail & Related papers  (2023-02-06T01:59:16Z)
• Predicting Out-of-Domain Generalization with Neighborhood Invariance [59.05399533508682]
  We propose a measure of a classifier's output invariance in a local transformation neighborhood. Our measure is simple to calculate, does not depend on the test point's true label, and can be applied even in out-of-domain (OOD) settings. In experiments on benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our measure and actual OOD generalization.
  arXiv  Detail & Related papers  (2022-07-05T14:55:16Z)
• Exploring Local Explanations of Nonlinear Models Using Animated Linear Projections [5.524804393257921]
  We show how to use eXplainable AI (XAI) to shed light on how a model use predictors to arrive at a prediction. To understand how the interaction between predictors affects the variable importance estimate, we can convert LVAs into linear projections. The approach is illustrated with examples from categorical (penguin species, chocolate types) and quantitative (soccer/football salaries, house prices) response models.
  arXiv  Detail & Related papers  (2022-05-11T09:11:02Z)
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model [78.55482897452417]
  We aim at improving data efficiency for both classification and regression setups in deep learning. To take the power of both worlds, we propose a novel X-model. X-model plays a minimax game between the feature extractor and task-specific heads.
  arXiv  Detail & Related papers  (2021-10-09T13:56:48Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Why do classifier accuracies show linear trends under distribution shift? [58.40438263312526]
  accuracies of models on one data distribution are approximately linear functions of the accuracies on another distribution. We assume the probability that two models agree in their predictions is higher than what we can infer from their accuracy levels alone. We show that a linear trend must occur when evaluating models on two distributions unless the size of the distribution shift is large.
  arXiv  Detail & Related papers  (2020-12-31T07:24:30Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.