Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics

• URL: http://arxiv.org/abs/2608.21165v1
• Date: Fri, 21 Aug 2026 14:36:23 GMT
• Title: Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics
• Abstract summary: We propose a two-stage fine-tuning pipeline that distills a fitted black-box estimator and its post hoc interpretation (the mentor) into a small, open-weight large language model (LLM)<n>The design is estimator-agnostic and paired with a faithfulness-first evaluation framework that audits every narration against the attribution it claims to describe.<n>We find that fluency is no evidence of correctness since narration quality is independent of signal quality, and decision quality collapses toward the majority action in severely imbalanced settings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning analytics increasingly relies on flexible machine learning (ML), but the model opacity and the burden of deployment prevent these tools from reaching educational practice. We propose a two-stage fine-tuning pipeline that distills a fitted black-box estimator and its post hoc interpretation (the mentor) into a small, open-weight large language model (LLM; the mentee) that returns an individual-level estimate and explains in natural language. The design is estimator-agnostic and paired with a faithfulness-first evaluation framework that audits every narration against the attribution it claims to describe. We design a simulation study that separates distillation loss from estimator loss by comparing an oracle mentor with a realistic ML mentor. Given an oracle signal, distillation with a two-billion-parameter LLM model is nearly lossless in recovering the effect surface (r > .90), perfectly ranking the important variables, and citing no spurious covariate. Under a realistic estimator, almost all remaining error originates upstream. We find that fluency is no evidence of correctness since narration quality is independent of signal quality, and decision quality collapses toward the majority action in severely imbalanced settings. Applied to a nationally representative dataset, the pipeline recovers the finding that advanced mathematics coursework benefits students least likely to enroll in four-year college the most, with 98.8% of narrations passing the audit and no fabricated quantities. The result is a single fine-tuned LLM that predicts and explains offline on a commodity laptop, so student records never leave the machine.

Related papers

• A Survey of On-Policy Distillation for Large Language Models [19.95776080082138]
  This survey provides the first comprehensive overview of On-Policy Distillation for Large Language Models (LLMs)<n>We introduce a unified $f$-divergence framework over on-policy samples and organize the landscape along three dimensions: emphfeedback signal (logit-based, outcome-based, or self-play), emphteacher access (white-box, black-box, or teacher-free), and emphloss minimization (token-level, sequence-level, or hybrid)
  arXiv  Detail & Related papers  (2026-04-01T08:32:34Z)
• Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning [59.19460954480119]
  We study whether forgotten knowledge originates from pretraining or supervised fine-tuning.<n>Our experiments show that pretrained and SFT models respond differently to unlearning.
  arXiv  Detail & Related papers  (2026-02-23T08:58:48Z)
• Distillation Robustifies Unlearning [36.27570321651185]
  We show that training a model can drastically modify its input-output behavior while leaving its underlying capabilities intact.<n>We propose Unlearn-Noise-Distill-on-Outputs (UNDO), a scalable method that distills an unlearned model into a noised copy of itself.
  arXiv  Detail & Related papers  (2025-06-06T17:58:54Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought [51.240387516059535]
  We introduce a novel framework, LM-Guided CoT, that leverages a lightweight (i.e., 1B) language model (LM) for guiding a black-box large (i.e., >10B) LM in reasoning tasks. We optimize the model through 1) knowledge distillation and 2) reinforcement learning from rationale-oriented and task-oriented reward signals.
  arXiv  Detail & Related papers  (2024-04-04T12:46:37Z)
• Democratizing Reasoning Ability: Tailored Learning from Large Language Model [97.4921006089966]
  We propose a tailored learning approach to distill such reasoning ability to smaller LMs. We exploit the potential of LLM as a reasoning teacher by building an interactive multi-round learning paradigm. To exploit the reasoning potential of the smaller LM, we propose self-reflection learning to motivate the student to learn from self-made mistakes.
  arXiv  Detail & Related papers  (2023-10-20T07:50:10Z)
• Task-Aware Machine Unlearning and Its Application in Load Forecasting [4.00606516946677]
  This paper introduces the concept of machine unlearning which is specifically designed to remove the influence of part of the dataset on an already trained forecaster. A performance-aware algorithm is proposed by evaluating the sensitivity of local model parameter change using influence function and sample re-weighting. We tested the unlearning algorithms on linear, CNN, andMixer based load forecasters with a realistic load dataset.
  arXiv  Detail & Related papers  (2023-08-28T08:50:12Z)
• An Interpretability Evaluation Benchmark for Pre-trained Language Models [37.16893581395874]
  We propose a novel evaluation benchmark providing with both English and Chinese annotated data. It tests LMs abilities in multiple dimensions, i.e., grammar, semantics, knowledge, reasoning and computation. It contains perturbed instances for each original instance, so as to use the rationale consistency under perturbations as the metric for faithfulness.
  arXiv  Detail & Related papers  (2022-07-28T08:28:09Z)
• Unrolling SGD: Understanding Factors Influencing Machine Unlearning [17.6607904333012]
  Machine unlearning is the process through which a deployed machine learning model forgets about one of its training data points. We first taxonomize approaches and metrics of approximate unlearning. We identify verification error, i.e., the L2 difference between the weights of an approximately unlearned and a naively retrained model.
  arXiv  Detail & Related papers  (2021-09-27T23:46:59Z)
• Language Model Prior for Low-Resource Neural Machine Translation [85.55729693003829]
  We propose a novel approach to incorporate a LM as prior in a neural translation model (TM) We add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data.
  arXiv  Detail & Related papers  (2020-04-30T16:29:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.