An Integrated Machine Learning and Hierarchical Variance Decomposition Pipeline for Student Performance Prediction and Metacognitive Calibration on Multi-Signal Telemetry

• URL: http://arxiv.org/abs/2606.28881v1
• Date: Sat, 27 Jun 2026 11:59:41 GMT
• Title: An Integrated Machine Learning and Hierarchical Variance Decomposition Pipeline for Student Performance Prediction and Metacognitive Calibration on Multi-Signal Telemetry
• Abstract summary: Prior research treats performance prediction, calibration error calculation, and variance decomposition as separate pipelines.<n>I propose the Unified Behavioral Prediction and Analysis Pipeline (UBP-CAP), an integrated framework processing student pre-execution behavioral telemetry.<n>I introduce the Predictive-Explanatory Divergence Index (PEDI), which quantifies structural divergence between predictive and explanatory feature profiles.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting student performance and characterizing metacognitive calibration are essential for personalization in intelligent tutoring systems. Prior research treats performance prediction, calibration error calculation, and variance decomposition as separate pipelines, preventing unified interpretation. I propose the Unified Behavioral Prediction and Calibration Analysis Pipeline (UBP-CAP), an integrated framework processing student pre-execution behavioral telemetry through three linked modules: (1) a LightGBM classifier with SHAP for binary correctness prediction, (2) formal calibration metrics (ECE, MCE, and Brier score decomposition) to evaluate metacognitive alignment, and (3) a crossed Generalized Linear Mixed-Effects Model (GLMM) for decomposing calibration deviations. I introduce the Predictive-Explanatory Divergence Index (PEDI), which quantifies structural divergence between predictive and explanatory feature profiles. Evaluated on 1,195 interaction records (27 students, 45 tasks), Logistic Regression achieves AUC-ROC = 0.903, outperforming LightGBM (0.878). Student naive ECE (0.109) significantly exceeds model ECE (0.068), confirming systematic miscalibration. The crossed GLMM yields ICCStudent = 0.123, showing calibration is situational rather than dispositional. PEDIcos = 0.081 (p = 0.327) indicates structural alignment between prediction and explanation on shared behavioral features.

Related papers

• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection [13.891522069967507]
  We present an information-geometric framework providing the first rigorous characterization of MoE dynamics specialization.<n>Our key insight is that expert routing distributions evolve on the probability simplex equipped with the Fisher information metric.
  arXiv  Detail & Related papers  (2026-04-16T00:28:41Z)
• Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning [0.0]
  A unified sparsification primitive would make accuracy-efficiency trade-offs comparable across modalities.<n>We propose a modality-agnostic, feature-wise hard-concrete gating framework that enforces L0-style sparsity directly on learned representations.
  arXiv  Detail & Related papers  (2026-03-26T03:33:42Z)
• PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering [71.15346406323827]
  We introduce PRIME, a benchmark for evaluating verifiers on Process-Outcome Alignment verification.<n>We find that current verifiers frequently fail to detect derivation flaws.<n>We propose a process-aware RLVR training paradigm utilizing verifiers selected via PRIME.
  arXiv  Detail & Related papers  (2026-02-12T04:45:01Z)
• Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods [0.0]
  We study model-agnostic post-hoc calibration methods to improve probabilistic predictions in supervised binary classification.<n>We benchmark 21 widely used classifiers, including linear models, SVMs, tree ensembles (CatBoost, XGBoost, LightGBM)<n>We find that commonly used calibration procedures, most notably Platt scaling and isotonic regression, can systematically degrade proper scoring performance.
  arXiv  Detail & Related papers  (2026-01-19T18:23:36Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency [47.90235939359225]
  We propose a novel mixup strategy for pre-trained language models that improves model calibration further. Our method achieves the lowest expected calibration error compared to strong baselines on both in-domain and out-of-domain test samples.
  arXiv  Detail & Related papers  (2022-03-14T23:45:08Z)
• MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration [45.440008446506404]
  We propose a feature-aware binning framework, called Multiple Boosting Trees (MBCT)<n>Our MBCT is non-monotonic, and has the potential to improve order accuracy, due to its learnable binning scheme and the individual calibration.<n>Results show that our method outperforms all competing models in terms of both calibration error and order accuracy.
  arXiv  Detail & Related papers  (2022-02-09T08:59:16Z)
• Should Ensemble Members Be Calibrated? [16.331175260764]
  Modern deep neural networks are often observed to be poorly calibrated. Deep learning approaches make use of large numbers of model parameters. This paper explores the application of calibration schemes to deep ensembles.
  arXiv  Detail & Related papers  (2021-01-13T23:59:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.