The "Astonishing Regularity'' Revisited: Sensitivity of Learning-Rate Estimates to Practice-Sequence Length

• URL: http://arxiv.org/abs/2605.01690v1
• Date: Sun, 03 May 2026 03:07:36 GMT
• Title: The "Astonishing Regularity'' Revisited: Sensitivity of Learning-Rate Estimates to Practice-Sequence Length
• Abstract summary: We probe unexamined assumption that observation length in student log data is ignorable for mixed-effects estimation.<n>We show that capping at the first ten opportunities per student-skill pair inflates the median estimated IQR of student learning rates by 75%.<n>We argue that practice sequence length distributions are an unexamined property of mixed-effects estimation on observational learning data.
• Score: 0.7966335996690441
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A 2023 \textit{PNAS} study by Koedinger et al. (2023) fit the individual Additive Factors Model (iAFM) to 27 educational datasets and reported an ``astonishing regularity'' in student learning rates: students vary substantially in initial knowledge but learn at remarkably similar rates with practice. We probe a largely unexamined assumption underlying this finding -- that observation length in student log data is ignorable for mixed-effects estimation -- by refitting the iAFM on 26 of the original datasets while systematically truncating practice sequences at various depths, holding the set of students and knowledge components constant. Capping at the first ten opportunities per student-skill pair inflates the median estimated IQR of student learning rates by 75\%; capping at five inflates it by 205\%, with individual datasets ranging from negligible to 17-fold. The magnitude of this sensitivity diverges from what standard estimation theory predicts under ignorable truncation, and the dataset-specific heterogeneity is substantial. Three candidate mechanisms from adjacent literatures could account for the pattern -- informative observation length, functional-form misspecification, and identification weakness from sparse per-pair data -- but observational analysis on these data alone cannot adjudicate among them. We argue that practice sequence length distributions are an unexamined property of mixed-effects estimation on observational learning data, deserving explicit reporting before conclusions about learning-rate heterogeneity are drawn.

Related papers

• Robust Molecular Property Prediction via Densifying Scarce Labeled Data [53.24886143129006]
  In drug discovery, compounds most critical for advancing research often lie beyond the training set.<n>We propose a novel bilevel optimization approach that leverages unlabeled data to interpolate between in-distribution (ID) and out-of-distribution (OOD) data.
  arXiv  Detail & Related papers  (2025-06-13T15:27:40Z)
• Leveraging Per-Instance Privacy for Machine Unlearning [43.55855852029299]
  We present a principled, per-instance approach to quantifying the difficulty of unlearning via fine-tuning.<n>Our findings provide a foundation for more efficient and adaptive unlearning strategies tailored to the unique properties of individual data points.
  arXiv  Detail & Related papers  (2025-05-24T16:55:57Z)
• On the Joint Interaction of Models, Data, and Features [82.60073661644435]
  We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
  arXiv  Detail & Related papers  (2023-06-07T21:35:26Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Learning Invariant Representations with Missing Data [18.307438471163774]
  Models that satisfy particular independencies involving correlation-inducing textitnuisance variables have guarantees on their test performance. We derive acrshortmmd estimators used for invariance objectives under missing nuisances. On simulations and clinical data, optimizing through these estimates achieves test performance similar to using estimators that make use of the full data.
  arXiv  Detail & Related papers  (2021-12-01T23:14:34Z)
• OR-Net: Pointwise Relational Inference for Data Completion under Partial Observation [51.083573770706636]
  This work uses relational inference to fill in the incomplete data. We propose Omni-Relational Network (OR-Net) to model the pointwise relativity in two aspects.
  arXiv  Detail & Related papers  (2021-05-02T06:05:54Z)
• Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding [38.09565581056218]
  We study the problem of learning conditional average treatment effects (CATE) from high-dimensional, observational data with unobserved confounders. We present a new parametric interval estimator suited for high-dimensional data.
  arXiv  Detail & Related papers  (2021-03-08T15:58:06Z)
• Correct block-design experiments mitigate temporal correlation bias in EEG classification [68.85562949901077]
  We show that the main claim in [1] is drastically overstated and their other analyses are seriously flawed by wrong methodological choices. We investigate the influence of EEG temporal correlation on classification accuracy by testing the same models in two additional experimental settings.
  arXiv  Detail & Related papers  (2020-11-25T22:25:21Z)
• On Disentangled Representations Learned From Correlated Data [59.41587388303554]
  We bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data. We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.
  arXiv  Detail & Related papers  (2020-06-14T12:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.