Difficult Lessons on Social Prediction from Wisconsin Public Schools

• URL: http://arxiv.org/abs/2304.06205v2
• Date: Mon, 18 Sep 2023 13:57:09 GMT
• Title: Difficult Lessons on Social Prediction from Wisconsin Public Schools
• Authors: Juan C. Perdomo and Tolani Britton and Moritz Hardt and Rediet Abebe
• Abstract summary: Early warning systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect.
• Score: 32.90759447739759
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Early warning systems (EWS) are predictive tools at the center of recent efforts to improve graduation rates in public schools across the United States. These systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS, and the role of statistical risk scores in education. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We also find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect. Going beyond a retrospective evaluation of DEWS, we draw attention to a central question at the heart of the use of EWS: Are individual risk scores necessary for effectively targeting interventions? We propose a simple mechanism that only uses information about students' environments -- such as their schools, and districts -- and argue that this mechanism can target interventions just as efficiently as the individual risk score-based mechanism. Our argument holds even if individual predictions are highly accurate and effective interventions exist. In addition to motivating this simple targeting mechanism, our work provides a novel empirical backbone for the robust qualitative understanding among education researchers that dropout is structurally determined. Combined, our insights call into question the marginal value of individual predictions in settings where outcomes are driven by high levels of inequality.

Related papers

• Performative Prediction on Games and Mechanism Design [69.7933059664256]
  We study a collective risk dilemma where agents decide whether to trust predictions based on past accuracy. As predictions shape collective outcomes, social welfare arises naturally as a metric of concern. We show how to achieve better trade-offs and use them for mechanism design.
  arXiv  Detail & Related papers  (2024-08-09T16:03:44Z)
• Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
  In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
  arXiv  Detail & Related papers  (2024-04-29T08:16:30Z)
• A Simple and Practical Method for Reducing the Disparate Impact of Differential Privacy [21.098175634158043]
  Differentially private (DP) mechanisms have been deployed in a variety of high-impact social settings. The impact of DP on utility can vary significantly among different sub-populations. A simple way to reduce this disparity is with stratification.
  arXiv  Detail & Related papers  (2023-12-18T21:19:35Z)
• Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting [74.68371461260946]
  SocialSense is a framework that induces a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings.
  arXiv  Detail & Related papers  (2023-10-20T06:17:02Z)
• Who will dropout from university? Academic risk prediction based on interpretable machine learning [0.0]
  It predicts academic risk based on the LightGBM model and the interpretable machine learning method of Shapley value. From the local perspective, the factors affecting academic risk vary from person to person.
  arXiv  Detail & Related papers  (2021-12-02T09:43:31Z)
• Should College Dropout Prediction Models Include Protected Attributes? [0.4125187280299248]
  We build machine learning models to predict student dropout after one academic year. We compare the overall performance and fairness of model predictions with or without four protected attributes. We find that including protected attributes does not impact the overall prediction performance and it only marginally improves algorithmic fairness of predictions.
  arXiv  Detail & Related papers  (2021-03-28T22:47:30Z)
• Predicting Early Dropout: Calibration and Algorithmic Fairness Considerations [2.7048165023994057]
  We develop a machine learning method to predict the risks of university dropout and underperformance. We analyze if this method leads to discriminatory outcomes for some sensitive groups in terms of prediction accuracy (AUC) and error rates (Generalized False Positive Rate, GFPR, or Generalized False Negative Rate, GFNR)
  arXiv  Detail & Related papers  (2021-03-16T13:42:16Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)
• Predicting MOOCs Dropout Using Only Two Easily Obtainable Features from the First Week's Activities [56.1344233010643]
  Several features are considered to contribute towards learner attrition or lack of interest, which may lead to disengagement or total dropout. This study aims to predict dropout early-on, from the first week, by comparing several machine-learning approaches.
  arXiv  Detail & Related papers  (2020-08-12T10:44:49Z)
• Reducing the Filtering Effect in Public School Admissions: A Bias-aware Analysis for Targeted Interventions [7.50215102665518]
  We show that there is a shift in the distribution of scores obtained by students that the DOE classifies as "disadvantaged" We show that centrally planned interventions can significantly reduce the impact of bias through scholarships or training.
  arXiv  Detail & Related papers  (2020-04-22T20:50:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.