Difficult Lessons on Social Prediction from Wisconsin Public Schools

• URL: http://arxiv.org/abs/2304.06205v2
• Date: Mon, 18 Sep 2023 13:57:09 GMT
• Title: Difficult Lessons on Social Prediction from Wisconsin Public Schools
• Authors: Juan C. Perdomo and Tolani Britton and Moritz Hardt and Rediet Abebe
• Abstract summary: Early warning systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect.
• Score: 32.90759447739759
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Early warning systems (EWS) are predictive tools at the center of recent efforts to improve graduation rates in public schools across the United States. These systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS, and the role of statistical risk scores in education. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We also find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect. Going beyond a retrospective evaluation of DEWS, we draw attention to a central question at the heart of the use of EWS: Are individual risk scores necessary for effectively targeting interventions? We propose a simple mechanism that only uses information about students' environments -- such as their schools, and districts -- and argue that this mechanism can target interventions just as efficiently as the individual risk score-based mechanism. Our argument holds even if individual predictions are highly accurate and effective interventions exist. In addition to motivating this simple targeting mechanism, our work provides a novel empirical backbone for the robust qualitative understanding among education researchers that dropout is structurally determined. Combined, our insights call into question the marginal value of individual predictions in settings where outcomes are driven by high levels of inequality.

Related papers

• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on causal inferences on a target experiment with unlabeled factual outcomes, retrieved by a predictive model fine-tuned on a labeled similar experiment. First, we show that factual outcome estimation via Empirical Risk Minimization (ERM) may fail to yield valid causal inferences on the target population. We propose Deconfounded Empirical Risk Minimization (DERM), a new simple learning procedure minimizing the risk over a fictitious target population.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• The Value of Prediction in Identifying the Worst-Off [3.468330970960535]
  Machine learning is increasingly used in government programs to identify and support the most vulnerable individuals. This paper examines the welfare impacts of prediction in equity-driven contexts, and how they compare to other policy levers.
  arXiv  Detail & Related papers  (2025-01-31T17:34:53Z)
• Trading off performance and human oversight in algorithmic policy: evidence from Danish college admissions [11.378331161188022]
  Student dropout is a significant concern for educational institutions. We show that sequential AI models offer more precise and fair predictions. We estimate that even the use of simple AI models to guide admissions decisions could yield significant economic benefits.
  arXiv  Detail & Related papers  (2024-11-22T21:12:54Z)
• Performative Prediction on Games and Mechanism Design [69.7933059664256]
  We study a collective risk dilemma where agents decide whether to trust predictions based on past accuracy. As predictions shape collective outcomes, social welfare arises naturally as a metric of concern. We show how to achieve better trade-offs and use them for mechanism design.
  arXiv  Detail & Related papers  (2024-08-09T16:03:44Z)
• Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
  In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
  arXiv  Detail & Related papers  (2024-04-29T08:16:30Z)
• Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting [74.68371461260946]
  SocialSense is a framework that induces a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings.
  arXiv  Detail & Related papers  (2023-10-20T06:17:02Z)
• Should College Dropout Prediction Models Include Protected Attributes? [0.4125187280299248]
  We build machine learning models to predict student dropout after one academic year. We compare the overall performance and fairness of model predictions with or without four protected attributes. We find that including protected attributes does not impact the overall prediction performance and it only marginally improves algorithmic fairness of predictions.
  arXiv  Detail & Related papers  (2021-03-28T22:47:30Z)
• Predicting Early Dropout: Calibration and Algorithmic Fairness Considerations [2.7048165023994057]
  We develop a machine learning method to predict the risks of university dropout and underperformance. We analyze if this method leads to discriminatory outcomes for some sensitive groups in terms of prediction accuracy (AUC) and error rates (Generalized False Positive Rate, GFPR, or Generalized False Negative Rate, GFNR)
  arXiv  Detail & Related papers  (2021-03-16T13:42:16Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)
• Reducing the Filtering Effect in Public School Admissions: A Bias-aware Analysis for Targeted Interventions [7.50215102665518]
  We show that there is a shift in the distribution of scores obtained by students that the DOE classifies as "disadvantaged" We show that centrally planned interventions can significantly reduce the impact of bias through scholarships or training.
  arXiv  Detail & Related papers  (2020-04-22T20:50:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.