Machine Learning Who to Nudge: Causal vs Predictive Targeting in a Field Experiment on Student Financial Aid Renewal

• URL: http://arxiv.org/abs/2310.08672v2
• Date: Fri, 31 May 2024 23:59:02 GMT
• Title: Machine Learning Who to Nudge: Causal vs Predictive Targeting in a Field Experiment on Student Financial Aid Renewal
• Authors: Susan Athey, Niall Keleher, Jann Spiess,
• Abstract summary: We analyze the value of targeting in a large-scale field experiment with over 53,000 college students. We show that targeting based on low baseline outcomes is most effective in our specific application.
• Score: 5.044100238869374
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many settings, interventions may be more effective for some individuals than others, so that targeting interventions may be beneficial. We analyze the value of targeting in the context of a large-scale field experiment with over 53,000 college students, where the goal was to use "nudges" to encourage students to renew their financial-aid applications before a non-binding deadline. We begin with baseline approaches to targeting. First, we target based on a causal forest that estimates heterogeneous treatment effects and then assigns students to treatment according to those estimated to have the highest treatment effects. Next, we evaluate two alternative targeting policies, one targeting students with low predicted probability of renewing financial aid in the absence of the treatment, the other targeting those with high probability. The predicted baseline outcome is not the ideal criterion for targeting, nor is it a priori clear whether to prioritize low, high, or intermediate predicted probability. Nonetheless, targeting on low baseline outcomes is common in practice, for example because the relationship between individual characteristics and treatment effects is often difficult or impossible to estimate with historical data. We propose hybrid approaches that incorporate the strengths of both predictive approaches (accurate estimation) and causal approaches (correct criterion); we show that targeting intermediate baseline outcomes is most effective in our specific application, while targeting based on low baseline outcomes is detrimental. In one year of the experiment, nudging all students improved early filing by an average of 6.4 percentage points over a baseline average of 37% filing, and we estimate that targeting half of the students using our preferred policy attains around 75% of this benefit.

Related papers

• Learning treatment effects while treating those in need [20.99198458867724]
  We propose a framework to design randomized allocation rules which optimally balance targeting high-need individuals with learning treatment effects. We apply our framework to data from human services in Allegheny County, Pennsylvania.
  arXiv  Detail & Related papers  (2024-07-10T12:29:46Z)
• Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
  In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
  arXiv  Detail & Related papers  (2024-04-29T08:16:30Z)
• Preference Poisoning Attacks on Reward Model Learning [49.806139447922526]
  We show how an attacker can flip a small subset of preference comparisons with the goal of either promoting or demoting a target outcome. We find that the best attacks are often highly successful, achieving in the most extreme case 100% success rate with only 0.3% of the data poisoned. We also show that several state-of-the-art defenses against other classes of poisoning attacks exhibit, at best, limited efficacy in our setting.
  arXiv  Detail & Related papers  (2024-02-02T21:45:24Z)
• Targeted Machine Learning for Average Causal Effect Estimation Using the Front-Door Functional [3.0232957374216953]
  evaluating the average causal effect (ACE) of a treatment on an outcome often involves overcoming the challenges posed by confounding factors in observational studies. Here, we introduce novel estimation strategies for the front-door criterion based on the targeted minimum loss-based estimation theory. We demonstrate the applicability of these estimators to analyze the effect of early stage academic performance on future yearly income.
  arXiv  Detail & Related papers  (2023-12-15T22:04:53Z)
• When is Off-Policy Evaluation Useful? A Data-Centric Perspective [60.76880827781716]
  evaluating the value of a hypothetical target policy with only a logged dataset is important but challenging. We propose DataCOPE, a data-centric framework for evaluating a target policy given a dataset.
  arXiv  Detail & Related papers  (2023-11-23T17:13:37Z)
• Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care [68.8204255655161]
  We introduce a deep Q-learning approach able to obtain more reliable critical care policies. We achieve this by first pruning the action set based on all available rewards, and second training a final model based on the sparse main reward but with a restricted action set.
  arXiv  Detail & Related papers  (2023-06-13T18:02:57Z)
• Difficult Lessons on Social Prediction from Wisconsin Public Schools [32.90759447739759]
  Early warning systems assist in targeting interventions to individual students by predicting which students are at risk of dropping out. Despite significant investments in their widespread adoption, there remain large gaps in our understanding of the efficacy of EWS. We present empirical evidence that the prediction system accurately sorts students by their dropout risk. We find that it may have caused a single-digit percentage increase in graduation rates, though our empirical analyses cannot reliably rule out that there has been no positive treatment effect.
  arXiv  Detail & Related papers  (2023-04-13T00:59:12Z)
• Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation [60.71312668265873]
  We develop a method to balance the need for personalization with confident predictions. We show that our method can be used to form accurate predictions of heterogeneous treatment effects.
  arXiv  Detail & Related papers  (2021-11-28T23:19:12Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Targeting for long-term outcomes [1.7205106391379026]
  Decision makers often want to target interventions so as to maximize an outcome that is observed only in the long-term. Here we build on the statistical surrogacy and policy learning literatures to impute the missing long-term outcomes. We apply our approach in two large-scale proactive churn management experiments at The Boston Globe.
  arXiv  Detail & Related papers  (2020-10-29T18:31:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.