Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education

• URL: http://arxiv.org/abs/2310.12324v2
• Date: Thu, 6 Jun 2024 21:30:07 GMT
• Title: Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education
• Authors: Ilya Musabirov, Angela Zavaleta-Bernuy, Pan Chen, Michael Liut, Joseph Jay Williams,
• Abstract summary: In adaptive experiments, data is analyzed and utilized as different conditions are deployed to students. These algorithms can then dynamically deploy the most effective conditions in subsequent interactions with students. This work paves the way for exploring the importance of adaptive experiments in bridging research and practice to achieve continuous improvement.
• Score: 7.50867730317249
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Randomized A/B comparisons of alternative pedagogical strategies or other course improvements could provide useful empirical evidence for instructor decision-making. However, traditional experiments do not provide a straightforward pathway to rapidly utilize data, increasing the chances that students in an experiment experience the best conditions. Drawing inspiration from the use of machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might aid continuous course improvement. In adaptive experiments, data is analyzed and utilized as different conditions are deployed to students. This can be achieved using machine learning algorithms to identify which actions are more beneficial in improving students' learning experiences and outcomes. These algorithms can then dynamically deploy the most effective conditions in subsequent interactions with students, resulting in better support for students' needs. We illustrate this approach with a case study that provides a side-by-side comparison of traditional and adaptive experiments on adding self-explanation prompts in online homework problems in a CS1 course. This work paves the way for exploring the importance of adaptive experiments in bridging research and practice to achieve continuous improvement in educational settings.

Related papers

• Dual Test-time Training for Out-of-distribution Recommender System [91.15209066874694]
  We propose a novel Dual Test-Time-Training framework for OOD Recommendation, termed DT3OR. In DT3OR, we incorporate a model adaptation mechanism during the test-time phase to carefully update the recommendation model. To the best of our knowledge, this paper is the first work to address OOD recommendation via a test-time-training strategy.
  arXiv  Detail & Related papers  (2024-07-22T13:27:51Z)
• Adaptive Experimentation When You Can't Experiment [55.86593195947978]
  This paper introduces the emphconfounded pure exploration transductive linear bandit (textttCPET-LB) problem. Online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment.
  arXiv  Detail & Related papers  (2024-06-15T20:54:48Z)
• Adaptive Instrument Design for Indirect Experiments [48.815194906471405]
  Unlike RCTs, indirect experiments estimate treatment effects by leveragingconditional instrumental variables. In this paper we take the initial steps towards enhancing sample efficiency for indirect experiments by adaptively designing a data collection policy. Our main contribution is a practical computational procedure that utilizes influence functions to search for an optimal data collection policy.
  arXiv  Detail & Related papers  (2023-12-05T02:38:04Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Conducting A/B Experiments with a Scalable Architecture [0.6990493129893112]
  A/B experiments are commonly used in research to compare the effects of changing one or more variables in two different experimental groups. We propose a four-principle approach for developing a software architecture to support A/B experiments that is domain agnostic.
  arXiv  Detail & Related papers  (2023-09-23T18:38:28Z)
• Designing Optimal Behavioral Experiments Using Machine Learning [8.759299724881219]
  We provide a tutorial on leveraging recent advances in BOED and machine learning to find optimal experiments for any kind of model. We consider theories of how people balance exploration and exploitation in multi-armed bandit decision-making tasks. As compared to experimental designs commonly used in the literature, we show that our optimal designs more efficiently determine which of a set of models best account for individual human behavior.
  arXiv  Detail & Related papers  (2023-05-12T18:24:30Z)
• Adaptive Experimental Design and Counterfactual Inference [20.666734673282495]
  This paper shares lessons learned regarding the challenges and pitfalls of naively using adaptive experimentation systems in industrial settings. We developed an adaptive experimental design framework for counterfactual inference based on these experiences.
  arXiv  Detail & Related papers  (2022-10-25T22:29:16Z)
• Using Adaptive Experiments to Rapidly Help Students [5.446351709118483]
  We evaluate the effect of homework email reminders in students by conducting an adaptive experiment using the Thompson Sampling algorithm. We raise a range of open questions about the conditions under which adaptive randomized experiments may be more or less useful.
  arXiv  Detail & Related papers  (2022-08-10T00:43:05Z)
• Increasing Students' Engagement to Reminder Emails Through Multi-Armed Bandits [60.4933541247257]
  This paper shows a real-world adaptive experiment on how students engage with instructors' weekly email reminders to build their time management habits. Using Multi-Armed Bandits (MAB) algorithms in adaptive experiments can increase students' chances of obtaining better outcomes. We highlight problems with these adaptive algorithms - such as possible exploitation of an arm when there is no significant difference.
  arXiv  Detail & Related papers  (2022-08-10T00:30:52Z)
• Sequential Bayesian experimental designs via reinforcement learning [0.0]
  We provide a new approach Sequential Experimental Design via Reinforcement Learning to construct BED in a sequential manner. By proposing a new real-world-oriented experimental environment, our approach aims to maximize the expected information gain. It is confirmed that our method outperforms the existing methods in various indices such as the EIG and sampling efficiency.
  arXiv  Detail & Related papers  (2022-02-14T04:29:04Z)
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data [100.83688818427915]
  We present training objectives that use offline datasets to learn a factored transition model. Our theoretical analysis shows that the learned latent action space can boost the sample-efficiency of downstream imitation learning. To learn the latent action space in practice, we propose TRAIL (Transition-Reparametrized Actions for Imitation Learning), an algorithm that learns an energy-based transition model.
  arXiv  Detail & Related papers  (2021-10-27T21:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.