The end of multiple choice tests: using AI to enhance assessment

• URL: http://arxiv.org/abs/2406.07481v1
• Date: Tue, 11 Jun 2024 17:24:30 GMT
• Title: The end of multiple choice tests: using AI to enhance assessment
• Authors: Michael Klymkowsky, Melanie M. Cooper,
• Abstract summary: Using research based distractors (wrong answers) are intrinsically limited in the insights they provide. To address these limitations, we recommend asking students to explain why they chose their answer. Using a discipline-trained artificial intelligence-based bot it is possible to analyze their explanations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Effective teaching relies on knowing what students know-or think they know. Revealing student thinking is challenging. Often used because of their ease of grading, even the best multiple choice (MC) tests, those using research based distractors (wrong answers) are intrinsically limited in the insights they provide due to two factors. When distractors do not reflect student beliefs they can be ignored, increasing the likelihood that the correct answer will be chosen by chance. Moreover, making the correct choice does not guarantee that the student understands why it is correct. To address these limitations, we recommend asking students to explain why they chose their answer, and why "wrong" choices are wrong. Using a discipline-trained artificial intelligence-based bot it is possible to analyze their explanations, identifying the concepts and scientific principles that maybe missing or misapplied. The bot also makes suggestions for how instructors can use these data to better guide student thinking. In a small "proof of concept" study, we tested this approach using questions from the Biology Concepts Instrument (BCI). The result was rapid, informative, and provided actionable feedback on student thinking. It appears that the use of AI addresses the weaknesses of conventional MC test. It seems likely that incorporating AI-analyzed formative assessments will lead to improved overall learning outcomes.

Related papers

• How Do Programming Students Use Generative AI? [7.863638253070439]
  We studied how programming students actually use generative AI tools like ChatGPT. We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions. Our findings indicate that concerns about potential decrease in programmers' agency and productivity with Generative AI are justified.
  arXiv  Detail & Related papers  (2025-01-17T10:25:41Z)
• Enhancing Knowledge Tracing with Concept Map and Response Disentanglement [5.201585012263761]
  We propose the Concept map-driven Response disentanglement method for enhancing Knowledge Tracing (CRKT) model. CRKT benefits KT by directly leveraging answer choices--beyond merely identifying correct or incorrect answers--to distinguish responses with different incorrect choices. We further introduce the novel use of unchosen responses by employing disentangled representations to get insights from options not selected by students.
  arXiv  Detail & Related papers  (2024-08-23T11:25:56Z)
• RECKONING: Reasoning through Dynamic Knowledge Encoding [51.076603338764706]
  We show that language models can answer questions by reasoning over knowledge provided as part of the context. In these situations, the model fails to distinguish the knowledge that is necessary to answer the question. We propose teaching the model to reason more robustly by folding the provided contextual knowledge into the model's parameters.
  arXiv  Detail & Related papers  (2023-05-10T17:54:51Z)
• Active Reward Learning from Multiple Teachers [17.10187575303075]
  Reward learning algorithms utilize human feedback to infer a reward function, which is then used to train an AI system. This human feedback is often a preference comparison, in which the human teacher compares several samples of AI behavior and chooses which they believe best accomplishes the objective. While reward learning typically assumes that all feedback comes from a single teacher, in practice these systems often query multiple teachers to gather sufficient training data.
  arXiv  Detail & Related papers  (2023-03-02T01:26:53Z)
• Towards Human Cognition Level-based Experiment Design for Counterfactual Explanations (XAI) [68.8204255655161]
  The emphasis of XAI research appears to have turned to a more pragmatic explanation approach for better understanding. An extensive area where cognitive science research may substantially influence XAI advancements is evaluating user knowledge and feedback. We propose a framework to experiment with generating and evaluating the explanations on the grounds of different cognitive levels of understanding.
  arXiv  Detail & Related papers  (2022-10-31T19:20:22Z)
• Increasing Students' Engagement to Reminder Emails Through Multi-Armed Bandits [60.4933541247257]
  This paper shows a real-world adaptive experiment on how students engage with instructors' weekly email reminders to build their time management habits. Using Multi-Armed Bandits (MAB) algorithms in adaptive experiments can increase students' chances of obtaining better outcomes. We highlight problems with these adaptive algorithms - such as possible exploitation of an arm when there is no significant difference.
  arXiv  Detail & Related papers  (2022-08-10T00:30:52Z)
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning [19.324012098032515]
  When people receive advice while making difficult decisions, they often make better decisions in the moment and also increase their knowledge in the process. How do people process the information and advice they receive from AI, and do they engage with it deeply enough to enable learning? This work provides some of the most direct evidence to date that it may not be sufficient to include explanations together with AI-generated recommendation.
  arXiv  Detail & Related papers  (2022-02-11T01:28:59Z)
• The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning [59.777127897688594]
  We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices.
  arXiv  Detail & Related papers  (2022-01-24T13:18:02Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• Option Tracing: Beyond Correctness Analysis in Knowledge Tracing [3.1798318618973362]
  We extend existing knowledge tracing methods to predict the exact option students select in multiple choice questions. We quantitatively evaluate the performance of our option tracing methods on two large-scale student response datasets.
  arXiv  Detail & Related papers  (2021-04-19T04:28:34Z)
• Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience [76.9910678786031]
  We propose a novel paradigm of explainable active learning (XAL), by introducing techniques from the recently surging field of explainable AI (XAI) into an Active Learning setting. Our study shows benefits of AI explanation as interfaces for machine teaching--supporting trust calibration and enabling rich forms of teaching feedback, and potential drawbacks--anchoring effect with the model judgment and cognitive workload.
  arXiv  Detail & Related papers  (2020-01-24T22:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.