Real-World Deployment and Evaluation of Kwame for Science, An AI Teaching Assistant for Science Education in West Africa

• URL: http://arxiv.org/abs/2302.10786v3
• Date: Fri, 26 Apr 2024 23:30:46 GMT
• Title: Real-World Deployment and Evaluation of Kwame for Science, An AI Teaching Assistant for Science Education in West Africa
• Abstract summary: Africa has a high student-to-teacher ratio which limits students' access to teachers for educational question answering. We extended Kwame, a bilingual AI teaching assistant for coding education, adapted it for science education, and deployed it as a web app. We deployed Kwame for Science in the real world over 8 months and had 750 users across 32 countries (15 in Africa) and 1.5K questions asked.
• Score: 0.769672852567215
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Africa has a high student-to-teacher ratio which limits students' access to teachers for learning support such as educational question answering. In this work, we extended Kwame, a bilingual AI teaching assistant for coding education, adapted it for science education, and deployed it as a web app. Kwame for Science provides passages from well-curated knowledge sources and related past national exam questions as answers to questions from students based on the Integrated Science subject of the West African Senior Secondary Certificate Examination (WASSCE). Furthermore, students can view past national exam questions along with their answers and filter by year, question type, and topics that were automatically categorized by a topic detection model which we developed (91% unweighted average recall). We deployed Kwame for Science in the real world over 8 months and had 750 users across 32 countries (15 in Africa) and 1.5K questions asked. Our evaluation showed an 87.2% top 3 accuracy (n=109 questions) implying that Kwame for Science has a high chance of giving at least one useful answer among the 3 displayed. We categorized the reasons the model incorrectly answered questions to provide insights for future improvements. We also share challenges and lessons with the development, deployment, and human-computer interaction component of such a tool to enable other researchers to deploy similar tools. With a first-of-its-kind tool within the African context, Kwame for Science has the potential to enable the delivery of scalable, cost-effective, and quality remote education to millions of people across Africa.

Related papers

• "Why SuaCode?": Understanding African Students' Motivations for Taking a Smartphone-Based Online Coding Course [0.1957338076370071]
  MOOCs are beneficial to post-secondary African students as less than 1% of them leave secondary school with fundamental coding skills.<n>One strategy for increasing their efficacy for African students is to understand students' motivation for enrolling.<n>We analyzed a random sample of 450 (of 3000) responses using a grounded theory approach.<n>We found that most African students participated in SuaCode for intrinsic reasons such as improving themselves, learning with like-minded individuals, and gaining skills to help address societal issues.
  arXiv  Detail & Related papers  (2026-07-24T22:49:51Z)
• Adesua: Development and Feasibility Study of an AI WhatsApp Bot for Science Learning in West Africa [0.17590081165362778]
  Adesua is a WhatsApp-based AI Teaching Assistant for science education.<n>System integrates curated textbooks and 33 years of national examination questions with generative AI.<n>Students can ask science questions, take timed or untimed multiple-choice tests by topic or exam year, and receive instant grading.
  arXiv  Detail & Related papers  (2026-05-14T20:04:39Z)
• Kwame 2.0: Human-in-the-Loop Generative AI Teaching Assistant for Large Scale Online Coding Education in Africa [0.548014499873554]
  We present Kwame 2.0, a bilingual (English-French) generative AI teaching assistant built using retrieval-augmented generation.<n>We deployed the system in a 15-month longitudinal study spanning 15 cohorts with 3,717 enrollments across 35 African countries.
  arXiv  Detail & Related papers  (2026-03-31T02:19:28Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education. Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs. We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants [175.9723801486487]
  We evaluate whether two AI assistants, GPT-3.5 and GPT-4, can adequately answer assessment questions. GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions. Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
  arXiv  Detail & Related papers  (2024-08-07T12:11:49Z)
• How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading [60.19226384241482]
  We introduce GuidingQ, a dataset of 10K in-text questions from textbooks and scientific articles. We explore various approaches to generate such questions using language models. We conduct a human study to understand the implication of such questions on reading comprehension.
  arXiv  Detail & Related papers  (2024-07-19T13:42:56Z)
• Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads [74.54183505245553]
  A systematic analysis of AI capabilities for joint vision and text reasoning is missing in the current scientific literature. We evaluate state-of-the-art LVLMs on their mathematical and algorithmic reasoning abilities using visuo-linguistic problems from children's Olympiads. Our results show that modern LVLMs do demonstrate increasingly powerful reasoning skills in solving problems for higher grades, but lack the foundations to correctly answer problems designed for younger children.
  arXiv  Detail & Related papers  (2024-06-22T05:04:39Z)
• Automated Question Generation for Science Tests in Arabic Language Using NLP Techniques [5.233512464561313]
  This research presents an innovative Arabic question-generation system built upon a three-stage process. The proposed approach and results show a precision of 83.50%, a recall of 78.68%, and an Fl score of 80.95%.
  arXiv  Detail & Related papers  (2024-06-11T20:27:45Z)
• Leveraging AI to Advance Science and Computing Education across Africa: Challenges, Progress and Opportunities [1.2691047660244332]
  We describe our work developing and deploying AI in Education tools in Africa for science and computing education. SuaCode is an AI-powered app that enables Africans to learn to code using their smartphones. AutoGrad is an automated grading, and feedback tool for graphical and interactive coding assignments. Kwame for Science is a web-based AI teaching assistant that provides instant answers to students' science questions.
  arXiv  Detail & Related papers  (2024-02-12T04:10:09Z)
• Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges [60.62904929065257]
  Large language models (LLMs) offer possibility for resolving this issue by comprehending individual requests. This paper reviews the recently emerged LLM research related to educational capabilities, including mathematics, writing, programming, reasoning, and knowledge-based question answering.
  arXiv  Detail & Related papers  (2023-12-27T14:37:32Z)
• Can ChatGPT pass the Vietnamese National High School Graduation Examination? [0.0]
  The study dataset included 30 essays in the literature test case and 1,700 multiple-choice questions designed for other subjects. ChatGPT was able to pass the examination with an average score of 6-7, demonstrating the technology's potential to revolutionize the educational landscape.
  arXiv  Detail & Related papers  (2023-06-15T14:47:03Z)
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering [124.16250115608604]
  We present Science Question Answering (SQA), a new benchmark that consists of 21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We show that SQA improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data.
  arXiv  Detail & Related papers  (2022-09-20T07:04:24Z)
• Kwame for Science: An AI Teaching Assistant Based on Sentence-BERT for Science Education in West Africa [1.6799377888527687]
  Africa has a high student-to-teacher ratio which limits students' access to teachers. We extended Kwame, our previous AI teaching assistant, adapted it for science education, and deployed it as a web app.
  arXiv  Detail & Related papers  (2022-06-28T02:27:23Z)
• Incremental Knowledge Based Question Answering [52.041815783025186]
  We propose a new incremental KBQA learning framework that can progressively expand learning capacity as humans do. Specifically, it comprises a margin-distilled loss and a collaborative selection method, to overcome the catastrophic forgetting problem. The comprehensive experiments demonstrate its effectiveness and efficiency when working with the evolving knowledge base.
  arXiv  Detail & Related papers  (2021-01-18T09:03:38Z)
• Kwame: A Bilingual AI Teaching Assistant for Online SuaCode Courses [2.4366811507669124]
  We developed a bilingual Artificial Intelligence (AI) Teaching Assistant (TA) that provides answers to students' coding questions from SuaCode courses in English and French. Kwame is a Sentence-BERT (SBERT)-based question-answering (QA) system that we trained and evaluated offline using question-answer pairs created from the course's quizzes, lesson notes and students' questions in past cohorts. Our results showed that fine-tuning on the course data and returning the top 3 and 5 answers improved the accuracy results.
  arXiv  Detail & Related papers  (2020-10-22T02:26:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.