Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work

• URL: http://arxiv.org/abs/2605.21413v2
• Date: Thu, 21 May 2026 09:20:07 GMT
• Title: Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work
• Abstract summary: We introduce a course-based practice that teaches AI through benchmark construction.<n>Students turn disciplinary knowledge into verifiable expert-level questions.<n>Students evaluate AI systems on the resulting tasks.
• Score: 9.434503228899455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As AI becomes part of everyday learning, many courses teach students to use it mainly as a productivity tool: how to prompt, search, summarize, write, code, and use tools more efficiently. We argue that AI education also needs a setting in which students learn to test AI and understand their own role in judging machine-produced knowledge. To this end, we introduce a course-based practice that teaches AI through benchmark construction, using deep research systems as a concrete example of AI-era knowledge work. Students turn disciplinary knowledge into verifiable expert-level questions, review one another's designs for ambiguity and shortcuts, and evaluate AI systems on the resulting tasks. This activity gives students direct exposure to a powerful tool while asking them to specify what a trustworthy answer would require. The produced benchmark, QuestBench, consists of 256 questions across 14 humanities and social-science domains. Evaluation on QuestBench shows that student-designed tasks reveal hidden failures in current deep research systems: across thirteen evaluated systems, the mean question-level pass rate is only 16.85%, and the best-performing system, GPT-5.5, reaches a 57.58% pass rate. The failures are educationally useful because they show how fluent, source-backed answers can still miss the right query, source, term, or evidence standard. Reflections from five student contributors suggest that benchmark construction can help students see professional knowledge not only as content AI may retrieve, but as the basis for judging AI outputs. We present QuestBench as a benchmark artifact and as a reusable classroom setting for a larger educational question: how students can remain responsible knowledge actors as AI enters learning and professional work. The dataset is available at https://huggingface.co/datasets/PKUAIWeb/QuestBench/tree/main.

Related papers

• ASI-Bench: At the Dawn of Artificial Superintelligence [46.937202634067766]
  ASI-Bench is the first benchmark to jointly evaluate AI systems' capabilities of innovative exploration and autonomous scientific execution.<n>It contains 60 project-level research tasks across 11 scientific domains and progressively reduces methodological guidance.<n>Across 18 state-of-the-art agent--model configurations, the average score drops from 50.91 with full methodological guidance to 29.10 with only the method specified.
  arXiv  Detail & Related papers  (2026-08-18T01:59:00Z)
• "Why Put in This Much Effort?": How AI Availability Shapes Students' Motivation in Introductory Programming [0.15293427903448018]
  We study how engineering students experience and respond to AI availability.<n>Our findings complicate the assumption that students need external constraints to protect their learning.<n>Students who managed the tension found motivation in the learning process itself.
  arXiv  Detail & Related papers  (2026-06-29T15:41:43Z)
• Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks [0.0]
  We tested eleven AI tools on their ability to categorize mathematics tasks across four levels of cognitive demand.<n>On average, AI tools accurately classified cognitive demand in only 63% of cases.<n>All tools struggled with tasks at the extremes of cognitive demand.
  arXiv  Detail & Related papers  (2026-03-03T20:39:55Z)
• Can AI Chatbots Provide Coaching in Engineering? Beyond Information Processing Toward Mastery [1.4837859618020621]
  Engineering education faces a double disruption: traditional apprenticeship models that cultivated judgment and tacit skill are eroding, just as generative AI emerges as an informal coaching partner.<n>This convergence rekindles long-standing questions in the philosophy of AI and cognition about the limits of computation, the nature of embodied rationality, and the distinction between information processing and wisdom.<n>We synthesize critical perspectives from decades of scholarship on expertise, tacit knowledge, and human-machine interaction, situating them within the context of contemporary AI-driven education.<n>We propose a multiplex coaching framework that integrates human wisdom within expert-in-the-loop models, preserving the depth
  arXiv  Detail & Related papers  (2026-01-07T08:28:47Z)
• Learning to Use AI for Learning: How Can We Effectively Teach and Measure Prompting Literacy for K-12 Students? [1.413488665073795]
  There is a growing need to equip the next generation with the ability to apply, interact with, evaluate, and collaborate with AI systems responsibly.<n>To address this need, we designed an Large-Language Model (LLM)-based module to teach prompting literacy.<n>We conducted two iterations of classroom deployment in 11 authentic secondary education classrooms, and evaluated 1) AI-based auto-grader's capability; 2) students' prompting performance and confidence changes towards using AI for learning; and 3) the quality of learning and assessment materials.
  arXiv  Detail & Related papers  (2025-08-19T15:54:51Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems.<n>We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure.<n>Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants [176.39275404745098]
  We evaluate whether two AI assistants, GPT-3.5 and GPT-4, can adequately answer assessment questions.<n>GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions.<n>Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
  arXiv  Detail & Related papers  (2024-08-07T12:11:49Z)
• CourseAssist: Pedagogically Appropriate AI Tutor for Computer Science Education [1.052788652996288]
  This poster introduces CourseAssist, a novel LLM-based tutoring system tailored for computer science education. Unlike generic LLM systems, CourseAssist uses retrieval-augmented generation, user intent classification, and question decomposition to align AI responses with specific course materials and learning objectives.
  arXiv  Detail & Related papers  (2024-05-01T20:43:06Z)
• UKP-SQuARE: An Interactive Tool for Teaching Question Answering [61.93372227117229]
  The exponential growth of question answering (QA) has made it an indispensable topic in any Natural Language Processing (NLP) course. We introduce UKP-SQuARE as a platform for QA education. Students can run, compare, and analyze various QA models from different perspectives.
  arXiv  Detail & Related papers  (2023-05-31T11:29:04Z)
• The MineRL BASALT Competition on Learning from Human Feedback [58.17897225617566]
  The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. We provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline.
  arXiv  Detail & Related papers  (2021-07-05T12:18:17Z)
• Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience [76.9910678786031]
  We propose a novel paradigm of explainable active learning (XAL), by introducing techniques from the recently surging field of explainable AI (XAI) into an Active Learning setting. Our study shows benefits of AI explanation as interfaces for machine teaching--supporting trust calibration and enabling rich forms of teaching feedback, and potential drawbacks--anchoring effect with the model judgment and cognitive workload.
  arXiv  Detail & Related papers  (2020-01-24T22:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.