AcademiClaw: When Students Set Challenges for AI Agents

• URL: http://arxiv.org/abs/2605.02661v1
• Date: Mon, 04 May 2026 14:40:42 GMT
• Title: AcademiClaw: When Students Set Challenges for AI Agents
• Abstract summary: AcademiClaw is a benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic disconnects.<n>We show that even the best of models achieves only a 55% pass rate.<n>We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community.
• Score: 69.70303995121154
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55\% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw.

Related papers

• OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios [25.68270661401607]
  We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces.<n>We construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes.<n>The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively.
  arXiv  Detail & Related papers  (2026-07-16T13:38:07Z)
• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering [0.0]
  Modern agentic systems operate at the granularity of a repository, a feature, or an algorithm.<n>We propose a six-layer reference architecture for agentic software engineering systems.<n>We argue that the central object of inquiry has shifted from code generation to delegated execution under human supervision.
  arXiv  Detail & Related papers  (2026-04-29T04:06:47Z)
• MatClaw: An Autonomous Code-First LLM Agent for End-to-End Materials Exploration [0.0]
  We present MatClaw, a code-first agent that writes and executes Python directly.<n>We show that MatClaw handles code generation reliably but struggles with tacit domain knowledge.<n>Our results demonstrate that the gap between guided and fully autonomous computational materials research is narrower than ever before.
  arXiv  Detail & Related papers  (2026-04-03T03:32:15Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)
• The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own [3.2334676138718677]
  This paper presents a comprehensive investigation into the capability of Large Language Models (LLMs) to successfully complete a control systems course.<n>We assess LLM performance using ChatGPT under a "minimal effort" protocol that simulates realistic student usage patterns.<n>Our analysis provides quantitative insights into AI's strengths and limitations in handling mathematical formulations, coding challenges, and theoretical concepts in control systems engineering.
  arXiv  Detail & Related papers  (2025-02-23T18:47:14Z)
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.