Fugu-MT: arxivの論文翻訳
Daily Report / 2026-08-18

Daily Paper Briefing

2026-08-18 1 Paper

ASI-Bench: At the Dawn of Artificial Superintelligence

Authors Junwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao, Yongchao Chen, Yuanning Feng, Junhao Wu, Cheng Zhang, Sijia Chen, Haoyu Xue, Chengsong You, Huan Wang, Koutian Wu, Peigan Gao, Jiakun Wu, Wenzhe Li, Ergan Shang, Qingyuan Zheng, Jingjing Zhou, Ruixuan Jia, Yan Xu, Hongrui Zhang, Xiao-Han Ma, Zhengxiang Cheng, Yuexing Hao, Liting Mai, Xianglin Ji, Wenjun Zhang, Zhuofan Chen, Yixiao Huang, Chi Wang, Wenyue Hua, Yilun Hao, Yuantao Zhai, Ziyan Zhao, Jingyan Xie
Affiliations Tsinghua University / Independent Researcher / AG2 AI / Carnegie Mellon University / Boston University / Harvard University / University of Michigan / University of Illinois Urbana-Champaign / University of Science and Technology of China / The Flatiron Institute / The University of Queensland / Microsoft / Massachusetts Institute of Technology
Categories Evaluation / Benchmarking / Multi-domain AI system evaluation, Application / Scientific AI / Autonomous scientific task performance, Method / Agent Models / Evaluation of agent-based AI systems
License CC BY 4.0

Abstract Overview

ASI-Bench is a benchmark for evaluating whether AI systems can conduct project-level scientific research as human methodological guidance is progressively reduced. It contains 60 tasks across 11 scientific domains, with matched B1-B4 settings that range from full procedural guidance to minimal guidance plus distractors. The benchmark was constructed and validated through extensive expert review, AI-assisted auditing, and sandbox execution to ensure scientific validity and scoring reliability. Across 18 agent-model configurations, the benchmark is used to measure not just execution under instructions, but also method selection, long-horizon research behavior, and the production of verifiable research artifacts.

Novelty

The paper introduces what it describes as the first benchmark to jointly assess innovative exploration and autonomous scientific execution across multiple research domains. Its distinctive feature is a within-task guidance gradient (B1-B4) that progressively removes methodological support, allowing direct measurement of how much current systems rely on human procedures.

Results

Performance declines sharply as guidance is withdrawn: the mean score across all evaluated systems drops from 50.91 in B1 to 29.10 in B2 and 26.62 in B3, while B4 remains similar at 26.99. The strongest reported B3 result is 51.60 from Codex with GPT-5.6 Sol (ultra), and it is the only evaluated configuration above 50 in that setting. The experiments also indicate that harness choice materially affects outcomes and that higher computational cost does not consistently yield better scientific performance.

Key Points

  1. ASI-Bench evaluates 60 project-level research tasks across 11 domains using a B1-B4 guidance gradient to probe scientific autonomy.
  2. Current agent-model systems remain strongly dependent on detailed procedural guidance, with much larger degradation from B1 to B2 than from B2 to B3.
  3. System performance depends not only on the backbone model but also on the agent harness, and compute spending shows an imperfect relationship with benchmark performance.

References

GitHub Hugging Face Hugging Face Project
This page was created using generative AI such as GPT-5, Claude Opus 4, Gemini 3, Gemini 3.1 Flash Image, and their higher-end successor versions. No guarantee can be made regarding its contents.