K-Bench: measuring model performance on real scientific agent requests

• URL: http://arxiv.org/abs/2608.21601v1
• Date: Fri, 21 Aug 2026 20:06:08 GMT
• Title: K-Bench: measuring model performance on real scientific agent requests
• Abstract summary: K-Bench 01 is an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web.<n>Three blinded language-model judges scored every run against an eight-dimension rubric.<n>Across all 39,934 scored judgments, 47.6% fall below the 8-point threshold.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference solutions, or simulators with a known generative structure. Real scientific requests arrive differently. They are underspecified, they carry attachments, and lack ground truth. We report K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs. Three blinded language-model judges scored every run against an eight-dimension rubric. On a rubric whose 8-anchor instructs judges that a domain scientist would accept the work with minor edits, no model clears the line under all three judges. gpt-5.6-sol has the highest pooled mean, 8.04, but its 95% interval [7.80, 8.23] spans the threshold, and two of the three judges rank claude-opus-5 first instead. We therefore report the ordering of systems as the reproducible quantity, the absolute level as an attribute of the instrument, and the top of the table as unresolved. Across all 39,934 scored judgments -- the eight dimension scores plus a holistic overall for each assessment, excluding not-applicable cells -- 47.6% fall below the 8-point threshold. Difficulty is not uniform across the rubric: scientific accuracy averages 6.22 against 7.33 for communication, on identical denominators and in the same direction within every one of the nine models. The single leading failure tag is overclaiming, on 31.4% of assessments. We argue that the informative quantity for scientific agents is not a leaderboard position but the joint distribution of what was delivered, what was claimed, and what artifacts were produced.

Related papers

• Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead [9.25138311822007]
  Small differences on coding-agent leaderboards are often read as an ordering of systems.<n>We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models.<n>We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution.
  arXiv  Detail & Related papers  (2026-09-15T16:29:12Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task [15.259880314614117]
  We benchmark eleven frontier and open multimodal models under one honest, per-sample protocol.<n>SciTrue placed first by a clear margin in three of the four evidence-category/subtask combinations.
  arXiv  Detail & Related papers  (2026-09-01T03:30:45Z)
• Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards [0.0]
  We show that preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail.<n>On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all.<n>Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced.
  arXiv  Detail & Related papers  (2026-08-17T00:19:52Z)
• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents [0.0]
  2.5 Flash, 3.5 Flash, and 3.1 Pro are tested across three models in Gemini family.<n>Our primary evidence base uses Gemini 2.5 Flash ground-truth model validated against three human raters.
  arXiv  Detail & Related papers  (2026-07-08T23:24:55Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks [0.0]
  A paraphrase-quality audit of MathCheck detected 4 semantically incorrect paraphrases in 129 groups.<n>Removing them drops GPT-4o from rank 2 to rank 4 and elevates Claude Haiku and DeepSeek V3 above it.
  arXiv  Detail & Related papers  (2026-05-27T18:59:18Z)
• Evaluating Commercial AI Chatbots as News Intermediaries [85.32040752972836]
  Best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier.<n>Every model achieves its lowest accuracy on Hindi.<n> retrieval, not reasoning, failures drive over 70% of all errors.
  arXiv  Detail & Related papers  (2026-05-21T17:42:07Z)
• Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps [0.0]
  Frontier deep research agents (DRAs) plan a research task, synthesize across documents, and return a structured deliverable on demand.<n>Existing benchmarks measure factual recall, single-hop QA, or generic agentic skill, missing the multi-document, decision-grade work DRAs are deployed to produce.<n>We grade three frontier agents, namely Claude Opus 4.6 with web search, OpenAI o3-deep-research, and Google Gemini 3.1 Pro deep-research, on 42 SME-authored prompts.
  arXiv  Detail & Related papers  (2026-05-17T17:32:52Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation [34.429649156970015]
  Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
  arXiv  Detail & Related papers  (2026-04-03T16:30:58Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.