Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft

• URL: http://arxiv.org/abs/2604.24697v1
• Date: Mon, 27 Apr 2026 16:58:04 GMT
• Title: Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft
• Abstract summary: We introduce SciCrafter, a Minecraft-based benchmark that operationalizes the discovery-to-application loop.<n>We evaluate frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold.<n>Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle.
• Score: 70.0241202837925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discovering causal regularities and applying them to build functional systems--the discovery-to-application loop--is a hallmark of general intelligence, yet evaluating this capacity has been hindered by the vast complexity gap between scientific discovery and real-world engineering. We introduce SciCrafter, a Minecraft-based benchmark that operationalizes this loop through parameterized redstone circuit tasks. Agents must ignite lamps in specified patterns (e.g., simultaneously or in timed sequences); scaling target parameters substantially increases construction complexity and required knowledge, forcing genuine discovery rather than reliance on memorized solutions. Evaluating frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold, we find that all plateau at approximately 26% success rate. To diagnose these failures, we decompose the loop into four capacities--knowledge gap identification, experimental discovery, knowledge consolidation, and knowledge application--and design targeted interventions whose marginal contributions serve as proxies for corresponding gaps. Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle--indicating the bottleneck is shifting from solving problems right to raising the right problems for current AI. We release SciCrafter as a diagnostic probe for future research on AI systems that navigate the full discovery-to-application loop.

Related papers

• CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery [30.76801786098245]
  We introduce CausalSTeward (CAST), a novel human-in-the-loop framework for interactively assembling large causal models.<n>CaulSteward is a multi-agent collaborative system that tackles high-dimensional causality through a divide-and-conquer approach.
  arXiv  Detail & Related papers  (2026-07-02T09:29:10Z)
• GENESIS: Harnessing AI Agents for Autonomous 6G RAN Synthesis, Research, and Testing [13.436656977622691]
  Large Language Models (LLMs) have compressed comparable R&D work in general software engineering from days to minutes.<n>We present GENESIS, an agentic Artificial Intelligence (AI) framework that converts intents into solutions validated with over-the-air experiments.
  arXiv  Detail & Related papers  (2026-05-26T17:58:43Z)
• SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning [54.194672921128785]
  We introduce SciResearcher, a fully automated agentic framework for frontier-science data construction.<n>SciResearcher synthesizes diverse conceptual and computational tasks grounded in academic evidence.<n>We develop SciResearcher-8B, an agent foundation model that achieves 19.46% on the HLE-Bio/Chem-Gold benchmark.
  arXiv  Detail & Related papers  (2026-05-02T15:26:45Z)
• SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition [28.17858615204594]
  Grounded Multimodal Named Entity Recognition (GMNER) aims to extract named entities and localize their visual regions within image-text pairs.<n>In open-world social media platforms, GMNER remains challenging due to the prevalence of long-tailed, rapidly evolving, and unseen entities.<n>We propose SAKE, an end-to-end agentic framework that harmonizes internal knowledge exploitation and external knowledge exploration.
  arXiv  Detail & Related papers  (2026-04-22T03:17:36Z)
• Exploration and Exploitation Errors Are Measurable for Language Model Agents [52.95061000593404]
  Language Model (LM) agents are increasingly used in complex openended decision tasks.<n>A core requirement is the ability to both explore the problem space and exploit acquired knowledge effectively.<n>We design controllable environments inspired by practical embodied AI scenarios.
  arXiv  Detail & Related papers  (2026-04-14T17:59:57Z)
• AI-Supervisor: Autonomous AI Research Supervision via a Persistent Research World Model [1.14219428942199]
  Existing automated research systems operate as stateless, linear pipelines.<n>We present textbfAI-Supervisor, a multi-agent orchestration framework.<n>Agents provide end-to-end AI research supervision driven by human interests.
  arXiv  Detail & Related papers  (2026-03-25T15:16:51Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• Probing the Knowledge Boundary: An Interactive Agentic Framework for Deep Knowledge Extraction [29.717986496967978]
  We propose an interactive agentic framework to systematically extract and quantify the knowledge of Large Language Models.<n>Our method includes four adaptive exploration policies to probe knowledge at different granularities.<n>We observe a clear knowledge scaling law, where larger models consistently extract more knowledge.
  arXiv  Detail & Related papers  (2026-02-01T01:43:44Z)
• Executable Knowledge Graphs for Replicating AI Research [65.41207324831583]
  Executable Knowledge Graphs (xKG) is a modular and pluggable knowledge base that automatically integrates technical insights, code snippets, and domain-specific knowledge extracted from scientific literature.<n>Code will released at https://github.com/zjunlp/xKG.
  arXiv  Detail & Related papers  (2025-10-20T17:53:23Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• Fine-grained Stateful Knowledge Exploration: Effective and Efficient Graph Retrieval with Large Language Models [19.049828741139425]
  Large Language Models (LLMs) have shown impressive capabilities, yet updating their knowledge remains a significant challenge.<n>Most existing methods use a paradigm that treats the whole question as the objective, with relevant knowledge being incrementally retrieved from the knowledge graph.<n>We propose FiSKE, a novel paradigm for Fine-grained Stateful Knowledge Exploration.
  arXiv  Detail & Related papers  (2024-01-24T13:36:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.