Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery

• URL: http://arxiv.org/abs/2603.03322v1
• Date: Tue, 10 Feb 2026 05:47:22 GMT
• Title: Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery
• Abstract summary: DBench-Bio is a dynamic and fully automated benchmark to evaluate AI's biological knowledge discovery ability.<n>We instantiate this pipeline to construct a monthly-updated benchmark covering 12 biomedical sub-domains.<n>Our work provides the first dynamic, automatic framework for assessing the new knowledge discovery capabilities of AI systems.
• Score: 81.03797680309154
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in Large Language Model (LLM) agents have demonstrated remarkable potential in automatic knowledge discovery. However, rigorously evaluating an AI's capacity for knowledge discovery remains a critical challenge. Existing benchmarks predominantly rely on static datasets, leading to inevitable data contamination where models have likely seen the evaluation knowledge during training. Furthermore, the rapid release cycles of modern LLMs render static benchmarks quickly outdated, failing to assess the ability to discover truly new knowledge. To address these limitations, we propose DBench-Bio, a dynamic and fully automated benchmark designed to evaluate AI's biological knowledge discovery ability. DBench-Bio employs a three-stage pipeline: (1) data acquisition of rigorous, authoritative paper abstracts; (2) QA extraction utilizing LLMs to synthesize scientific hypothesis questions and corresponding discovery answers; and (3) QA filter to ensure quality based on relevance, clarity, and centrality. We instantiate this pipeline to construct a monthly-updated benchmark covering 12 biomedical sub-domains. Extensive evaluations of SOTA models reveal current limitations in discovering new knowledge. Our work provides the first dynamic, automatic framework for assessing the new knowledge discovery capabilities of AI systems, establishing a living, evolving resource for AI research community to catalyze the development of knowledge discovery.

Related papers

• EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge [53.44952808455985]
  EvoBrowseComp is an evolving benchmark of 400 English and 400 Chinese complex questions.<n>EvoBrowseComp can be regularly updated to prevent data contamination.<n>It establishes a scalable paradigm for auto-updatable, high-difficulty benchmarking.
  arXiv  Detail & Related papers  (2026-06-11T09:48:32Z)
• Revisiting Gene Ontology Knowledge Discovery with Hierarchical Feature Selection and Virtual Study Group of AI Agents [0.0]
  We propose a novel agentic AI-based knowledge discovery-oriented virtual study group.<n>It aims to extract meaningful ageing-related biological knowledge considering highly ageing-related Gene Ontology terms.<n>It is found that the majority of the AI agent-generated scientific claims can be supported by existing literatures.
  arXiv  Detail & Related papers  (2026-03-20T17:00:36Z)
• HealthFlow: A Self-Evolving AI Agent with Meta Planning for Autonomous Healthcare Research [32.21457361323802]
  This paper introduces HealthFlow, a self-evolving AI agent that overcomes limitations through a novel meta-level evolution mechanism.<n>HealthFlow autonomously refines its high-level problem-solving policies by distilling procedural successes and failures into a durable, structured knowledge base.<n>Our experiments demonstrate that HealthFlow's self-evolving approach significantly outperforms state-of-the-art agent frameworks.
  arXiv  Detail & Related papers  (2025-08-04T17:08:47Z)
• Contemporary AI foundation models increase biological weapons risk [0.0]
  We argue current foundation AI models underestimate the risk of facilitating biological weapons development.<n>We identify "elements of success" for biological weapons development that large language models can describe in words.<n>We advocate for improved benchmarks, while acknowledging the window for meaningful implementation may have already closed.
  arXiv  Detail & Related papers  (2025-06-12T19:53:38Z)
• AutoMind: Adaptive Knowledgeable Agent for Automated Data Science [70.33796196103499]
  Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems.<n>Existing frameworks depend on rigid, pre-defined and inflexible coding strategies.<n>We introduce AutoMind, an adaptive, knowledgeable LLM-agent framework.
  arXiv  Detail & Related papers  (2025-06-12T17:59:32Z)
• Benchmarking AI scientists in omics data-driven biological research [3.3605177939410713]
  We introduce the Biological AI Scientist Benchmark (BaisBench) to assess AI scientists' ability to generate biological discoveries.<n>BaisBench comprises two tasks: cell type annotation on 31 expert-labeled single-cell datasets, and scientific discovery through answering 198 multiple-choice questions.
  arXiv  Detail & Related papers  (2025-05-13T08:33:54Z)
• Information Retrieval in the Age of Generative AI: The RGB Model [77.96475639967431]
  This paper presents a novel quantitative approach to shed light on the complex information dynamics arising from the growing use of generative AI tools.<n>We propose a model to characterize the generation, indexing, and dissemination of information in response to new topics.<n>Our findings suggest that the rapid pace of generative AI adoption, combined with increasing user reliance, can outpace human verification, escalating the risk of inaccurate information proliferation.
  arXiv  Detail & Related papers  (2025-04-29T10:21:40Z)
• BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology [4.099098082010236]
  Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research.<n>We present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis.<n>We evaluate the performance of two frontier LLMs using a custom agent framework we open source.
  arXiv  Detail & Related papers  (2025-02-28T18:47:57Z)
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models [71.25225058845324]
  Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation. Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge. RA-LLMs have emerged to harness external and authoritative knowledge bases, rather than relying on the model's internal knowledge.
  arXiv  Detail & Related papers  (2024-05-10T02:48:45Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.