Building Agent Harnesses for Scientific Curation from Multimodal Sources

• URL: http://arxiv.org/abs/2606.21005v1
• Date: Fri, 19 Jun 2026 00:36:34 GMT
• Title: Building Agent Harnesses for Scientific Curation from Multimodal Sources
• Abstract summary: We study scientific curation from multimodal sources and introduce Beaver, an agent harness that extracts structured information from scientific papers.<n>We show that Beaver reaches 81.0 on Gold-Referenced Attribute Score (GRAS), an attribute-level measure of agreement with gold curated records, outperforming frontier agents by over 23 absolute points.
• Score: 17.12384426105798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific discovery workflows often depend on structured curation from the literature. This is difficult for current agents because the key evidence is scattered across long text, dense tables, and figures, and the final records often require reasoning across multiple evidence fragments rather than copying a single span. We study scientific curation from multimodal sources and introduce Beaver, an agent harness that extracts structured information from scientific papers while preserving provenance to the supporting evidence. Beaver combines a frontier agent with multimodal evidence tooling, task scaffolding, and artifact-grounded autoresearch. These components turn curation into a staged, auditable workflow and enable an iterative evaluate--diagnose--revise loop, where persistent run artifacts expose stage-localized failures and guide harness updates. Experiments show that Beaver reaches 81.0 on Gold-Referenced Attribute Score (GRAS), an attribute-level measure of agreement with gold curated records, outperforming frontier agents by over 23 absolute points. Ablations show that task scaffolding, multimodal evidence tooling, and provenance traces each contribute meaningfully to performance, while attribute-level analysis shows the largest gains on high-value attributes that require cross-modal reasoning and normalization. These results show that, for scientific curation from papers with multimodal evidence, harness design is a central determinant of agent performance.

Related papers

• EviGraph: Evidence-Guided Autonomous Research Agents [12.29802995923839]
  We introduce EviGraph, an autonomous research framework that represents the research process as a typed evidence graph.<n>EviGraph inspects evidence chains for missing dependencies, semantic misalignment, and result-claim inconsistencies.<n>It localizes the earliest weak node, and regenerates its affected downstream subgraph.
  arXiv  Detail & Related papers  (2026-08-05T12:02:55Z)
• Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents [63.2133766267587]
  Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence.<n>We study these questions through a trajectory-level diagnosis of long-horizon search agents.<n>We evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence.
  arXiv  Detail & Related papers  (2026-08-03T08:46:48Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments [51.049999642138]
  We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence.<n>Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly.<n>Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops.
  arXiv  Detail & Related papers  (2026-07-19T15:20:40Z)
• WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning [63.9071746889252]
  Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages.<n>This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it.<n>We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources.
  arXiv  Detail & Related papers  (2026-07-10T12:09:21Z)
• TVIR: Building Deep Research Agents Towards Text--Visual Interleaved Report Generation [63.89818340842003]
  TVIR (Text--Visual Interleaved Report Generation) is a benchmark of 100 expert-curated multimodal deep research tasks.<n>TVIR-Agent is a hierarchical multi-agent framework that serves as a strong baseline for constructing outlines and retrieving images.<n>We develop a dual-path evaluation framework that combines Textual Assessment and Visual Assessment.
  arXiv  Detail & Related papers  (2026-06-01T14:35:14Z)
• Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation [74.0621258662676]
  We propose Ptah, a multi-agent harness for interleaved report generation.<n>Ptah orchestrates the lifecycle from user query to rendered web report through planning, research, and writing stages.<n>A verifier agent serves as the harness's acceptance function, enforcing factual grounding, citation fidelity, and cross-modal consistency throughout the workflow.
  arXiv  Detail & Related papers  (2026-05-28T12:40:34Z)
• The Last Human-Written Paper: Agent-Native Research Artifacts [106.47848184955576]
  We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package.<n>Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste.
  arXiv  Detail & Related papers  (2026-04-27T16:23:09Z)
• EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents [38.845762856105175]
  We introduce EpiBench, an episodic multi-turn benchmark that instantiates short research.<n>Given a research task, agents must navigate across papers over multiple turns, align evidence from figures and tables, and use the accumulated evidence in the memory to answer objective questions.<n>Our experiments show that even the leading model achieves an accuracy of only 29.23% on the hard split, indicating substantial room for improvement.
  arXiv  Detail & Related papers  (2026-04-07T07:58:55Z)
• FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases [10.00386797940562]
  We present FlyBench to evaluate AI agents on end-to-end agentic curation from scientific literature.<n>Given only a gene symbol, agents must search and read from a corpus of 16,898 full-text papers to produce structured annotations.<n>The benchmark includes 7,397 expert-curated annotations across 100 genes drawn from FlyBase.
  arXiv  Detail & Related papers  (2026-02-09T20:12:38Z)
• FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks with File-System-Based Agents [53.03492387564392]
  We introduce FS-Researcher, a file-system-based framework that scales deep research beyond the context window via a persistent workspace.<n>A Context Builder agent browses the internet, writes structured notes, and archives raw sources into a hierarchical knowledge base that can grow far beyond context length.<n>A Report Writer agent then composes the final report section by section, treating the knowledge base as the source of facts.
  arXiv  Detail & Related papers  (2026-02-02T03:00:19Z)
• Multimodal Fact-Checking: An Agent-based Approach [9.55806677152407]
  We introduce RW-Post, a high-quality and explainable dataset for real-world multimodal fact-checking.<n> RW-Post aligns real-world multimodal claims with their original social media posts, preserving the rich contextual information in which the claims are made.<n>Building upon RW-Post, we propose AgentFact, an agent-based multimodal fact-checking framework designed to emulate the human verification workflow.
  arXiv  Detail & Related papers  (2025-12-28T13:58:33Z)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.