Ontology-Guided Multi-Agent Extraction of Evaluation Objects from Academic Review Texts: Evidence from Chinese Library and Information Science

• URL: http://arxiv.org/abs/2608.29526v1
• Date: Sun, 30 Aug 2026 03:09:47 GMT
• Title: Ontology-Guided Multi-Agent Extraction of Evaluation Objects from Academic Review Texts: Evidence from Chinese Library and Information Science
• Abstract summary: This study proposes an ontology-guided multi-agent framework for evaluation object extraction.<n>It achieves a Precision of 90.33%, Recall of 84.55%, Entity-level F1 of 87.34%, Strict Typed F1 of 79.78%, and Type Accuracy of 91.35%.
• Score: 6.306930443447107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Academic reviews, scholarly commentaries, and book reviews serve as sources of evaluative statements about theories, methods, literature, institutions, and policies, providing valuable evidence for scholarly evaluation. Existing scientific entity extraction methods mainly target research articles and are less effective for evaluation objects, which are often abstract, context-dependent, and characterized by ambiguous type boundaries. This study proposes an ontology-guided multi-agent framework for evaluation object extraction. The framework combines candidate discovery, ontology-constrained classification, and domain review. Experimental results show that it achieves a Precision of 90.33%, Recall of 84.55%, Entity-level F1 of 87.34%, Strict Typed F1 of 79.78%, and Type Accuracy of 91.35%, substantially outperforming rule-based and zero-shot baselines. Ablation results indicate that the multi-agent workflow improves recall and stability, while ontology-based boundary constraints enhance fine-grained classification and reduce category confusion. The framework supports the structured utilization of evaluative scholarly texts and provides methodological support for evidence-based research evaluation and STI mining.

Related papers

• HexEval: An Evidence-Driven Hexagonal Framework for Multidimensional Scholar Assessment [4.103608919706051]
  HexEval is an evidence-driven hexagonal framework for multidimensional scholar assessment.<n>We show that HexEval preserves intermediate evidence, dimension-specific rationales, and verification signals throughout the evaluation process.
  arXiv  Detail & Related papers  (2026-08-11T07:11:49Z)
• From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent [52.42081442204093]
  We argue that a key limitation is the lack of flexibility to proactively investigate suspicious parts of a paper.<n>We propose ProReviewer, a scientific peer review agent that proactively reviews a paper guided by a maintained, structured review log.<n> Experiments show that ProReviewer with an 8B backbone, trained by supervised fine-tuning and optimized by reinforcement learning, achieves the highest average score across five quality dimensions.
  arXiv  Detail & Related papers  (2026-06-11T13:38:23Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• Rethinking Artifact Evaluation for Software Engineering in the Age of Generative AI [20.515496306312873]
  We argue that artifact evaluation should be treated as a first-class component of peer review.<n>We frame peer review as an attention allocation problem, examine how generative AI weakens narrative quality as a signal of rigor, and argue that artifact evaluation should play a more prominent role in peer review decisions.
  arXiv  Detail & Related papers  (2026-01-26T09:36:52Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark [5.549390587677274]
  We present a benchmark for measuring the methodological strength of biomedical papers.<n>We draw on the risk-of-bias framework used for systematic reviews.<n>Our analyses show that large language models' reasoning and retrieval capabilities impact their effectiveness with risk-of-bias assessment.
  arXiv  Detail & Related papers  (2024-11-28T00:21:31Z)
• STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond [68.47402386668846]
  We introduce Structured Reasoning In Critical Text Assessment (STRICTA) to model text assessment as an explicit, step-wise reasoning process.<n>STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory.<n>We apply STRICTA to a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers.
  arXiv  Detail & Related papers  (2024-09-09T06:55:37Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.