RWGBench: Evaluating Scholarly Positioning in Related Work Generation

• URL: http://arxiv.org/abs/2606.24894v1
• Date: Sat, 30 May 2026 16:53:14 GMT
• Title: RWGBench: Evaluating Scholarly Positioning in Related Work Generation
• Abstract summary: We introduce textbfRWGBench, a benchmark that evaluates RWG from the perspective of citation decision-making rather than text similarity.<n>RWGBench is constructed from a large-scale collection of 40,108 computer science papers and a retrieval corpus of 1.09 million documents.<n>Experiments reveal systematic limitations in current systems that are obscured by standard evaluations, while Oracle studies further disentangle retrieval-level and generation-level bottlenecks.
• Score: 43.26813089097539
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models have shown strong fluency in scientific writing, yet the evaluation of related work generation (RWG) remains limited. Existing RWG evaluations largely inherit summarization-oriented metrics, using lexical or semantic similarity to reference sections as proxies for quality. However, related work writing is fundamentally a citation-level scholarly positioning task: it requires selecting, organizing, and framing prior work to clarify how a target paper relates to, differs from, and contributes beyond existing research.As a result, models may generate coherent and semantically-relevant text while exhibiting academically critical failures, such as inappropriate citation selection or misplaced references, that conventional metrics do not capture.To this end, we introduce \textbf{RWGBench}, a benchmark that evaluates RWG from the perspective of citation decision-making rather than text similarity. RWGBench is constructed from a large-scale collection of 40,108 computer science papers and a retrieval corpus of 1.09 million documents, with a carefully curated test set comprising 100 papers and their corresponding published related work sections.We propose a multi-dimensional evaluation framework that assesses citation selection, contextual appropriateness, organization, and discourse structure.Experiments reveal systematic limitations in current systems that are obscured by standard evaluations, while Oracle studies further disentangle retrieval-level and generation-level bottlenecks. Human evaluation further shows that our citation-centric metrics align substantially better with expert judgment than surface-level text metrics. RWGBench offers a citation-centric testbed for developing and evaluating related work generation systems that are better aligned with scholarly writing practices.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents [7.397429318828859]
  Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated.<n>Existing benchmarks focus on domain knowledge and question answering, largely overlooking quality review for professional documents.<n>We introduce GB/T-Bench, the first benchmark for the structured review of national standard documents.
  arXiv  Detail & Related papers  (2026-08-06T17:27:23Z)
• Reading Between the Citations: A Typed Claim Network for Scientific Literature [6.6267663941512005]
  We give a construction pipeline applicable to any corpus of scholarly inter-referencing documents.<n>We instantiate it on a corpus of 127 papers in 3D point cloud semantic segmentation, producing a network of 8,260 typed claims.<n>Head-to-head evaluation against standard Retrieval-Augmented Generation (RAG) baselines shows that the gain over flat retrieval is the gain from the right intermediate representation rather than the wrong one.
  arXiv  Detail & Related papers  (2026-05-29T08:02:45Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation [2.985620880452744]
  SciSage is a multi-agent framework employing a reflect-when-you-write paradigm.<n>It critically evaluates drafts at outline, section, and document levels, collaborating with specialized agents for query interpretation, content retrieval, and refinement.<n>We also release SurveyScope, a benchmark of 46 high-impact papers ( 2020-2025) across 11 computer science domains.
  arXiv  Detail & Related papers  (2025-06-15T02:23:47Z)
• XtraGPT: Context-Aware and Controllable Academic Paper Revision [43.263488839387584]
  We propose a human-AI collaboration framework for academic paper revision centered on criteria-guided intent alignment and context-aware modeling.<n>We instantiate the framework in XtraGPT, the first suite of open-source LLMs for context-aware, instruction-guided writing assistance.
  arXiv  Detail & Related papers  (2025-05-16T15:02:19Z)
• PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models [72.57329554067195]
  ProxyQA is an innovative framework dedicated to assessing longtext generation. It comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. It assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions.
  arXiv  Detail & Related papers  (2024-01-26T18:12:25Z)
• Quantitative Discourse Cohesion Analysis of Scientific Scholarly Texts using Multilayer Networks [10.556468838821338]
  We aim to computationally analyze the discourse cohesion in scientific scholarly texts using multilayer network representation. We design section-level and document-level metrics to assess the extent of lexical cohesion in text. We present an analytical framework, CHIAA (CHeck It Again, Author), to provide pointers to the author for potential improvements in the manuscript.
  arXiv  Detail & Related papers  (2022-05-16T09:10:41Z)
• Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
  We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document. We also simultaneously cluster users, removing the need for post-hoc cluster estimation. Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
  arXiv  Detail & Related papers  (2021-06-15T20:55:55Z)
• SPECTER: Document-level Representation Learning using Citation-informed Transformers [51.048515757909215]
  SPECTER generates document-level embedding of scientific documents based on pretraining a Transformer language model. We introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction to document classification and recommendation.
  arXiv  Detail & Related papers  (2020-04-15T16:05:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.