SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era

• URL: http://arxiv.org/abs/2603.16131v1
• Date: Tue, 17 Mar 2026 05:34:46 GMT
• Title: SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era
• Abstract summary: SciZoom is a benchmark comprising 44,946 papers from four top-tier ML venues spanning 2020 to 2025.<n>Our linguistic analysis reveals striking shifts in phrase patterns (up to 10x for formulaic expressions) and rhetorical style (23% decline in hedging)<n>SciZoom serves as both a challenging benchmark and a unique resource for mining the evolution of scientific discourse in the generative AI era.
• Score: 2.2090506971647144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The explosive growth of AI research has created unprecedented information overload, increasing the demand for scientific summarization at multiple levels of granularity beyond traditional abstracts. While LLMs are increasingly adopted for summarization, existing benchmarks remain limited in scale, target only a single granularity, and predate the LLM era. Moreover, since the release of ChatGPT in November 2022, researchers have rapidly adopted LLMs for drafting manuscripts themselves, fundamentally transforming scientific writing, yet no resource exists to analyze how this writing has evolved. To bridge these gaps, we introduce SciZoom, a benchmark comprising 44,946 papers from four top-tier ML venues (NeurIPS, ICLR, ICML, EMNLP) spanning 2020 to 2025, explicitly stratified into Pre-LLM and Post-LLM eras. SciZoom provides three hierarchical summarization targets (Abstract, Contributions, and TL;DR) achieving compression ratios up to 600:1, enabling both multi-granularity summarization research and temporal mining of scientific writing patterns. Our linguistic analysis reveals striking shifts in phrase patterns (up to 10x for formulaic expressions) and rhetorical style (23% decline in hedging), suggesting that LLM-assisted writing produces more confident yet homogenized prose. SciZoom serves as both a challenging benchmark and a unique resource for mining the evolution of scientific discourse in the generative AI era. Our code and dataset are publicly available on GitHub (https://github.com/janghana/SciZoom) and Hugging Face (https://huggingface.co/datasets/hanjang/SciZoom), respectively.

Related papers

• Paper Espresso: From Paper Overload to Research Insight [53.45135239997754]
  We present Paper Espresso, an open-source platform that automatically discovers, summarizes, and analyzes trending arXiv papers.<n>The system uses large language models (LLMs) to generate structured summaries with topical labels and keywords.<n>Over 35 months of continuous deployment, Paper Espresso has processed over 13,300 papers and publicly released all structured metadata.
  arXiv  Detail & Related papers  (2026-04-06T09:45:21Z)
• Not too long do read: Evaluating LLM-generated extreme scientific summaries [0.0]
  We propose a novel dataset, BiomedTLDR, containing a large sample of researcher-authored summaries from scientific papers.<n>We then test popular open-weight LLMs for generating extreme summaries based on abstracts.<n>Our analysis reveals that, although some of them successfully produce humanoid summaries, LLMs generally exhibit a greater affinity for the original text's lexical choices and rhetorical structures.
  arXiv  Detail & Related papers  (2025-12-29T05:03:02Z)
• Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprints [15.070885964897734]
  Generative large language models (LLMs) introduce a further potential disruption by altering how manuscripts are written.<n>This paper addresses the gap through a large-scale analysis of more than 2.1 million preprints spanning 2016--2025 (115 months) across four major repositories.<n>Our findings reveal that LLMs have accelerated submission and revision cycles, modestly increased linguistic complexity, and disproportionately expanded AI-related topics.
  arXiv  Detail & Related papers  (2025-10-18T01:37:40Z)
• Let's Use ChatGPT To Write Our Paper! Benchmarking LLMs To Write the Introduction of a Research Paper [64.50822834679101]
  SciIG is a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works.<n>We assess five state-of-the-art models, including open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems.<n>Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness.
  arXiv  Detail & Related papers  (2025-08-19T21:11:11Z)
• LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models [15.588997848081046]
  This survey addresses the gap by presenting a data-driven, semi-automated review of research on limitations of large language models (LLLMs) from 2022 to 2025.<n>From a corpus of 250,000 ACL and arXiv papers, we extract 14,648 relevant limitation papers using keyword filtering and LLM-based classification.<n>We find that LLM-related research increases nearly sixfold in ACL and nearly fifteenfold in arXiv between 2022 and 2025.
  arXiv  Detail & Related papers  (2025-05-25T17:38:32Z)
• Integrating Planning into Single-Turn Long-Form Text Generation [66.08871753377055]
  We propose to use planning to generate long form content. Our main novelty lies in a single auxiliary task that does not require multiple rounds of prompting or planning. Our experiments demonstrate on two datasets from different domains, that LLMs fine-tuned with the auxiliary task generate higher quality documents.
  arXiv  Detail & Related papers  (2024-10-08T17:02:40Z)
• One Thousand and One Pairs: A "novel" challenge for long-context language models [56.60667988954638]
  NoCha is a dataset of 1,001 pairs of true and false claims about 67 fictional books. Our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. On average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning.
  arXiv  Detail & Related papers  (2024-06-24T02:03:57Z)
• Attribution in Scientific Literature: New Benchmark and Methods [41.64918533152914]
  Large language models (LLMs) present a promising yet challenging frontier for automated source citation in scientific communication.<n>We introduce REASONS, a novel dataset with sentence-level annotations across 12 scientific domains from arXiv.<n>We conduct extensive experiments with models such as GPT-O1, GPT-4O, GPT-3.5, DeepSeek, and other smaller models like Perplexity AI (7B)
  arXiv  Detail & Related papers  (2024-05-03T16:38:51Z)
• Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
  We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
  arXiv  Detail & Related papers  (2024-04-01T17:45:15Z)
• Large Language Models: A Survey [66.39828929831017]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks.<n>LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.