Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

• URL: http://arxiv.org/abs/2608.25826v1
• Date: Wed, 26 Aug 2026 14:06:29 GMT
• Title: Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
• Abstract summary: We present a pipeline that unfolds each paper into a multi-turn generation trajectory.<n>All section texts and the abstract are kept verbatim from the source paper.<n>Treating real paper text as the answer yields an SFT dataset.
• Score: 45.97752484594236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leaves the structure of whole documents untouched. Scientific papers are written to a clear and largely uniform structure and make a natural substrate for lifting this paradigm to the document level. We present a pipeline that unfolds each paper into a multi-turn generation trajectory in which a teacher model reconstructs the writing process of the whole paper: a writing request, a global plan, and pre-writing deliberation for each section. All section texts and the abstract are kept verbatim from the source paper. We apply the pipeline to quality-filtered arXiv papers and obtain a corpus for continued pre-training (CPT) that is roughly twice the size of the source text. The same reverse construction extends to instruction data and evaluation. Treating real paper text as the answer yields an SFT dataset. Anchoring tasks in held-out papers yields PAW-Bench, an academic-writing benchmark whose tasks carry their own rubrics and checklists. In controlled experiments CPT on our corpus followed by supervised fine-tuning on public datasets improves writing benchmarks broadly while preserving general reasoning and improving long-document reading. The writing gain persists even when every model is fine-tuned on a dedicated writing SFT dataset. Mixing our SFT data into that recipe lifts academic writing further.

Related papers

• Code as Representation: A Compilable Parsing Paradigm for Academic Documents [50.40789738182074]
  Compilable Academic Document Parsing (CADP) is a paradigm that reconstructs a full page as contextual La plus executable Python.<n>We introduce CADP-Bench, an expert-verified benchmark of full academic pages containing text and multiple SAE types.<n>Results show that even frontier models still struggle to produce high-fidelity executable reconstructions.
  arXiv  Detail & Related papers  (2026-08-18T09:10:43Z)
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training [12.81568269791334]
  We study whether related content is organized into coherent book-level documents.<n>We retrieve source material from a pre-training corpus, clusters it into topical units, plans hierarchical tables of contents, and assembles source-grounded sections into complete books.<n> Replacing natural books in a mid-training mix with this corpus improves downstream performance by +1.09 on average.
  arXiv  Detail & Related papers  (2026-07-30T12:15:41Z)
• LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation [57.70907667652966]
  Introduction writing is especially challenging, which demands logical soundness and verifiable faithfulness.<n>Most AI-assisted methods treat the task as text generation instead of reasoning and structuring, leading to severe drawbacks.<n>We propose LECTOR, a novel Logic-Expression Co-Reinforcement Learning framework that can strictly follow the scientist's logic, add high-quality citations and keep structured expressions.
  arXiv  Detail & Related papers  (2026-05-25T15:41:16Z)
• sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing [0.0]
  Science currently offers two options for quality assurance, both inadequate.<n>We propose a third option: measure the paper itself.<n> sciwrite-lint is an open-source linter for scientific manuscripts that runs entirely on the researcher's machine.<n>We evaluate the pipeline on 30 unseen papers from arXiv and bioRxiv with error injection and LLM-adjudicated false positive analysis.
  arXiv  Detail & Related papers  (2026-04-09T17:46:44Z)
• EarlySciRev: A Dataset of Early-Stage Scientific Revisions Extracted from LaTeX Writing Traces [5.062670468960367]
  We introduce EarlySciRev, a dataset of early-stage scientific text revisions automatically extracted from arXiv source files.<n>Our key observation is that commented-out text in often preserves discarded or alternative formulations written by the authors themselves.<n>Our pipeline yields 578k validated revision pairs, grounded in authentic early drafting traces.
  arXiv  Detail & Related papers  (2026-03-30T14:47:04Z)
• Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training [23.87978106727431]
  We propose a data-training co-design framework for robust end-to-end document parsing.<n>Our method achieves superior accuracy and robustness across both scanned/digital and real-world captured scenarios.
  arXiv  Detail & Related papers  (2026-03-25T03:19:09Z)
• Training-Free Acceleration for Document Parsing Vision-Language Model with Hierarchical Speculative Decoding [102.88996030431662]
  We propose a training-free and highly efficient acceleration method for document parsing tasks.<n>Inspired by speculative decoding, we employ a lightweight document parsing pipeline as a draft model to predict batches of future tokens.<n>We demonstrate the effectiveness of our approach on the general-purpose OmniDocBench.
  arXiv  Detail & Related papers  (2026-02-13T14:22:10Z)
• Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering [51.7493726399073]
  We present a discourse-aware hierarchical framework to enhance long document question answering.<n>The framework involves three key innovations: specialized discourse parsing for lengthy documents, LLM-based enhancement of discourse relation nodes, and structure-guided hierarchical retrieval.
  arXiv  Detail & Related papers  (2025-05-26T14:45:12Z)
• CiteBench: A benchmark for Scientific Citation Text Generation [69.37571393032026]
  CiteBench is a benchmark for citation text generation. We make the code for CiteBench publicly available at https://github.com/UKPLab/citebench.
  arXiv  Detail & Related papers  (2022-12-19T16:10:56Z)
• Automatic Document Sketching: Generating Drafts from Analogous Texts [44.626645471195495]
  We introduce a new task, document sketching, which involves generating entire draft documents for the writer to review and revise. These drafts are built from sets of documents that overlap in form - sharing large segments of potentially reusable text - while diverging in content. We investigate the application of weakly supervised methods, including use of a transformer-based mixture of experts, together with reinforcement learning.
  arXiv  Detail & Related papers  (2021-06-14T06:46:06Z)
• Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation [50.01708049531156]
  We focus on a new practical task, document-scale text content manipulation, which is the opposite of text style transfer. In detail, the input is a set of structured records and a reference text for describing another recordset. The output is a summary that accurately describes the partial content in the source recordset with the same writing style of the reference.
  arXiv  Detail & Related papers  (2020-02-24T12:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.