A Reproducible, Scalable Pipeline for Synthesizing Autoregressive Model Literature

• URL: http://arxiv.org/abs/2508.04612v1
• Date: Wed, 06 Aug 2025 16:33:20 GMT
• Title: A Reproducible, Scalable Pipeline for Synthesizing Autoregressive Model Literature
• Authors: Faruk Alpay, Bugra Kilictas, Hamdi Alakkad,
• Abstract summary: Autoregressive generative models have produced thousands of papers, making manual literature surveys and reproduction studies increasingly impractical.<n>We present a fully open-source, reproducible pipeline that automatically retrieves candidate documents from public repositories.<n>Experiments on corpora of up to 1000 papers demonstrate near-linear scalability with eight CPU workers.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The accelerating pace of research on autoregressive generative models has produced thousands of papers, making manual literature surveys and reproduction studies increasingly impractical. We present a fully open-source, reproducible pipeline that automatically retrieves candidate documents from public repositories, filters them for relevance, extracts metadata, hyper-parameters and reported results, clusters topics, produces retrieval-augmented summaries and generates containerised scripts for re-running selected experiments. Quantitative evaluation on 50 manually-annotated papers shows F1 scores above 0.85 for relevance classification, hyper-parameter extraction and citation identification. Experiments on corpora of up to 1000 papers demonstrate near-linear scalability with eight CPU workers. Three case studies -- AWD-LSTM on WikiText-2, Transformer-XL on WikiText-103 and an autoregressive music model on the Lakh MIDI dataset -- confirm that the extracted settings support faithful reproduction, achieving test perplexities within 1--3% of the original reports.

Related papers

• Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models [53.17363502535395]
  Trustworthy language models should provide both correct and verifiable answers.<n>Current systems insert citations by querying an external retriever at inference time.<n>We propose Active Indexing, which continually pretrains on synthetic QA pairs.
  arXiv  Detail & Related papers  (2025-06-21T04:48:05Z)
• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs [54.5729817345543]
  MOLE is a framework that automatically extracts metadata attributes from scientific papers covering datasets of languages other than Arabic.<n>Our methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output.
  arXiv  Detail & Related papers  (2025-05-26T10:31:26Z)
• Enhanced Multi-Tuple Extraction for Alloys: Integrating Pointer Networks and Augmented Attention [6.938202451113495]
  We present a novel framework that combines an extraction model based on MatSciBERT with pointer and an allocation model.<n>Our experiments on extraction demonstrate impressive F1 scores of 0.947, 0.93 and 0.753 across datasets.<n>These results highlight the model's capacity to deliver precise and structured information.
  arXiv  Detail & Related papers  (2025-03-10T02:39:06Z)
• BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression [91.23933111083389]
  Retrieval-augmented generation (RAG) can supplement large language models (LLMs) by integrating external knowledge.<n>This paper presents BRIEF, a lightweight approach that performs query-aware multi-hop reasoning.<n>Based on our synthetic data built entirely by open-source models, BRIEF generates more concise summaries.
  arXiv  Detail & Related papers  (2024-10-20T04:24:16Z)
• REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Models [14.023953508288628]
  Retrieval augmented generation (RAG) pipelines are commonly used in tasks such as question-answering (QA) We propose REFINE, a novel technique that generates synthetic data from available documents and then uses a model fusion approach to fine-tune embeddings.
  arXiv  Detail & Related papers  (2024-10-16T08:43:39Z)
• Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilities [2.7311827519141363]
  We propose Text to Topic (Text2Topic), which achieves high multi-label classification performance. Text2Topic supports zero-shot predictions, produces domain-specific text embeddings, and enables production-scale batch-inference. The model is deployed on a real-world stream processing platform, and it outperforms other models with 92.9% micro mAP.
  arXiv  Detail & Related papers  (2023-10-23T11:33:24Z)
• How Does Generative Retrieval Scale to Millions of Passages? [68.98628807288972]
  We conduct the first empirical study of generative retrieval techniques across various corpus scales. We scale generative retrieval to millions of passages with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. While generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge.
  arXiv  Detail & Related papers  (2023-05-19T17:33:38Z)
• BeCAPTCHA-Type: Biometric Keystroke Data Generation for Improved Bot Detection [63.447493500066045]
  This work proposes a data driven learning model for the synthesis of keystroke biometric data. The proposed method is compared with two statistical approaches based on Universal and User-dependent models. Our experimental framework considers a dataset with 136 million keystroke events from 168 thousand subjects.
  arXiv  Detail & Related papers  (2022-07-27T09:26:15Z)
• Learning Summary Statistics for Bayesian Inference with Autoencoders [58.720142291102135]
  We use the inner dimension of deep neural network based Autoencoders as summary statistics. To create an incentive for the encoder to encode all the parameter-related information but not the noise, we give the decoder access to explicit or implicit information that has been used to generate the training data.
  arXiv  Detail & Related papers  (2022-01-28T12:00:31Z)
• The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild [0.0]
  The dataset consists of various challenges related to web-based data, such as machine translated content, encoding errors, multiple contents presented in one document etc. The initial machine learning experiments on the dataset show that (1) pre-Transformer models are drastically less able to model the phenomena, with macro F1 metrics ranging around 0.22, while Transformer-based models achieve scores of around 0.58, and (2) multilingual Transformer models work as well on the task as the monolingual models that were previously proven to be superior to multilingual models on standard NLP tasks.
  arXiv  Detail & Related papers  (2022-01-11T09:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.