Measuring the State of Open Science in Transportation Using Large Language Models

• URL: http://arxiv.org/abs/2601.14429v1
• Date: Tue, 20 Jan 2026 19:39:52 GMT
• Title: Measuring the State of Open Science in Transportation Using Large Language Models
• Authors: Junyi Ji, Ruth Lu, Linda Belkessa, Liming Wang, Silvia Varotto, Yongqi Dong, Nicolas Saunier, Mostafa Ameli, Gregory S. Macfarlane, Bahman Madadi, Cathy Wu,
• Abstract summary: Open science initiatives have strengthened scientific integrity and accelerated research progress across many fields.<n>Key features of open science, defined here as data and code availability, are difficult to extract due to the inherent complexity of the field.<n>This paper introduces an automatic and scalable feature-extraction pipeline to measure data and code availability in transportation research.
• Score: 8.915048816245394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open science initiatives have strengthened scientific integrity and accelerated research progress across many fields, but the state of their practice within transportation research remains under-investigated. Key features of open science, defined here as data and code availability, are difficult to extract due to the inherent complexity of the field. Previous work has either been limited to small-scale studies due to the labor-intensive nature of manual analysis or has relied on large-scale bibliometric approaches that sacrifice contextual richness. This paper introduces an automatic and scalable feature-extraction pipeline to measure data and code availability in transportation research. We employ Large Language Models (LLMs) for this task and validate their performance against a manually curated dataset and through an inter-rater agreement analysis. We applied this pipeline to examine 10,724 research articles published in the Transportation Research Part series of journals between 2019 and 2024. Our analysis found that only 5% of quantitative papers shared a code repository, 4% of quantitative papers shared a data repository, and about 3% of papers shared both, with trends differing across journals, topics, and geographic regions. We found no significant difference in citation counts or review duration between papers that provided data and code and those that did not, suggesting a misalignment between open science efforts and traditional academic metrics. Consequently, encouraging these practices will likely require structural interventions from journals and funding agencies to supplement the lack of direct author incentives. The pipeline developed in this study can be readily scaled to other journals, representing a critical step toward the automated measurement and monitoring of open science practices in transportation research.

Related papers

• Analyzing Political Text at Scale with Online Tensor LDA [53.16930342547758]
  This paper proposes a topic modeling method that scales linearly to billions of documents.<n>We show that this method is computationally and memory efficient (achieving speeds over 3-4x those of prior parallelized Latent Dirichlet Allocation (LDA) methods)<n>We perform two real-world, large-scale new studies of interest to political scientists.
  arXiv  Detail & Related papers  (2025-11-11T03:58:48Z)
• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery [68.48094108571432]
  Large language models (LLMs) have revolutionized the way text and other modalities of data are handled. We aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs.
  arXiv  Detail & Related papers  (2024-06-16T08:03:24Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [97.31347312130119]
  SciRIFF (Scientific Resource for Instruction-Following and Finetuning) is a dataset of 137K instruction-following instances for training and evaluation, covering 54 tasks.<n>These tasks span five core scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification.<n> SciRIFF is unique in being entirely expert-written, high-quality instruction-following dataset for extracting and synthesizing information from research literature across diverse scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows [58.56005277371235]
  We introduce MASSW, a comprehensive text dataset on Multi-Aspect Summarization of ScientificAspects. MASSW includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years. We demonstrate the utility of MASSW through multiple novel machine-learning tasks that can be benchmarked using this new dataset.
  arXiv  Detail & Related papers  (2024-06-10T15:19:09Z)
• Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
  We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
  arXiv  Detail & Related papers  (2024-04-01T17:45:15Z)
• The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics [0.0]
  We introduce the new comprehensive Open Review-Based dataset (ORB) It includes a curated list of more than 36,000 scientific papers with their more than 89,000 reviews and final decisions. This paper presents our data architecture and an overview of the collected data along with relevant statistics.
  arXiv  Detail & Related papers  (2023-11-29T20:52:02Z)
• All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction [39.05577374775964]
  We propose a semi-supervised pipeline for annotating entities in text, as well as entities and relations in tables, in an iterative procedure. We release novel resources for the scientific community, including a high-quality benchmark, a large-scale corpus, and a semi-supervised annotation pipeline.
  arXiv  Detail & Related papers  (2023-11-14T14:22:47Z)
• Research Topic Flows in Co-Authorship Networks [0.0]
  We propose a graph structure for the analysis of research topic flows between scientific authors and their respective research fields. Our method requires for the construction of a TFN solely a corpus of publications (i.e., author and abstract information) We demonstrate the utility of TFNs by applying our method to two comprehensive corpora of altogether 20 Mio. publications spanning more than 60 years of research in the fields computer science and mathematics.
  arXiv  Detail & Related papers  (2022-06-16T07:45:53Z)
• CitationIE: Leveraging the Citation Graph for Scientific Information Extraction [89.33938657493765]
  We use the citation graph of referential links between citing and cited papers. We observe a sizable improvement in end-to-end information extraction over the state-of-the-art.
  arXiv  Detail & Related papers  (2021-06-03T03:00:12Z)
• Evaluating the state-of-the-art in mapping research spaces: a Brazilian case study [0.0]
  Two recent works propose methods for creating research maps from scientists' publication records. We evaluate these models' ability to predict whether a given entity will enter a new field. We conduct a case study to showcase how these models can be used to characterize science dynamics in the context of Brazil.
  arXiv  Detail & Related papers  (2021-04-07T18:14:41Z)
• TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics [32.4845534482475]
  We present a new corpus that contains domain expert annotations for Task (T), dataset (D), Metric (M) entities on 2,000 sentences extracted from NLP papers. We report experiment results on TDM extraction using a simple data augmentation strategy and apply our tagger to around 30,000 NLP papers from the ACL.
  arXiv  Detail & Related papers  (2021-01-25T17:54:06Z)
• What's New? Summarizing Contributions in Scientific Literature [85.95906677964815]
  We introduce a new task of disentangled paper summarization, which seeks to generate separate summaries for the paper contributions and the context of the work. We extend the S2ORC corpus of academic articles by adding disentangled "contribution" and "context" reference labels. We propose a comprehensive automatic evaluation protocol which reports the relevance, novelty, and disentanglement of generated outputs.
  arXiv  Detail & Related papers  (2020-11-06T02:23:01Z)
• Topic Space Trajectories: A case study on machine learning literature [0.0]
  We present topic space trajectories, a structure that allows for the comprehensible tracking of research topics. We show the applicability of our approach on a publication corpus spanning 50 years of machine learning research from 32 publication venues. Our novel analysis method may be employed for paper classification, for the prediction of future research topics, and for the recommendation of fitting conferences and journals for submitting unpublished work.
  arXiv  Detail & Related papers  (2020-10-23T10:53:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.