Does the Market of Citations Reward Reproducible Work?

• URL: http://arxiv.org/abs/2204.03829v1
• Date: Fri, 8 Apr 2022 04:03:17 GMT
• Title: Does the Market of Citations Reward Reproducible Work?
• Abstract summary: We show that certain fields of study such as Medicine and Machine Learning (ML) do correlate reproducible works with more citations. We find that making code available and thoroughly referencing prior works appear to also positively correlate with increased citations.
• Score: 36.754180527471895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The field of bibliometrics, studying citations and behavior, is critical to the discussion of reproducibility. Citations are one of the primary incentive and reward systems for academic work, and so we desire to know if this incentive rewards reproducible work. Yet to the best of our knowledge, only one work has attempted to look at this combined space, concluding that non-reproducible work is more highly cited. We show that answering this question is more challenging than first proposed, and subtle issues can inhibit a robust conclusion. To make inferences with more robust behavior, we propose a hierarchical Bayesian model that incorporates the citation rate over time, rather than the total number of citations after a fixed amount of time. In doing so we show that, under current evidence the answer is more likely that certain fields of study such as Medicine and Machine Learning (ML) do correlate reproducible works with more citations, but other fields appear to have no relationship. Further, we find that making code available and thoroughly referencing prior works appear to also positively correlate with increased citations. Our code and data can be found at https://github.com/EdwardRaff/ReproducibleCitations .

Related papers

• Citation Failure: Definition, Analysis and Efficient Mitigation [56.09968229868067]
  Citations from LLM-based RAG systems are supposed to simplify response verification.<n>This does not hold for citation failure, when a model generates a helpful response, but fails to cite complete evidence.<n>We propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible.
  arXiv  Detail & Related papers  (2025-10-23T07:47:22Z)
• VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification [107.75781898355562]
  We introduce a novel framework, called VeriCite, designed to rigorously validate supporting evidence and enhance answer attribution.<n>We conduct experiments across five open-source LLMs and four datasets, demonstrating that VeriCite can significantly improve citation quality while maintaining the correctness of the answers.
  arXiv  Detail & Related papers  (2025-10-13T13:38:54Z)
• In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis [52.42612945266194]
  We propose a new task: generating nuanced, expressive, and time-aware impact summaries.<n>We show that these summaries capture both praise (confirmation citations) and critique (correction citations) through the evolution of fine-grained citation intents.
  arXiv  Detail & Related papers  (2025-05-20T19:11:06Z)
• The Noisy Path from Source to Citation: Measuring How Scholars Engage with Past Research [20.649638393774048]
  We introduce a computational pipeline to quantify citation fidelity at scale. Using full texts of papers, the pipeline identifies citations in citing papers and the corresponding claims in cited papers. Using a quasi-experiment, we establish the "telephone effect" - when citing papers have low fidelity to the original claim, future papers that cite the citing paper and the original have lower fidelity to the original.
  arXiv  Detail & Related papers  (2025-02-27T22:47:03Z)
• SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models [51.90867482317985]
  SelfCite is a self-supervised approach that aligns LLMs to generate high-quality, fine-grained, sentence-level citations for statements in generated responses. Instead of relying on costly and labor-intensive annotations, SelfCite leverages a reward signal provided by the LLM itself through context ablation. The effectiveness of SelfCite is demonstrated by increasing citation F1 up to 5.3 points on the LongBench-Cite benchmark across five long-form question answering tasks.
  arXiv  Detail & Related papers  (2025-02-13T18:55:13Z)
• HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction [14.731720495144112]
  We introduce the novel concept of core citation, which identifies the critical references that go beyond superficial mentions. We propose $textbfHLM-Cite, a $textbfH$ybrid $textbfL$anguage $textbfM$odel workflow for citation prediction. We evaluate HLM-Cite across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods.
  arXiv  Detail & Related papers  (2024-10-10T10:46:06Z)
• Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation [51.8188846284153]
  RAG has been widely adopted to enhance Large Language Models (LLMs) Attributed Text Generation (ATG) has attracted growing attention, which provides citations to support the model's responses in RAG. This paper proposes a fine-grained ATG method called ReClaim(Refer & Claim), which alternates the generation of references and answers step by step.
  arXiv  Detail & Related papers  (2024-07-01T20:47:47Z)
• Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias [1.7812428873698407]
  Citation practices are crucial in shaping the structure of scientific knowledge, yet they are often influenced by contemporary norms and biases. The emergence of Large Language Models (LLMs) introduces a new dynamic to these practices. Here, we analyze these characteristics in an experiment using a dataset from AAAI, NeurIPS, ICML, and ICLR.
  arXiv  Detail & Related papers  (2024-05-24T17:34:32Z)
• Can citations tell us about a paper's reproducibility? A case study of machine learning papers [3.5120846057971065]
  Resource constraints and inadequate documentation can make running replications particularly challenging. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges.
  arXiv  Detail & Related papers  (2024-05-07T03:29:11Z)
• CiteBench: A benchmark for Scientific Citation Text Generation [69.37571393032026]
  CiteBench is a benchmark for citation text generation. We make the code for CiteBench publicly available at https://github.com/UKPLab/citebench.
  arXiv  Detail & Related papers  (2022-12-19T16:10:56Z)
• A Siren Song of Open Source Reproducibility [36.754180527471895]
  We argue that this focus on code for replication is misguided if we want to improve the state of reproducible research. There is a lack of evidence for effective actions taken by conferences to encourage and reward. We argue that venues must take more action to advance reproducible machine learning research today.
  arXiv  Detail & Related papers  (2022-04-09T03:06:40Z)
• Deep Graph Learning for Anomalous Citation Detection [55.81334139806342]
  We propose a novel deep graph learning model, namely GLAD (Graph Learning for Anomaly Detection), to identify anomalies in citation networks. Within the GLAD framework, we propose an algorithm called CPU (Citation PUrpose) to discover the purpose of citation based on citation texts.
  arXiv  Detail & Related papers  (2022-02-23T09:05:28Z)
• Towards generating citation sentences for multiple references with intent control [86.53829532976303]
  We build a novel generation model with the Fusion-in-Decoder approach to cope with multiple long inputs. Experiments demonstrate that the proposed approaches provide much more comprehensive features for generating citation sentences.
  arXiv  Detail & Related papers  (2021-12-02T15:32:24Z)
• Enhancing Scientific Papers Summarization with Citation Graph [78.65955304229863]
  We redefine the task of scientific papers summarization by utilizing their citation graph. We construct a novel scientific papers summarization dataset Semantic Scholar Network (SSN) which contains 141K research papers in different domains. Our model can achieve competitive performance when compared with the pretrained models.
  arXiv  Detail & Related papers  (2021-04-07T11:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.