Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences

• URL: http://arxiv.org/abs/2309.06578v3
• Date: Tue, 26 Mar 2024 03:33:45 GMT
• Title: Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences
• Authors: Sai Koneru, Jian Wu, Sarah Rajtmajer,
• Abstract summary: A strong hypothesis is a best guess based on existing evidence and informed by a comprehensive view of relevant literature. With exponential increase in the number of scientific articles published annually, manual aggregation and synthesis of evidence related to a given hypothesis is a challenge. We share a novel dataset for the task of scientific hypothesis evidencing using community-driven annotations of studies in the social sciences.
• Score: 3.9985385067438344
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hypothesis formulation and testing are central to empirical research. A strong hypothesis is a best guess based on existing evidence and informed by a comprehensive view of relevant literature. However, with exponential increase in the number of scientific articles published annually, manual aggregation and synthesis of evidence related to a given hypothesis is a challenge. Our work explores the ability of current large language models (LLMs) to discern evidence in support or refute of specific hypotheses based on the text of scientific abstracts. We share a novel dataset for the task of scientific hypothesis evidencing using community-driven annotations of studies in the social sciences. We compare the performance of LLMs to several state-of-the-art benchmarks and highlight opportunities for future research in this area. The dataset is available at https://github.com/Sai90000/ScientificHypothesisEvidencing.git

Related papers

• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery [68.48094108571432]
  We aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs. We comprehensively survey over 250 scientific LLMs, discuss their commonalities and differences, as well as summarize pre-training datasets and evaluation tasks for each field and modality.
  arXiv  Detail & Related papers  (2024-06-16T08:03:24Z)
• Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
  This paper looks at the causal inference task of treatment effect estimation. We assume binary effects that are recorded as high-dimensional images in a Randomized Controlled Trial. We compare 6 480 models fine-tuned from state-of-the-art visual backbones. We find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.
  arXiv  Detail & Related papers  (2024-05-27T13:26:34Z)
• LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery [141.39722070734737]
  We propose to enhance the knowledge-driven, abstract reasoning abilities of Large Language Models with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework. We conduct experiments to demonstrate our framework's efficacy in law discovery and molecular design.
  arXiv  Detail & Related papers  (2024-05-16T03:04:10Z)
• Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
  We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
  arXiv  Detail & Related papers  (2024-04-01T17:45:15Z)
• Retrieve to Explain: Evidence-driven Predictions with Language Models [0.791663505497707]
  We introduce Retrieve to Explain (R2E), a retrieval-based language model. R2E scores and ranks all possible answers to a research question based on evidence retrieved from a document corpus. We assess on the challenging task of drug target identification from scientific literature.
  arXiv  Detail & Related papers  (2024-02-06T15:13:17Z)
• Large Language Models are Zero Shot Hypothesis Proposers [17.612235393984744]
  Large Language Models (LLMs) hold a wealth of global and interdisciplinary knowledge that promises to break down information barriers. We construct a dataset consist of background knowledge and hypothesis pairs from biomedical literature. We evaluate the hypothesis generation capabilities of various top-tier instructed models in zero-shot, few-shot, and fine-tuning settings.
  arXiv  Detail & Related papers  (2023-11-10T10:03:49Z)
• Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
  This work proposes the first dataset for social science academic hypotheses discovery. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
  arXiv  Detail & Related papers  (2023-09-06T05:19:41Z)
• Modeling Information Change in Science Communication with Semantically Matched Paraphrases [50.67030449927206]
  SPICED is the first paraphrase dataset of scientific findings annotated for degree of information change. SPICED contains 6,000 scientific finding pairs extracted from news stories, social media discussions, and full texts of original papers. Models trained on SPICED improve downstream performance on evidence retrieval for fact checking of real-world scientific claims.
  arXiv  Detail & Related papers  (2022-10-24T07:44:38Z)
• Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference [5.283529004179579]
  Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) is the task of predicting the entailment relation between a pair of sentences. Models that understand entailment should encode both, the premise and the hypothesis. Experiments by Poliak et al. revealed a strong preference of these models towards patterns observed only in the hypothesis.
  arXiv  Detail & Related papers  (2021-01-19T01:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.