A Survey of Pre-trained Language Models for Processing Scientific Text
- URL: http://arxiv.org/abs/2401.17824v1
- Date: Wed, 31 Jan 2024 13:35:07 GMT
- Title: A Survey of Pre-trained Language Models for Processing Scientific Text
- Authors: Xanh Ho, Anh Khoa Duong Nguyen, An Tuan Dao, Junfeng Jiang, Yuki
Chida, Kaito Sugimoto, Huy Quoc To, Florian Boudin and Akiko Aizawa
- Abstract summary: The number of Language Models (LMs) dedicated to processing scientific text is on the rise.
This work provides a comprehensive review of SciLMs, including an analysis of their effectiveness across different domains, tasks and datasets.
- Score: 26.986805626077892
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The number of Language Models (LMs) dedicated to processing scientific text
is on the rise. Keeping pace with the rapid growth of scientific LMs (SciLMs)
has become a daunting task for researchers. To date, no comprehensive surveys
on SciLMs have been undertaken, leaving this issue unaddressed. Given the
constant stream of new SciLMs, appraising the state-of-the-art and how they
compare to each other remain largely unknown. This work fills that gap and
provides a comprehensive review of SciLMs, including an extensive analysis of
their effectiveness across different domains, tasks and datasets, and a
discussion on the challenges that lie ahead.
Related papers
- What is the Role of Large Language Models in the Evolution of Astronomy Research? [0.0]
ChatGPT and other state-of-the-art large language models (LLMs) are rapidly transforming multiple fields.
These models, commonly trained on vast datasets, exhibit human-like text generation capabilities.
arXiv Detail & Related papers (2024-09-30T12:42:25Z) - Towards Efficient Large Language Models for Scientific Text: A Review [4.376712802685017]
Large language models (LLMs) have ushered in a new era for processing complex information in various fields, including science.
Due to the power of LLMs, they require extremely expensive computational resources, intense amounts of data, and training time.
In recent years, researchers have proposed various methodologies to make scientific LLMs more affordable.
arXiv Detail & Related papers (2024-08-20T10:57:34Z) - A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery [68.48094108571432]
Large language models (LLMs) have revolutionized the way text and other modalities of data are handled.
We aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs.
arXiv Detail & Related papers (2024-06-16T08:03:24Z) - Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals.
Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
arXiv Detail & Related papers (2024-04-01T17:45:15Z) - Scientific Large Language Models: A Survey on Biological & Chemical Domains [47.97810890521825]
Large Language Models (LLMs) have emerged as a transformative power in enhancing natural language comprehension.
The application of LLMs extends beyond conventional linguistic boundaries, encompassing specialized linguistic systems developed within various scientific disciplines.
As a burgeoning area in the community of AI for Science, scientific LLMs warrant comprehensive exploration.
arXiv Detail & Related papers (2024-01-26T05:33:34Z) - SciGLM: Training Scientific Language Models with Self-Reflective
Instruction Annotation and Tuning [60.14510984576027]
SciGLM is a suite of scientific language models able to conduct college-level scientific reasoning.
We apply a self-reflective instruction annotation framework to generate step-by-step reasoning for unlabelled scientific questions.
We fine-tuned the ChatGLM family of language models with SciInstruct, enhancing their scientific and mathematical reasoning capabilities.
arXiv Detail & Related papers (2024-01-15T20:22:21Z) - An Interdisciplinary Outlook on Large Language Models for Scientific
Research [3.4108358650013573]
We describe the capabilities and constraints of Large Language Models (LLMs) within disparate academic disciplines, aiming to delineate their strengths and limitations with precision.
We examine how LLMs augment scientific inquiry, offering concrete examples such as accelerating literature review by summarizing vast numbers of publications.
We articulate the challenges LLMs face, including their reliance on extensive and sometimes biased datasets, and the potential ethical dilemmas stemming from their use.
arXiv Detail & Related papers (2023-11-03T19:41:09Z) - A Comprehensive Overview of Large Language Models [68.22178313875618]
Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks.
This article provides an overview of the existing literature on a broad range of LLM-related concepts.
arXiv Detail & Related papers (2023-07-12T20:01:52Z) - Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks.
We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
arXiv Detail & Related papers (2023-05-24T10:45:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.