Position: Key Claims in LLM Research Have a Long Tail of Footnotes

• URL: http://arxiv.org/abs/2308.07120v2
• Date: Sat, 1 Jun 2024 15:20:25 GMT
• Title: Position: Key Claims in LLM Research Have a Long Tail of Footnotes
• Authors: Anna Rogers, Alexandra Sasha Luccioni,
• Abstract summary: We argue that we do not have a working definition of Large Language Models (LLMs) We critically examine five common claims regarding their properties. We conclude with suggestions for future research directions and their framing.
• Score: 81.14898541318198
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Much of the recent discourse within the ML community has been centered around Large Language Models (LLMs), their functionality and potential -- yet not only do we not have a working definition of LLMs, but much of this discourse relies on claims and assumptions that are worth re-examining. We contribute a definition of LLMs, critically examine five common claims regarding their properties (including 'emergent properties'), and conclude with suggestions for future research directions and their framing.

Related papers

• Understanding the Role of LLMs in Multimodal Evaluation Benchmarks [77.59035801244278]
  This paper investigates the role of the Large Language Model (LLM) backbone in Multimodal Large Language Models (MLLMs) evaluation. Our study encompasses four diverse MLLM benchmarks and eight state-of-the-art MLLMs. Key findings reveal that some benchmarks allow high performance even without visual inputs and up to 50% of error rates can be attributed to insufficient world knowledge in the LLM backbone.
  arXiv  Detail & Related papers  (2024-10-16T07:49:13Z)
• TruthEval: A Dataset to Evaluate LLM Truthfulness and Reliability [0.0]
  We present a curated collection of challenging statements on sensitive topics for benchmarking called TruthEval. These statements were curated by hand and contain known truth values. We perform some initial analyses using this dataset and find several instances of LLMs failing in simple tasks showing their inability to understand simple questions.
  arXiv  Detail & Related papers  (2024-06-04T00:01:35Z)
• Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL [78.80673954827773]
  Large Language Models (LLMs) play a crucial role in capturing structured semantics to enhance language understanding, improve interpretability, and reduce bias. We propose using Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to extract structured semantics. We find interesting potential: LLMs can indeed capture semantic structures, and scaling-up doesn't always mirror potential. We are surprised to discover that significant overlap in the errors is made by both LLMs and untrained humans, accounting for almost 30% of all errors.
  arXiv  Detail & Related papers  (2024-05-10T11:44:05Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks. The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
  arXiv  Detail & Related papers  (2024-01-30T17:38:54Z)
• Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding [1.3654846342364308]
  Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. This position paper critically assesses three points recurring in critiques of LLM capacities. We outline a pragmatic perspective on the issue of real' understanding and intentionality in LLMs.
  arXiv  Detail & Related papers  (2023-10-30T15:51:04Z)
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity [61.54815512469125]
  This survey addresses the crucial issue of factuality in Large Language Models (LLMs) As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital.
  arXiv  Detail & Related papers  (2023-10-11T14:18:03Z)
• Limits for Learning with Language Models [4.20859414811553]
  We show that large language models (LLMs) are unable to learn concepts beyond the first level of the Borel Hierarchy. LLMs will continue to operate without formal guarantees on tasks that require entailments and deep linguistic understanding.
  arXiv  Detail & Related papers  (2023-06-21T12:11:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.