If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History

• URL: http://arxiv.org/abs/2310.10808v1
• Date: Mon, 16 Oct 2023 20:12:06 GMT
• Title: If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History
• Authors: Giselle Gonzalez Garcia, Christian Weilbach
• Abstract summary: We show that by augmenting Large-Language Models with vector embeddings from highly specialized academic sources, a conversational methodology can be made accessible to historians and other researchers in the Humanities. Compared to established search interfaces for digital catalogues, such as metadata and full-text search, we evaluate the richer conversational style of LLMs on the performance of two main types of tasks.
• Score: 1.3325600043256554
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The recent advent of powerful Large-Language Models (LLM) provides a new conversational form of inquiry into historical memory (or, training data, in this case). We show that by augmenting such LLMs with vector embeddings from highly specialized academic sources, a conversational methodology can be made accessible to historians and other researchers in the Humanities. Concretely, we evaluate and demonstrate how LLMs have the ability of assisting researchers while they examine a customized corpora of different types of documents, including, but not exclusive to: (1). primary sources, (2). secondary sources written by experts, and (3). the combination of these two. Compared to established search interfaces for digital catalogues, such as metadata and full-text search, we evaluate the richer conversational style of LLMs on the performance of two main types of tasks: (1). question-answering, and (2). extraction and organization of data. We demonstrate that LLMs semantic retrieval and reasoning abilities on problem-specific tasks can be applied to large textual archives that have not been part of the its training data. Therefore, LLMs can be augmented with sources relevant to specific research projects, and can be queried privately by researchers.

Related papers

• What is the Role of Large Language Models in the Evolution of Astronomy Research? [0.0]
  ChatGPT and other state-of-the-art large language models (LLMs) are rapidly transforming multiple fields. These models, commonly trained on vast datasets, exhibit human-like text generation capabilities.
  arXiv  Detail & Related papers  (2024-09-30T12:42:25Z)
• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
  Large language models (LLMs) have shown remarkable versatility in various generative tasks. This study focuses on the topic of LLMs assist NLP Researchers. To our knowledge, this is the first work to provide such a comprehensive analysis.
  arXiv  Detail & Related papers  (2024-06-24T01:30:22Z)
• ResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents [21.17856299966841]
  Large language models (LLMs) have exhibited remarkable performance across various tasks in natural language processing. We develop ResearchArena, a benchmark that measures LLM agents' ability to conduct academic surveys.
  arXiv  Detail & Related papers  (2024-06-13T03:26:30Z)
• CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search [67.6104548484555]
  We introduce CHIQ, a two-step method that leverages the capabilities of open-source large language models (LLMs) to resolve ambiguities in the conversation history before query rewriting. We demonstrate on five well-established benchmarks that CHIQ leads to state-of-the-art results across most settings.
  arXiv  Detail & Related papers  (2024-06-07T15:23:53Z)
• PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval [76.50690734636477]
  We propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations.
  arXiv  Detail & Related papers  (2024-04-29T04:51:30Z)
• Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUT [2.8000537365271367]
  Large language models (LLMs) have emerged as a vibrant research topic. LLMs face challenges in remembering events, incorporating new information, and addressing domain-specific issues or hallucinations. This article proposes a method for automatically constructing a Knowledge Graph from multiple data sources.
  arXiv  Detail & Related papers  (2024-04-14T16:34:31Z)
• UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models [73.73303148524398]
  Large language models (LLMs) may generate text that lacks consistency with human knowledge, leading to factual inaccuracies or textithallucination. We propose textttUFO, an LLM-based unified and flexible evaluation framework to verify facts against plug-and-play fact sources.
  arXiv  Detail & Related papers  (2024-02-22T16:45:32Z)
• Quantitative knowledge retrieval from large language models [4.155711233354597]
  Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences. This paper explores the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid data analysis tasks.
  arXiv  Detail & Related papers  (2024-02-12T16:32:37Z)
• Exploring the Potential of Large Language Models in Computational Argumentation [54.85665903448207]
  Large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2023-11-15T15:12:15Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.