If the Sources Could Talk: Evaluating Large Language Models for Research
Assistance in History
- URL: http://arxiv.org/abs/2310.10808v1
- Date: Mon, 16 Oct 2023 20:12:06 GMT
- Title: If the Sources Could Talk: Evaluating Large Language Models for Research
Assistance in History
- Authors: Giselle Gonzalez Garcia, Christian Weilbach
- Abstract summary: We show that by augmenting Large-Language Models with vector embeddings from highly specialized academic sources, a conversational methodology can be made accessible to historians and other researchers in the Humanities.
Compared to established search interfaces for digital catalogues, such as metadata and full-text search, we evaluate the richer conversational style of LLMs on the performance of two main types of tasks.
- Score: 1.3325600043256554
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: The recent advent of powerful Large-Language Models (LLM) provides a new
conversational form of inquiry into historical memory (or, training data, in
this case). We show that by augmenting such LLMs with vector embeddings from
highly specialized academic sources, a conversational methodology can be made
accessible to historians and other researchers in the Humanities. Concretely,
we evaluate and demonstrate how LLMs have the ability of assisting researchers
while they examine a customized corpora of different types of documents,
including, but not exclusive to: (1). primary sources, (2). secondary sources
written by experts, and (3). the combination of these two. Compared to
established search interfaces for digital catalogues, such as metadata and
full-text search, we evaluate the richer conversational style of LLMs on the
performance of two main types of tasks: (1). question-answering, and (2).
extraction and organization of data. We demonstrate that LLMs semantic
retrieval and reasoning abilities on problem-specific tasks can be applied to
large textual archives that have not been part of the its training data.
Therefore, LLMs can be augmented with sources relevant to specific research
projects, and can be queried privately by researchers.
Related papers
- When Search Engine Services meet Large Language Models: Visions and Challenges [53.32948540004658]
This paper conducts an in-depth examination of how integrating Large Language Models with search engines can mutually benefit both technologies.
We focus on two main areas: using search engines to improve LLMs (Search4LLM) and enhancing search engine functions using LLMs (LLM4Search)
arXiv Detail & Related papers (2024-06-28T03:52:13Z) - LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
Large language models (LLMs) have shown remarkable versatility in various generative tasks.
This study focuses on the topic of LLMs assist NLP Researchers.
To our knowledge, this is the first work to provide such a comprehensive analysis.
arXiv Detail & Related papers (2024-06-24T01:30:22Z) - ResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents [21.17856299966841]
Large language models (LLMs) have exhibited remarkable performance across various tasks in natural language processing.
We develop ResearchArena, a benchmark that measures LLM agents' ability to conduct academic surveys.
arXiv Detail & Related papers (2024-06-13T03:26:30Z) - CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search [67.6104548484555]
We introduce CHIQ, a two-step method that leverages the capabilities of open-source large language models (LLMs) to resolve ambiguities in the conversation history before query rewriting.
We demonstrate on five well-established benchmarks that CHIQ leads to state-of-the-art results across most settings.
arXiv Detail & Related papers (2024-06-07T15:23:53Z) - Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A~Case~Study~at~HCMUT [2.8000537365271367]
Large language models (LLMs) have emerged as a vibrant research topic.
LLMs face challenges in remembering events, incorporating new information, and addressing domain-specific issues or hallucinations.
This article proposes a method for automatically constructing a Knowledge Graph from multiple data sources.
arXiv Detail & Related papers (2024-04-14T16:34:31Z) - UFO: a Unified and Flexible Framework for Evaluating Factuality of Large
Language Models [73.73303148524398]
Large language models (LLMs) may generate text that lacks consistency with human knowledge, leading to factual inaccuracies or textithallucination.
We propose textttUFO, an LLM-based unified and flexible evaluation framework to verify facts against plug-and-play fact sources.
arXiv Detail & Related papers (2024-02-22T16:45:32Z) - Quantitative knowledge retrieval from large language models [4.155711233354597]
Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences.
This paper explores the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid data analysis tasks.
arXiv Detail & Related papers (2024-02-12T16:32:37Z) - Exploring the Potential of Large Language Models in Computational Argumentation [54.85665903448207]
Large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language.
This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings.
arXiv Detail & Related papers (2023-11-15T15:12:15Z) - DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain
Question Answering over Knowledge Base and Text [73.68051228972024]
Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge.
Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
arXiv Detail & Related papers (2023-10-31T04:37:57Z) - Towards an Understanding of Large Language Models in Software
Engineering Tasks [32.09925582943177]
Large Language Models (LLMs) have drawn widespread attention and research due to their astounding performance in tasks such as text generation and reasoning.
This paper is the first to comprehensively investigate and collate the research and products combining LLMs with software engineering.
We have collected related literature as extensively from seven mainstream databases, and selected 123 papers for analysis.
arXiv Detail & Related papers (2023-08-22T12:37:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.