Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysis

• URL: http://arxiv.org/abs/2508.06729v1
• Date: Fri, 08 Aug 2025 22:06:23 GMT
• Title: Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysis
• Abstract summary: This paper presents a scalable framework to automate semantic and sentiment annotation for Japanese American Incarceration Oral History.<n>Using LLMs, we construct a high-quality dataset, evaluate multiple models, and test prompt engineering strategies.<n>For sentiment analysis, Llama slightly outperformed Qwen (82.66%) and ChatGPT (82.29%), with all models showing comparable results.
• Score: 0.25165775267615204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Oral histories are vital records of lived experience, particularly within communities affected by systemic injustice and historical erasure. Effective and efficient analysis of their oral history archives can promote access and understanding of the oral histories. However, Large-scale analysis of these archives remains limited due to their unstructured format, emotional complexity, and high annotation costs. This paper presents a scalable framework to automate semantic and sentiment annotation for Japanese American Incarceration Oral History. Using LLMs, we construct a high-quality dataset, evaluate multiple models, and test prompt engineering strategies in historically sensitive contexts. Our multiphase approach combines expert annotation, prompt design, and LLM evaluation with ChatGPT, Llama, and Qwen. We labeled 558 sentences from 15 narrators for sentiment and semantic classification, then evaluated zero-shot, few-shot, and RAG strategies. For semantic classification, ChatGPT achieved the highest F1 score (88.71%), followed by Llama (84.99%) and Qwen (83.72%). For sentiment analysis, Llama slightly outperformed Qwen (82.66%) and ChatGPT (82.29%), with all models showing comparable results. The best prompt configurations were used to annotate 92,191 sentences from 1,002 interviews in the JAIOH collection. Our findings show that LLMs can effectively perform semantic and sentiment annotation across large oral history collections when guided by well-designed prompts. This study provides a reusable annotation pipeline and practical guidance for applying LLMs in culturally sensitive archival analysis. By bridging archival ethics with scalable NLP techniques, this work lays the groundwork for responsible use of artificial intelligence in digital humanities and preservation of collective memory. GitHub: https://github.com/kc6699c/LLM4OralHistoryAnalysis.

Related papers

• The Shape of Testimony: A Scalable Framework for Oral History Archive Comparison [2.12833538322985]
  Researchers in Holocaust studies have often distinguished between two styles of oral survivor testimony.<n>The USC Shoah Foundation's interviews tend to follow a structured, interviewer-guided format, whereas the Yale Fortunoff Video Archive favors a more free-form, open-ended style.<n>This study critically examines that claim by conducting a large-scale computational analysis of more than 1,600 testimonies from both collections.
  arXiv  Detail & Related papers  (2026-05-20T18:36:33Z)
• Applying Large Language Models to Characterize Public Narratives [10.803799931002061]
  We propose a novel computational framework that leverages large language models (LLMs) to automate the qualitative annotation of public narratives.<n>Our work reveals that LLMs can achieve near-human-expert performance, achieving an average F1 score of 0.80 across 8 narratives and 14 codes.
  arXiv  Detail & Related papers  (2025-11-17T15:41:55Z)
• LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives [0.0]
  Large Language Models (LLMs) promise enhanced context understanding.<n>We evaluated three Dutch-specific LLMs against LIWC and Pattern.<n>Surprisingly, despite architectural advancements, the Dutch-tuned LLMs underperformed compared to traditional methods.
  arXiv  Detail & Related papers  (2025-11-10T21:26:13Z)
• A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950 [0.0]
  This paper compares large language models (LLMs) and traditional natural language processing (NLP) tools for performing word segmentation, part-of-speech (POS) tagging, and named entity recognition (NER)<n> Historical Chinese documents pose challenges for text analysis due to their logographic script, the absence of natural word boundaries, and significant linguistic changes.
  arXiv  Detail & Related papers  (2025-03-25T17:07:21Z)
• Narrative Analysis of True Crime Podcasts With Knowledge Graph-Augmented Large Language Models [8.78598447041169]
  Large language models (LLMs) still struggle with complex narrative arcs as well as narratives containing conflicting information. Recent work indicates LLMs augmented with external knowledge bases can improve the accuracy and interpretability of the resulting models. In this work, we analyze the effectiveness of applying knowledge graphs (KGs) in understanding true-crime podcast data.
  arXiv  Detail & Related papers  (2024-11-01T21:49:00Z)
• LFED: A Literary Fiction Evaluation Dataset for Large Language Models [58.85989777743013]
  We collect 95 literary fictions that are either originally written in Chinese or translated into Chinese, covering a wide range of topics across several centuries. We define a question taxonomy with 8 question categories to guide the creation of 1,304 questions. We conduct an in-depth analysis to ascertain how specific attributes of literary fictions (e.g., novel types, character numbers, the year of publication) impact LLM performance in evaluations.
  arXiv  Detail & Related papers  (2024-05-16T15:02:24Z)
• NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens [63.7488938083696]
  We introduce NovelQA, a benchmark tailored for evaluating Large Language Models (LLMs) with complex, extended narratives.<n>NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding.<n>Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses.
  arXiv  Detail & Related papers  (2024-03-18T17:32:32Z)
• StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding [72.38872974837462]
  We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus. textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
  arXiv  Detail & Related papers  (2023-10-19T16:29:23Z)
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models [91.05820785008527]
  We propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) We build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment.
  arXiv  Detail & Related papers  (2023-07-20T17:59:41Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology [9.941695905504282]
  This study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation.<n>We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder.
  arXiv  Detail & Related papers  (2023-05-15T04:10:13Z)
• Active Learning for Abstractive Text Summarization [50.79416783266641]
  We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
  arXiv  Detail & Related papers  (2023-01-09T10:33:14Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.