LLM-assisted sentiment analysis for integrated computational and qualitative mixed methods education research: A case study of students' written reflection assignments

• URL: http://arxiv.org/abs/2605.27403v1
• Date: Fri, 24 Apr 2026 02:02:18 GMT
• Title: LLM-assisted sentiment analysis for integrated computational and qualitative mixed methods education research: A case study of students' written reflection assignments
• Abstract summary: Large language models (LLMs) have recently begun to be critically evaluated for use as qualitative research assistants.<n>This study uses a longitudinal case of written student reflections from a study abroad program.<n> statistical testing is used to quantitatively compare sentiment differences according to seven different student identity/lived experience variables.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Written reflection assignments give students valuable opportunities for critical self-assessment, meaning making, and learning processing. Additionally, such reflections provide rich data for qualitative education research. However, qualitative data can be time-consuming to analyze. It is even more time-intensive to qualitatively compare findings between different groups of participants, usually limiting comparison to, at most, one variable (e.g., binary gender). Large language models (LLMs) have recently begun to be critically evaluated for use as qualitative research assistants. Using a longitudinal case of written student reflections (n=151) from a study abroad program, we investigate how LLM-assisted sentiment analysis can enable longitudinal mixed-methods research combining computational and thematic analyses. First, statistical testing is used to quantitatively compare sentiment differences according to seven different student identity/lived experience variables. Then, these results inform qualitative data analysis to investigate the reasons underlying these differences. For the case of undergraduate students studying abroad, we found that prior experience living abroad was the only personal variable impacting students' sentiments of their verbal language and communication behaviors. This workflow has implications for how qualitative researchers can more easily probe multiple variables when comparing participants from different demographic groups.

Related papers

• Sociodemographic Biases in Educational Counselling by Large Language Models [61.30199414932204]
  This study examines sociodemographic biases in Large Language Models (LLMs)<n>We evaluate responses from six LLMs answering questions about 900 vignettes describing students in diverse circumstances.
  arXiv  Detail & Related papers  (2026-04-03T14:08:22Z)
• Methodological Variation in Studying Staff and Student Perceptions of AI [1.701796716399534]
  We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure.<n>To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups.<n>We show that different analyses can produce different results - for a single data source.
  arXiv  Detail & Related papers  (2025-12-22T11:04:46Z)
• Using AI to replicate human experimental results: a motion study [0.11838866556981258]
  This paper explores the potential of large language models (LLMs) as reliable analytical tools in linguistic research.<n>It focuses on the emergence of affective meanings in temporal expressions involving manner-of-motion verbs.
  arXiv  Detail & Related papers  (2025-07-14T14:47:01Z)
• How do Humans and Language Models Reason About Creativity? A Comparative Analysis [12.398832289718703]
  We conducted two experiments examining how including example solutions with ratings impact creativity evaluation.<n>In Study 1, we analyzed creativity ratings from 72 experts with formal science or engineering training.<n>In Study 2, parallel analyses with state-of-the-art LLMs revealed that models prioritized uncommonness and remoteness of ideas when rating originality.
  arXiv  Detail & Related papers  (2025-02-05T15:08:43Z)
• Evaluating Large Language Models with Psychometrics [59.821829073478376]
  This paper offers a comprehensive benchmark for quantifying psychological constructs of Large Language Models (LLMs)<n>Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets.<n>We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation [5.249002650134171]
  Large Language Models (LLMs) have emerged as powerful support tools across various natural language tasks and a range of application domains.<n>This paper provides a comparative overview of twelve studies investigating the potential of LLMs in labelling data.<n>While the models demonstrate promising cost and time-saving benefits, there exist considerable limitations, such as representativeness, bias, sensitivity to prompt variations and English language preference.
  arXiv  Detail & Related papers  (2024-05-02T14:00:22Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting [64.80538055623842]
  sociodemographic prompting is a technique that steers the output of prompt-based models towards answers that humans with specific sociodemographic profiles would give. We show that sociodemographic information affects model predictions and can be beneficial for improving zero-shot learning in subjective NLP tasks.
  arXiv  Detail & Related papers  (2023-09-13T15:42:06Z)
• Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model [0.0]
  The paper presents results and reflection of an experiment done to use the model GPT 3.5-Turbo to emulate some aspects of an inductive Thematic Analysis. The objective of the paper is not to replace human analysts in qualitative analysis but to learn if some elements of LLM data manipulation can to an extent be of support for qualitative research.
  arXiv  Detail & Related papers  (2023-05-22T13:16:07Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Dataset Bias in Few-shot Image Recognition [57.25445414402398]
  We first investigate the impact of transferable capabilities learned from base categories. Second, we investigate performance differences on different datasets from dataset structures and different few-shot learning methods.
  arXiv  Detail & Related papers  (2020-08-18T14:46:23Z)
• Sentiment Analysis Based on Deep Learning: A Comparative Study [69.09570726777817]
  The study of public opinion can provide us with valuable information. The efficiency and accuracy of sentiment analysis is being hindered by the challenges encountered in natural language processing. This paper reviews the latest studies that have employed deep learning to solve sentiment analysis problems.
  arXiv  Detail & Related papers  (2020-06-05T16:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.