LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations

• URL: http://arxiv.org/abs/2511.14528v1
• Date: Tue, 18 Nov 2025 14:32:48 GMT
• Title: LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations
• Authors: Tatiane Ornelas, Allysson Allex Araújo, Júlia Araújo, Marina Araújo, Bianca Trinkenreich, Marcos Kalinowski,
• Abstract summary: Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE)<n>This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis.
• Score: 5.660100855602864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: [Context] Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE), yet the methodological implications of this usage remain underexplored. Their integration into interpretive processes such as thematic analysis raises fundamental questions about rigor, transparency, and researcher agency. [Objective] This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis. [Method] A reflective workshop with 25 ISERN researchers guided participants through structured discussions of LLM-assisted open coding, theme generation, and theme reviewing, using color-coded canvases to document perceived opportunities, limitations, and recommendations. [Results] Participants recognized potential efficiency and scalability gains, but highlighted risks related to bias, contextual loss, reproducibility, and the rapid evolution of LLMs. They also emphasized the need for prompting literacy and continuous human oversight. [Conclusion] Findings portray LLMs as tools that can support, but not substitute, interpretive analysis. The study contributes to ongoing community reflections on how LLMs can responsibly enhance qualitative research in SE.

Related papers

• LLM-REVal: Can We Trust LLM Reviewers Yet? [70.58742663985652]
  Large language models (LLMs) have inspired researchers to integrate them extensively into the academic workflow.<n>This study focuses on how the deep integration of LLMs into both peer-review and research processes may influence scholarly fairness.
  arXiv  Detail & Related papers  (2025-10-14T10:30:20Z)
• Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research [11.38769375570817]
  We argue that the SE research community must proactively engage with and shape the integration of Large Language Models into research practices.<n>This position paper employs McLuhan's Tetrad of Media Laws to analyze the impact of LLMs on SE research.
  arXiv  Detail & Related papers  (2025-06-15T02:25:50Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering [0.46426852157920906]
  The study emphasizes the need for structured strategies and guidelines to optimize LLM use in qualitative research within software engineering.<n>While LLMs show promise in supporting qualitative analysis, human expertise remains crucial for interpreting data, and ongoing exploration of best practices will be vital for their successful integration into empirical software engineering research.
  arXiv  Detail & Related papers  (2024-12-09T15:17:36Z)
• The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? [60.01746782465275]
  Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership.
  arXiv  Detail & Related papers  (2024-10-07T02:30:18Z)
• Automating Thematic Analysis: How LLMs Analyse Controversial Topics [5.025737475817937]
  Large Language Models (LLMs) are promising analytical tools. This paper explores how LLMs can support thematic analysis of controversial topics. Our findings highlight intriguing overlaps and variances in thematic categorisation between human and machine agents.
  arXiv  Detail & Related papers  (2024-05-11T05:28:25Z)
• A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law [65.87885628115946]
  Large language models (LLMs) are revolutionizing the landscapes of finance, healthcare, and law. We highlight the instrumental role of LLMs in enhancing diagnostic and treatment methodologies in healthcare, innovating financial analytics, and refining legal interpretation and compliance strategies. We critically examine the ethics for LLM applications in these fields, pointing out the existing ethical concerns and the need for transparent, fair, and robust AI systems.
  arXiv  Detail & Related papers  (2024-05-02T22:43:02Z)
• A critical review of methods and challenges in large language models [6.850038413666062]
  Review provides in-depth analysis of Large Language Models (LLMs)<n>Examines the evolution from Recurrent Neural Networks (RNNs) to Transformer models.<n>Describes state-of-the-art techniques such as in-context learning and various fine-tuning approaches.
  arXiv  Detail & Related papers  (2024-04-18T08:01:20Z)
• Apprentices to Research Assistants: Advancing Research with Large Language Models [0.0]
  Large Language Models (LLMs) have emerged as powerful tools in various research domains. This article examines their potential through a literature review and firsthand experimentation.
  arXiv  Detail & Related papers  (2024-04-09T15:53:06Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.