A Human-Centered Workflow for Using Large Language Models in Content Analysis

• URL: http://arxiv.org/abs/2603.19271v1
• Date: Fri, 27 Feb 2026 14:32:29 GMT
• Title: A Human-Centered Workflow for Using Large Language Models in Content Analysis
• Abstract summary: This paper conceptualizes Large Language Models as universal text processing machines.<n>It presents a comprehensive workflow for employing LLMs in three qualitative and quantitative content analysis tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While many researchers use Large Language Models (LLMs) through chat-based access, their real potential lies in leveraging LLMs via application programming interfaces (APIs). This paper conceptualizes LLMs as universal text processing machines and presents a comprehensive workflow for employing LLMs in three qualitative and quantitative content analysis tasks: (1) annotation (an umbrella term for qualitative coding, labeling and text classification), (2) summarization, and (3) information extraction. The workflow is explicitly human-centered. Researchers design, supervise, and validate each stage of the LLM process to ensure rigor and transparency. Our approach synthesizes insights from extensive methodological literature across multiple disciplines: political science, sociology, computer science, psychology, and management. We outline validation procedures and best practices to address key limitations of LLMs, such as their black-box nature, prompt sensitivity, and tendency to hallucinate. To facilitate practical implementation, we provide supplementary materials, including a prompt library and Python code in Jupyter Notebook format, accompanied by detailed usage instructions.

Related papers

• User Reviews as a Source for Usability Requirements: A Precursor Study on Using Large Language Models [1.6498033620778052]
  This paper provides a dataset of 300 user reviews containing usability-relevant aspects from three different types of apps.<n>We also analyze if the performance of LLMs in understanding user reviews is comparable to the performance of human raters.
  arXiv  Detail & Related papers  (2026-05-12T19:05:04Z)
• A Comprehensive Survey on Long Context Language Modeling [118.5540791080351]
  Long Context Language Models (LCLMs) process and analyze extensive inputs in an effective and efficient way.<n>Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively.
  arXiv  Detail & Related papers  (2025-03-20T17:06:28Z)
• Analysis on LLMs Performance for Code Summarization [0.0]
  Large Language Models (LLMs) have significantly advanced the field of code summarization.<n>This study aims to perform a comparative analysis of several open-source LLMs, namely LLaMA-3, Phi-3, Mistral, and Gemma.
  arXiv  Detail & Related papers  (2024-12-22T17:09:34Z)
• To Err Is Human; To Annotate, SILICON? Reducing Measurement Error in LLM Annotation [11.470318058523466]
  Large Language Models (LLMs) promise a cost-effective scalable alternative to human annotation.<n>We develop the SILICON methodology to systematically reduce measurement error from LLM annotation.<n>Our evidence indicates that reducing each error source is necessary, and that SILICON supports rigorous annotation in management research.
  arXiv  Detail & Related papers  (2024-12-19T02:21:41Z)
• Personalized Multimodal Large Language Models: A Survey [127.9521218125761]
  Multimodal Large Language Models (MLLMs) have become increasingly important due to their state-of-the-art performance and ability to integrate multiple data modalities.<n>This paper presents a comprehensive survey on personalized multimodal large language models, focusing on their architecture, training methods, and applications.
  arXiv  Detail & Related papers  (2024-12-03T03:59:03Z)
• Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5 [0.0]
  We present a novel approach wherein we distill document understanding knowledge from the proprietary LLM ChatGPT into FLAN-T5. Our findings underscore the potential of distillation techniques in facilitating the deployment of sophisticated language models in real-world scenarios.
  arXiv  Detail & Related papers  (2024-09-17T15:37:56Z)
• Fine-tuning Multimodal Large Language Models for Product Bundling [53.01642741096356]
  We introduce Bundle-MLLM, a novel framework that fine-tunes large language models (LLMs) through a hybrid item tokenization approach.<n>Specifically, we integrate textual, media, and relational data into a unified tokenization, introducing a soft separation token to distinguish between textual and non-textual tokens.<n>We propose a progressive optimization strategy that fine-tunes LLMs for disentangled objectives: 1) learning bundle patterns and 2) enhancing multimodal semantic understanding specific to product bundling.
  arXiv  Detail & Related papers  (2024-07-16T13:30:14Z)
• Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts [7.005758904228446]
  We gather and curate three real-world political science codebooks -- covering protest events, political violence and manifestos.<n>We propose a five-stage framework for codebook-LLM measurement: preparing a codebook for both humans and LLMs, testing LLMs' basic capabilities on a codebook, evaluating zero-shot measurement accuracy.<n>We find current open-weight LLMs have limitations in following codebooks zero-shot, but that supervised instruction tuning can substantially improve performance.
  arXiv  Detail & Related papers  (2024-07-15T14:20:09Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations [50.81844184210381]
  We propose a general paradigm that augments large language models with DOmain-specific KnowledgE to enhance their performance on practical applications, namely DOKE. This paradigm relies on a domain knowledge extractor, working in three steps: 1) preparing effective knowledge for the task; 2) selecting the knowledge for each specific sample; and 3) expressing the knowledge in an LLM-understandable way.
  arXiv  Detail & Related papers  (2023-11-16T07:09:38Z)
• LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis [18.775126929754833]
  Thematic analysis (TA) has been widely used for analyzing qualitative data in many disciplines and fields. Human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensive and time-consuming. We propose a human-LLM collaboration framework (i.e., LLM-in-the-loop) to conduct TA with in-context learning (ICL)
  arXiv  Detail & Related papers  (2023-10-23T17:05:59Z)
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond [48.70557995528463]
  This guide aims to provide researchers and practitioners with valuable insights and best practices for working with Large Language Models. We present various use cases and non-use cases to illustrate the practical applications and limitations of LLMs in real-world scenarios.
  arXiv  Detail & Related papers  (2023-04-26T17:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.