Qualitative Coding Analysis through Open-Source Large Language Models: A User Study and Design Recommendations

• URL: http://arxiv.org/abs/2602.18352v1
• Date: Fri, 20 Feb 2026 17:04:02 GMT
• Title: Qualitative Coding Analysis through Open-Source Large Language Models: A User Study and Design Recommendations
• Abstract summary: ChatQDA is an on-device framework powered by open-source LLMs for privacy-preserving open coding.<n>Our mixed-methods user study reveals that participants rated the system highly for usability and perceived efficiency.<n>We conclude with design recommendations for local-first analysis tools that prioritize verifiable privacy and methodological rigor.
• Score: 0.037282630026096594
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Qualitative data analysis is labor-intensive, yet the privacy risks associated with commercial Large Language Models (LLMs) often preclude their use in sensitive research. To address this, we introduce ChatQDA, an on-device framework powered by open-source LLMs designed for privacy-preserving open coding. Our mixed-methods user study reveals that while participants rated the system highly for usability and perceived efficiency, they exhibited "conditional trust", valuing the tool for surface-level extraction while questioning its interpretive nuance and consistency. Furthermore, despite the technical security of local deployment, participants reported epistemic uncertainty regarding data protection, suggesting that invisible security measures are insufficient to foster trust. We conclude with design recommendations for local-first analysis tools that prioritize verifiable privacy and methodological rigor.

Related papers

• Automating structural reliability analysis with a multi-agent large language model framework [12.330022806125138]
  This study presents a multi-agent large language model framework that automates the structural reliability analysis process.<n> Specialized agents handle problem formulation, method planning, code generation, execution, and result interpretation.<n>Results show that it lowers the expertise barrier to structural reliability assessment while preserving computational trustworthiness.
  arXiv  Detail & Related papers  (2026-07-18T01:45:27Z)
• From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP [0.28675177318965045]
  Large language models are widely deployed in high-stakes NLP tasks, yet risks as bias, hallucination, adversarial vulnerability and unreliable generalization remain.<n>This survey examines recent human-in-the-loop methods that shift NLP from automation toward collaboration for safety and robustness.
  arXiv  Detail & Related papers  (2026-05-24T19:27:20Z)
• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• Leveraging Large Language Models for Trustworthiness Assessment of Web Applications [13.909850314037653]
  This study presents an empirical methodology to automate the trustworthiness assessment of web applications by leveraging Large Language Models (LLMs)<n>We propose an extension of a hierarchical Quality Model (QM) based on the Logic Score of Preference (LSP)<n> Experimental results indicate that excessive structural context can introduce noise, whereas rule-based instructional prompting improves assessment reliability.
  arXiv  Detail & Related papers  (2026-03-24T23:33:54Z)
• Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval [60.25608870901428]
  Trustworthiness is a core research challenge for agentic AI systems built on Large Language Models (LLMs)<n>We propose the task of fact-checking without retrieval, focusing on the verification of arbitrary natural language claims, independent of their source robustness.
  arXiv  Detail & Related papers  (2026-03-05T18:42:51Z)
• Privacy-preserving formal concept analysis: A homomorphic encryption-based concept construction [7.56423213879122]
  We introduce a Privacy-preserving Formal Context Analysis (PFCA) framework that combines binary data representation with homomorphic encryption techniques.<n>This method enables secure and efficient concept construction without revealing private data.<n>These findings have important implications for privacy-preserving data mining and secure knowledge discovery in large-scale FCA applications.
  arXiv  Detail & Related papers  (2025-11-27T05:16:01Z)
• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• Evaluating Language Model Reasoning about Confidential Information [95.64687778185703]
  We study whether language models exhibit contextual robustness, or the capability to adhere to context-dependent safety specifications.<n>We develop a benchmark (PasswordEval) that measures whether language models can correctly determine when a user request is authorized.<n>We find that current open- and closed-source models struggle with this seemingly simple task, and that, perhaps surprisingly, reasoning capabilities do not generally improve performance.
  arXiv  Detail & Related papers  (2025-08-27T15:39:46Z)
• PBa-LLM: Privacy- and Bias-aware NLP using Named-Entity Recognition (NER) [45.870212237420226]
  This work explores the use of Named- Entity Recognition (NER) to facilitate the privacy-preserving training of Large Language Models (LLMs)<n>We propose a framework that uses NER technologies to anonymize sensitive information in text data, such as personal identities or geographic locations.<n>The study involved two language models (BERT and RoBERTa) and six anonymization algorithms (based on Presidio, FLAIR, BERT, and different versions of GPT) applied to a database of 24,000 candidate profiles.
  arXiv  Detail & Related papers  (2025-06-30T14:42:49Z)
• SoK: Semantic Privacy in Large Language Models [24.99241770349404]
  This paper introduces a lifecycle-centric framework to analyze semantic privacy risks across input processing, pretraining, fine-tuning, and alignment stages of Large Language Models (LLMs)<n>We categorize key attack vectors and assess how current defenses, such as differential privacy, embedding encryption, edge computing, and unlearning, address these threats.<n>We conclude by outlining open challenges, including quantifying semantic leakage, protecting multimodal inputs, balancing de-identification with generation quality, and ensuring transparency in privacy enforcement.
  arXiv  Detail & Related papers  (2025-06-30T08:08:15Z)
• Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation [52.83870601473094]
  Embodied agents exhibit immense potential across a multitude of domains.<n>Existing research predominantly concentrates on the security of general large language models.<n>This paper introduces a novel input moderation framework, meticulously designed to safeguard embodied agents.
  arXiv  Detail & Related papers  (2025-04-22T08:34:35Z)
• Communication-Efficient and Privacy-Adaptable Mechanism for Federated Learning [54.20871516148981]
  We introduce the Communication-Efficient and Privacy-Adaptable Mechanism (CEPAM)<n>CEPAM achieves communication efficiency and privacy protection simultaneously.<n>We theoretically analyze the privacy guarantee of CEPAM and investigate the trade-offs among user privacy and accuracy of CEPAM.
  arXiv  Detail & Related papers  (2025-01-21T11:16:05Z)
• An applied Perspective: Estimating the Differential Identifiability Risk of an Exemplary SOEP Data Set [2.66269503676104]
  We show how to compute the risk metric efficiently for a set of basic statistical queries. Our empirical analysis based on an extensive, real-world scientific data set expands the knowledge on how to compute risks under realistic conditions.
  arXiv  Detail & Related papers  (2024-07-04T17:50:55Z)
• Noisy Neighbors: Efficient membership inference attacks against LLMs [2.666596421430287]
  This paper introduces an efficient methodology that generates textitnoisy neighbors for a target sample by adding noise in the embedding space. Our findings demonstrate that this approach closely matches the effectiveness of employing shadow models, showing its usability in practical privacy auditing scenarios.
  arXiv  Detail & Related papers  (2024-06-24T12:02:20Z)
• Distributed Machine Learning and the Semblance of Trust [66.1227776348216]
  Federated Learning (FL) allows the data owner to maintain data governance and perform model training locally without having to share their data. FL and related techniques are often described as privacy-preserving. We explain why this term is not appropriate and outline the risks associated with over-reliance on protocols that were not designed with formal definitions of privacy in mind.
  arXiv  Detail & Related papers  (2021-12-21T08:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.