Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations

• URL: http://arxiv.org/abs/2311.03999v1
• Date: Tue, 7 Nov 2023 13:54:56 GMT
• Title: Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
• Authors: Lixiang Yan, Vanessa Echeverria, Gloria Fernandez Nieto, Yueqiao Jin, Zachari Swiecki, Linxuan Zhao, Dragan Ga\v{s}evi\'c, Roberto Martinez-Maldonado
• Abstract summary: Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.

Related papers

• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• Generative AI Tools in Academic Research: Applications and Implications for Qualitative and Quantitative Research Methodologies [0.0]
  This study examines the impact of Generative Artificial Intelligence (GenAI) on academic research, focusing on its application to qualitative and quantitative data analysis. GenAI tools evolve rapidly, they offer new possibilities for enhancing research productivity and democratising complex analytical processes. Their integration into academic practice raises significant questions regarding research integrity and security, authorship, and the changing nature of scholarly work.
  arXiv  Detail & Related papers  (2024-08-13T13:10:03Z)
• Augmenting the Author: Exploring the Potential of AI Collaboration in Academic Writing [25.572926673827165]
  This case study highlights the importance of prompt design, output analysis, and recognizing the AI's limitations to ensure responsible and effective AI integration in scholarly work. The paper contributes to the field of Human-Computer Interaction by exploring effective prompt strategies and providing a comparative analysis of Gen AI models.
  arXiv  Detail & Related papers  (2024-04-23T19:06:39Z)
• Negotiating the Shared Agency between Humans & AI in the Recommender System [1.4249472316161877]
  Concerns about user agency have arisen due to the inherent opacity (information asymmetry) and the nature of one-way output (power asymmetry) on algorithms. We seek to understand how types of agency impact user perception and experience, and bring empirical evidence to refine the guidelines and designs for human-AI interactive systems.
  arXiv  Detail & Related papers  (2024-03-23T19:23:08Z)
• How Human-Centered Explainable AI Interface Are Designed and Evaluated: A Systematic Survey [48.97104365617498]
  The emerging area of em Explainable Interfaces (EIs) focuses on the user interface and user experience design aspects of XAI. This paper presents a systematic survey of 53 publications to identify current trends in human-XAI interaction and promising directions for EI design and development.
  arXiv  Detail & Related papers  (2024-03-21T15:44:56Z)
• A Survey on Human-AI Teaming with Large Pre-Trained Models [7.280953657497549]
  Human-AI (HAI) Teaming has emerged as a cornerstone for advancing problem-solving and decision-making processes. The advent of Large Pre-trained Models (LPtM) has significantly transformed this landscape. It offers unprecedented capabilities by leveraging vast amounts of data to understand and predict complex patterns.
  arXiv  Detail & Related papers  (2024-03-07T22:37:49Z)
• Confounding-Robust Policy Improvement with Human-AI Teams [9.823906892919746]
  We propose a novel solution to address unobserved confounding in human-AI collaboration by employing the marginal sensitivity model (MSM) Our approach combines domain expertise with AI-driven statistical modeling to account for potential confounders that may otherwise remain hidden.
  arXiv  Detail & Related papers  (2023-10-13T02:39:52Z)
• BO-Muse: A human expert and AI teaming framework for accelerated experimental design [58.61002520273518]
  Our algorithm lets the human expert take the lead in the experimental process. We show that our algorithm converges sub-linearly, at a rate faster than the AI or human alone.
  arXiv  Detail & Related papers  (2023-03-03T02:56:05Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• Human-Robot Collaboration and Machine Learning: A Systematic Review of Recent Research [69.48907856390834]
  Human-robot collaboration (HRC) is the approach that explores the interaction between a human and a robot. This paper proposes a thorough literature review of the use of machine learning techniques in the context of HRC.
  arXiv  Detail & Related papers  (2021-10-14T15:14:33Z)
• You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
  The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
  arXiv  Detail & Related papers  (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.