Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results

• URL: http://arxiv.org/abs/2411.06316v1
• Date: Sun, 10 Nov 2024 00:23:55 GMT
• Title: Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results
• Authors: John Chen, Alexandros Lotsos, Lexie Zhao, Grace Wang, Uri Wilensky, Bruce Sherin, Michael Horn,
• Abstract summary: generative AI (GAI) tools rely on instructions to work, and how to instruct it may matter. This study applied two known and two theory-informed novel approaches to an online community dataset and evaluated the resulting coding results. Our findings show significant discrepancies between ML/GAI approaches and demonstrate the advantage of our approaches.
• Score: 39.96179530555875
• License:
• Abstract: Inductive qualitative methods have been a mainstay of education research for decades, yet it takes much time and effort to conduct rigorously. Recent advances in artificial intelligence, particularly with generative AI (GAI), have led to initial success in generating inductive coding results. Like human coders, GAI tools rely on instructions to work, and how to instruct it may matter. To understand how ML/GAI approaches could contribute to qualitative coding processes, this study applied two known and two theory-informed novel approaches to an online community dataset and evaluated the resulting coding results. Our findings show significant discrepancies between ML/GAI approaches and demonstrate the advantage of our approaches, which introduce human coding processes into GAI prompts.

Related papers

• A Computational Method for Measuring "Open Codes" in Qualitative Analysis [47.358809793796624]
  Open coding is an inductive qualitative process that identifies and interprets "open codes" from datasets. We present a computational method to measure and identify potential biases from "open codes" systematically.
  arXiv  Detail & Related papers  (2024-11-19T00:44:56Z)
• An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We? [8.0988059417354]
  We propose a range of approaches to improve the performance of AI-generated code detection. Our best model outperforms state-of-the-art AI-generated code detector (GPTSniffer) and achieves an F1 score of 82.55.
  arXiv  Detail & Related papers  (2024-11-06T22:48:18Z)
• ChatGPT Code Detection: Techniques for Uncovering the Source of Code [0.0]
  We use advanced classification techniques to differentiate between code written by humans and that generated by ChatGPT. We employ a new approach that combines powerful embedding features (black-box) with supervised learning algorithms. We show that untrained humans solve the same task not better than random guessing.
  arXiv  Detail & Related papers  (2024-05-24T12:56:18Z)
• Instruction Fusion: Advancing Prompt Evolution through Hybridization [27.321629102942754]
  This paper examines the constraints of existing prompt evolution techniques and introduces a novel approach, Instruction Fusion (IF) IF innovatively combines two distinct prompts through a hybridization process, thereby enhancing the evolution of training prompts for code LLMs. Our experimental results reveal that the proposed novel method effectively addresses the shortcomings of prior methods, significantly improving the performance of Code LLMs.
  arXiv  Detail & Related papers  (2023-12-25T11:00:37Z)
• Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach [12.214585409361126]
  Large language models (LLMs)- based code generation is a complex and powerful black-box model. We propose a novel causal graph-based representation of the prompt and the generated code. We illustrate the insights that our framework can provide by studying over 3 popular LLMs with over 12 prompt adjustment strategies.
  arXiv  Detail & Related papers  (2023-10-10T14:56:26Z)
• Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks [59.761411682238645]
  Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks. We introduce a method to incorporate evidentiality of passages -- whether a passage contains correct evidence to support the output -- into training the generator.
  arXiv  Detail & Related papers  (2021-12-16T08:18:47Z)
• Data-Driven and SE-assisted AI Model Signal-Awareness Enhancement and Introspection [61.571331422347875]
  We propose a data-driven approach to enhance models' signal-awareness. We combine the SE concept of code complexity with the AI technique of curriculum learning. We achieve up to 4.8x improvement in model signal awareness.
  arXiv  Detail & Related papers  (2021-11-10T17:58:18Z)
• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning [65.52675802289775]
  We show that an uncertainty aware classifier can solve challenging reinforcement learning problems. We propose a novel method for computing the normalized maximum likelihood (NML) distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions.
  arXiv  Detail & Related papers  (2021-07-15T08:19:57Z)
• Generative Data Augmentation for Commonsense Reasoning [75.26876609249197]
  G-DAUGC is a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. G-DAUGC consistently outperforms existing data augmentation methods based on back-translation. Our analysis demonstrates that G-DAUGC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.
  arXiv  Detail & Related papers  (2020-04-24T06:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.