How to Use Large Language Models for Text Coding: The Case of Fatherhood
Roles in Public Policy Documents
- URL: http://arxiv.org/abs/2311.11844v2
- Date: Fri, 15 Dec 2023 17:18:48 GMT
- Title: How to Use Large Language Models for Text Coding: The Case of Fatherhood
Roles in Public Policy Documents
- Authors: Lorenzo Lupo, Oscar Magnusson, Dirk Hovy, Elin Naurin, Lena
W\"angnerud
- Abstract summary: Large language models (LLMs) have opened up new opportunities for text analysis in political science.
In this study, we evaluate LLMs on three original coding tasks of non-English political science texts.
- Score: 21.090506974145566
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent advances in large language models (LLMs) like GPT-3 and GPT-4 have
opened up new opportunities for text analysis in political science. They
promise automation with better results and less programming. In this study, we
evaluate LLMs on three original coding tasks of non-English political science
texts, and we provide a detailed description of a general workflow for using
LLMs for text coding in political science research. Our use case offers a
practical guide for researchers looking to incorporate LLMs into their research
on text analysis. We find that, when provided with detailed label definitions
and coding examples, an LLM can be as good as or even better than a human
annotator while being much faster (up to hundreds of times), considerably
cheaper (costing up to 60% less than human coding), and much easier to scale to
large amounts of text. Overall, LLMs present a viable option for most text
coding projects.
Related papers
- ReMoDetect: Reward Models Recognize Aligned LLM's Generations [55.06804460642062]
Large language models (LLMs) generate human-preferable texts.
We propose two training schemes to further improve the detection ability of the reward model.
arXiv Detail & Related papers (2024-05-27T17:38:33Z) - Large Language Models: A Survey [69.72787936480394]
Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks.
LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
arXiv Detail & Related papers (2024-02-09T05:37:09Z) - Large Language Models on Graphs: A Comprehensive Survey [81.7684686396014]
We provide a systematic review of scenarios and techniques related to large language models on graphs.
We first summarize potential scenarios of adopting LLMs on graphs into three categories, namely pure graphs, text-attributed graphs, and text-paired graphs.
We discuss the real-world applications of such methods and summarize open-source codes and benchmark datasets.
arXiv Detail & Related papers (2023-12-05T14:14:27Z) - LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis [18.775126929754833]
Thematic analysis (TA) has been widely used for analyzing qualitative data in many disciplines and fields.
Human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensive and time-consuming.
We propose a human-LLM collaboration framework (i.e., LLM-in-the-loop) to conduct TA with in-context learning (ICL)
arXiv Detail & Related papers (2023-10-23T17:05:59Z) - Large Language Model-Aware In-Context Learning for Code Generation [75.68709482932903]
Large language models (LLMs) have shown impressive in-context learning (ICL) ability in code generation.
We propose a novel learning-based selection approach named LAIL (LLM-Aware In-context Learning) for code generation.
arXiv Detail & Related papers (2023-10-15T06:12:58Z) - How to use LLMs for Text Analysis [0.0]
This guide introduces Large Language Models (LLM) as a highly versatile text analysis method within the social sciences.
As LLMs are easy-to-use, cheap, fast, and applicable on a broad range of text analysis tasks, many scholars believe that LLMs will transform how we do text analysis.
arXiv Detail & Related papers (2023-07-24T19:54:15Z) - The potential of LLMs for coding with low-resource and domain-specific
programming languages [0.0]
This study focuses on the econometric scripting language named hansl of the open-source software gretl.
Our findings suggest that LLMs can be a useful tool for writing, understanding, improving, and documenting gretl code.
arXiv Detail & Related papers (2023-07-24T17:17:13Z) - Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning [5.822010906632045]
This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research.
By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis.
arXiv Detail & Related papers (2023-07-05T10:15:07Z) - Towards Coding Social Science Datasets with Language Models [4.280286557747323]
Researchers often rely on humans to code (label, annotate, etc.) large sets of texts.
Recent advances in a specific kind of artificial intelligence tool - language models (LMs) - provide a solution.
We find that GPT-3 can match the performance of typical human coders and offers benefits over other machine learning methods of coding text.
arXiv Detail & Related papers (2023-06-03T19:11:34Z) - Low-code LLM: Graphical User Interface over Large Language Models [115.08718239772107]
This paper introduces a novel human-LLM interaction framework, Low-code LLM.
It incorporates six types of simple low-code visual programming interactions to achieve more controllable and stable responses.
We highlight three advantages of the low-code LLM: user-friendly interaction, controllable generation, and wide applicability.
arXiv Detail & Related papers (2023-04-17T09:27:40Z) - Can Large Language Models Transform Computational Social Science? [79.62471267510963]
Large Language Models (LLMs) are capable of performing many language processing tasks zero-shot (without training data)
This work provides a road map for using LLMs as Computational Social Science tools.
arXiv Detail & Related papers (2023-04-12T17:33:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.