Accuracy of a Large Language Model in Distinguishing Anti- And Pro-vaccination Messages on Social Media: The Case of Human Papillomavirus Vaccination

• URL: http://arxiv.org/abs/2404.06731v1
• Date: Wed, 10 Apr 2024 04:35:54 GMT
• Title: Accuracy of a Large Language Model in Distinguishing Anti- And Pro-vaccination Messages on Social Media: The Case of Human Papillomavirus Vaccination
• Authors: Soojong Kim, Kwanho Kim, Claire Wonjeong Jo,
• Abstract summary: This research assesses the accuracy of ChatGPT for sentiment analysis to discern different stances toward HPV vaccination. Messages related to HPV vaccination were collected from social media supporting different message formats: Facebook (long format) and Twitter (short format) Accuracy was measured for each message as the level of concurrence between human and machine decisions, ranging between 0 and 1.
• Score: 1.8434042562191815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Objective. Vaccination has engendered a spectrum of public opinions, with social media acting as a crucial platform for health-related discussions. The emergence of artificial intelligence technologies, such as large language models (LLMs), offers a novel opportunity to efficiently investigate public discourses. This research assesses the accuracy of ChatGPT, a widely used and freely available service built upon an LLM, for sentiment analysis to discern different stances toward Human Papillomavirus (HPV) vaccination. Methods. Messages related to HPV vaccination were collected from social media supporting different message formats: Facebook (long format) and Twitter (short format). A selection of 1,000 human-evaluated messages was input into the LLM, which generated multiple response instances containing its classification results. Accuracy was measured for each message as the level of concurrence between human and machine decisions, ranging between 0 and 1. Results. Average accuracy was notably high when 20 response instances were used to determine the machine decision of each message: .882 (SE = .021) and .750 (SE = .029) for anti- and pro-vaccination long-form; .773 (SE = .027) and .723 (SE = .029) for anti- and pro-vaccination short-form, respectively. Using only three or even one instance did not lead to a severe decrease in accuracy. However, for long-form messages, the language model exhibited significantly lower accuracy in categorizing pro-vaccination messages than anti-vaccination ones. Conclusions. ChatGPT shows potential in analyzing public opinions on HPV vaccination using social media content. However, understanding the characteristics and limitations of a language model within specific public health contexts remains imperative.

Related papers

• Large Language Models' Varying Accuracy in Recognizing Risk-Promoting and Health-Supporting Sentiments in Public Health Discourse: The Cases of HPV Vaccination and Heated Tobacco Products [2.0618817976970103]
  Large Language Models (LLMs) have gained attention as a powerful technology, yet their accuracy and feasibility in capturing different opinions on health issues are largely unexplored.<n>This research examines how accurate the three prominent LLMs are in detecting risk-promoting versus health-supporting sentiments.<n>Specifically, models often show higher accuracy for risk-promoting sentiment on Facebook, whereas health-supporting messages on Twitter are more accurately detected.
  arXiv  Detail & Related papers  (2025-07-06T11:57:02Z)
• Conversations with AI Chatbots Increase Short-Term Vaccine Intentions But Do Not Outperform Standard Public Health Messaging [5.816741004594914]
  Large language model (LLM) based chatbots show promise in persuasive communication. This randomized controlled trial involved 930 vaccine-hesitant parents. Discussions significantly increased self-reported vaccination intent (by 7.1-10.3 points on a 100-point scale) compared to no message.
  arXiv  Detail & Related papers  (2025-04-29T07:59:46Z)
• Working with Large Language Models to Enhance Messaging Effectiveness for Vaccine Confidence [0.276240219662896]
  Vaccine hesitancy and misinformation are significant barriers to achieving widespread vaccination coverage. This paper explores the potential of ChatGPT-augmented messaging to promote confidence in vaccination uptake.
  arXiv  Detail & Related papers  (2025-04-14T04:06:46Z)
• ChatGPT and Vaccine Hesitancy: A Comparison of English, Spanish, and French Responses Using a Validated Scale [1.5133368155322295]
  We use the Vaccine Hesitancy Scale to measure the hesitancy of ChatGPT responses in English, Spanish, and French. Results have implications for researchers interested in evaluating and improving the quality and equity of health-related web information.
  arXiv  Detail & Related papers  (2024-05-06T11:13:03Z)
• Analyzing COVID-19 Vaccination Sentiments in Nigerian Cyberspace: Insights from a Manually Annotated Twitter Dataset [2.820717448579396]
  We explore the use of transformer-based language models to study people's acceptance of vaccines in Nigeria. We developed a novel dataset by crawling multi-lingual tweets using relevant hashtags and keywords. Our analysis and visualizations revealed that most tweets expressed neutral sentiments about COVID-19 vaccines, with some individuals expressing positive views.
  arXiv  Detail & Related papers  (2024-01-23T22:49:19Z)
• Agent-Based Model: Simulating a Virus Expansion Based on the Acceptance of Containment Measures [65.62256987706128]
  Compartmental epidemiological models categorize individuals based on their disease status. We propose an ABM architecture that combines an adapted SEIRD model with a decision-making model for citizens. We illustrate the designed model by examining the progression of SARS-CoV-2 infections in A Coruna, Spain.
  arXiv  Detail & Related papers  (2023-07-28T08:01:05Z)
• Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification [60.49594822215981]
  This paper presents a classification model for detecting COVID-19 vaccination related search queries. We propose a novel approach of considering dense features as memory tokens that the model can attend to. We show that this new modeling approach enables a significant improvement to the Vaccine Search Insights (VSI) task.
  arXiv  Detail & Related papers  (2022-12-16T13:57:41Z)
• Doctors vs. Nurses: Understanding the Great Divide in Vaccine Hesitancy among Healthcare Workers [64.1526243118151]
  We find that doctors are overall more positive toward the COVID-19 vaccines. Doctors are more concerned with the effectiveness of the vaccines over newer variants. Nurses pay more attention to the potential side effects on children.
  arXiv  Detail & Related papers  (2022-09-11T14:22:16Z)
• Detection of Increased Time Intervals of Anti-Vaccine Tweets for COVID-19 Vaccine with BERT Model [0.0]
  Anti-vaccination increases in social media can help institutions determine the strategy to be used in combating anti-vaccination. Recording and tracking every tweet entered with human labor would be inefficient. A deep learning-based natural language processing (NLP) model was used to determine the intervals in which anti-vaccine tweets are concentrated.
  arXiv  Detail & Related papers  (2022-01-12T18:30:23Z)
• Insta-VAX: A Multimodal Benchmark for Anti-Vaccine and Misinformation Posts Detection on Social Media [32.252687203366605]
  Anti-vaccine posts on social media have been shown to create confusion and reduce the publics confidence in vaccines. Insta-VAX is a new multi-modal dataset consisting of a sample of 64,957 Instagram posts related to human vaccines.
  arXiv  Detail & Related papers  (2021-12-15T20:34:57Z)
• Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU [9.8215089151757]
  Vaccine hesitancy and refusal can create clusters of low vaccine coverage and reduce the effectiveness of vaccination programs. Social media provides an opportunity to estimate emerging risks to vaccine acceptance by including geographical location and detailing vaccine-related concerns. Methods for classifying social media posts, such as vaccine-related tweets, use language models (LMs) trained on general domain text. We present a novel end-to-end framework consisting of interconnected components that use domain-specific LM trained on vaccine-related tweets and models commonsense knowledge into a bidirectional gated recurrent network (CK-BiGRU) with context-aware attention.
  arXiv  Detail & Related papers  (2021-06-17T15:16:08Z)
• Assessing the Severity of Health States based on Social Media Posts [62.52087340582502]
  We propose a multiview learning framework that models both the textual content as well as contextual-information to assess the severity of the user's health state. The diverse NLU views demonstrate its effectiveness on both the tasks and as well as on the individual disease to assess a user's health.
  arXiv  Detail & Related papers  (2020-09-21T03:45:14Z)
• Cross-lingual Transfer Learning for COVID-19 Outbreak Alignment [90.12602012910465]
  We train on Italy's early COVID-19 outbreak through Twitter and transfer to several other countries. Our experiments show strong results with up to 0.85 Spearman correlation in cross-country predictions.
  arXiv  Detail & Related papers  (2020-06-05T02:04:25Z)
• Falling into the Echo Chamber: the Italian Vaccination Debate on Twitter [65.7192861893042]
  We examine the extent to which the vaccination debate on Twitter is conductive to potential outreach to the vaccination hesitant. We discover that the vaccination skeptics, as well as the advocates, reside in their own distinct "echo chambers" At the center of these echo chambers we find the ardent supporters, for which we build highly accurate network- and content-based classifiers.
  arXiv  Detail & Related papers  (2020-03-26T13:55:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.