Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models

• URL: http://arxiv.org/abs/2403.16851v3
• Date: Thu, 25 Sep 2025 14:58:23 GMT
• Title: Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models
• Abstract summary: This study analysed 3,815 tweets referencing 604 retracted articles and 3,373 tweets referencing 668 comparable non-retracted articles.<n>Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction.<n>This study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.
• Score: 3.789359673129504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Timely detection of problematic research is essential for safeguarding scientific integrity. To explore whether social media commentary can serve as an early indicator of potentially problematic articles, this study analysed 3,815 tweets referencing 604 retracted articles and 3,373 tweets referencing 668 comparable non-retracted articles. Tweets critical of the articles were identified through both human annotation and large language models (LLMs). Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction, compared to only 1.5% of non-retracted articles, highlighting the potential of tweets as early warning signals of retraction. However, critical tweets identified by LLMs (GPT-4o mini, Gemini 2.0 Flash-Lite, and Claude 3.5 Haiku) only partially aligned with human annotation, suggesting that fully automated monitoring of post-publication discourse should be applied with caution. A human-AI collaborative approach may offer a more reliable and scalable alternative, with human expertise helping to filter out tweets critical of issues unrelated to the research integrity of the articles. Overall, this study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• Science discussions of retracted articles on Bluesky: public scrutiny or misinformation spreading? [5.065311321920548]
  Post-publication peer review has emerged as an important supplement to traditional peer review.<n>It remains unclear whether social media discussions of retracted articles primarily reflect good practices.<n>We manually coded Bluesky posts discussing retracted articles to identify instances of good and bad practice.
  arXiv  Detail & Related papers  (2026-05-05T22:41:04Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild [47.03825808787752]
  "Fake news familiarity" emerges as a candidate mediator, suggesting that exposure may function as adversarial training for human discriminators.<n>These findings suggest that "pre-bunking" interventions should target cognitive source monitoring rather than demographic segmentation.
  arXiv  Detail & Related papers  (2026-01-30T11:49:58Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers [61.57691030102618]
  We propose a novel jailbreaking method, Paper Summary Attack (llmnamePSA)<n>It synthesizes content from either attack-focused or defense-focused LLM safety paper to construct an adversarial prompt template.<n>Experiments show significant vulnerabilities not only in base LLMs, but also in state-of-the-art reasoning model like Deepseek-R1.
  arXiv  Detail & Related papers  (2025-07-17T18:33:50Z)
• When Detection Fails: The Power of Fine-Tuned Models to Generate Human-Like Social Media Text [13.14749943120523]
  Social media represents a significant attack vector in online influence campaigns.<n>We create a dataset of 505,159 AI-generated social media posts from a combination of open-source, closed-source, and fine-tuned LLMs.<n>We show that while the posts can be detected under typical research assumptions, under the more realistic assumption that an attacker will not release their fine-tuned model to the public, detectability drops dramatically.
  arXiv  Detail & Related papers  (2025-06-11T17:51:28Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Vulnerability of LLMs to Vertically Aligned Text Manipulations [108.6908427615402]
  Large language models (LLMs) have become highly effective at performing text classification tasks. modifying input formats, such as vertically aligning words for encoder-based models, can substantially lower accuracy in text classification tasks. Do decoder-based LLMs exhibit similar vulnerabilities to vertically formatted text input?
  arXiv  Detail & Related papers  (2024-10-26T00:16:08Z)
• Public interest in science or bots? Selective amplification of scientific articles on Twitter [1.1606619391009658]
  We used the Altmetric dataset in combination with data collected through the Twitter API and the Botometer API. We analyzed the data to see the possibility of bot activity based on different characteristics of the article. We trained machine-learning models using this dataset to identify possible bot activity in any given article.
  arXiv  Detail & Related papers  (2024-09-28T07:48:44Z)
• Modes of Analyzing Disinformation Narratives With AI/ML/Text Mining to Assist in Mitigating the Weaponization of Social Media [0.8287206589886879]
  This paper highlights the developing need for quantitative modes for capturing and monitoring malicious communication in social media. There has been a deliberate "weaponization" of messaging through the use of social networks including by politically oriented entities both state sponsored and privately run. Despite attempts to introduce moderation on major platforms like Facebook and X/Twitter, there are now established alternative social networks that offer completely unmoderated spaces.
  arXiv  Detail & Related papers  (2024-05-25T00:02:14Z)
• Exploring the Potential of the Large Language Models (LLMs) in Identifying Misleading News Headlines [2.0330684186105805]
  This study explores the efficacy of Large Language Models (LLMs) in identifying misleading versus non-misleading news headlines. Our analysis reveals significant variance in model performance, with ChatGPT-4 demonstrating superior accuracy.
  arXiv  Detail & Related papers  (2024-05-06T04:06:45Z)
• A survey of recent methods for addressing AI fairness and bias in biomedicine [48.46929081146017]
  Artificial intelligence systems may perpetuate social inequities or demonstrate biases, such as those based on race or gender. We surveyed recent publications on different debiasing methods in the fields of biomedical natural language processing (NLP) or computer vision (CV) We performed a literature search on PubMed, ACM digital library, and IEEE Xplore of relevant articles published between January 2018 and December 2023 using multiple combinations of keywords. We reviewed other potential methods from the general domain that could be applied to biomedicine to address bias and improve fairness.
  arXiv  Detail & Related papers  (2024-02-13T06:38:46Z)
• Efficacy of Utilizing Large Language Models to Detect Public Threat Posted Online [0.0]
  This paper examines the efficacy of utilizing large language models (LLMs) to detect public threats posted online. Data collection tools were developed to amass post titles from a popular Korean online community. GPT-4 performed best overall with 97.9% non-threat and 100% threat accuracy.
  arXiv  Detail & Related papers  (2023-12-29T16:42:02Z)
• Topological Data Mapping of Online Hate Speech, Misinformation, and General Mental Health: A Large Language Model Based Study [6.803493330690884]
  Recent advances in machine learning and large language models have made such an analysis possible. In this study, we collected thousands of posts from carefully selected communities on the social media site Reddit. We performed various machine-learning classifications based on embeddings in order to understand the role of hate speech/misinformation in various communities.
  arXiv  Detail & Related papers  (2023-09-22T15:10:36Z)
• Unsupervised Sentiment Analysis of Plastic Surgery Social Media Posts [91.3755431537592]
  The massive collection of user posts across social media platforms is primarily untapped for artificial intelligence (AI) use cases. Natural language processing (NLP) is a subfield of AI that leverages bodies of documents, known as corpora, to train computers in human-like language understanding. This study demonstrates that the applied results of unsupervised analysis allow a computer to predict either negative, positive, or neutral user sentiment towards plastic surgery.
  arXiv  Detail & Related papers  (2023-07-05T20:16:20Z)
• Measuring the Effect of Influential Messages on Varying Personas [67.1149173905004]
  We present a new task, Response Forecasting on Personas for News Media, to estimate the response a persona might have upon seeing a news message. The proposed task not only introduces personalization in the modeling but also predicts the sentiment polarity and intensity of each response. This enables more accurate and comprehensive inference on the mental state of the persona.
  arXiv  Detail & Related papers  (2023-05-25T21:01:00Z)
• ManiTweet: A New Benchmark for Identifying Manipulation of News on Social Media [74.93847489218008]
  We present a novel task, identifying manipulation of news on social media, which aims to detect manipulation in social media posts and identify manipulated or inserted information. To study this task, we have proposed a data collection schema and curated a dataset called ManiTweet, consisting of 3.6K pairs of tweets and corresponding articles. Our analysis demonstrates that this task is highly challenging, with large language models (LLMs) yielding unsatisfactory performance.
  arXiv  Detail & Related papers  (2023-05-23T16:40:07Z)
• Can AI-Generated Text be Reliably Detected? [50.95804851595018]
  Large Language Models (LLMs) perform impressively well in various applications. The potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use. We stress-test the robustness of these AI text detectors in the presence of an attacker.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)
• Predicting Seriousness of Injury in a Traffic Accident: A New Imbalanced Dataset and Benchmark [62.997667081978825]
  The paper introduces a new dataset to assess the performance of machine learning algorithms in the prediction of the seriousness of injury in a traffic accident. The dataset is created by aggregating publicly available datasets from the UK Department for Transport.
  arXiv  Detail & Related papers  (2022-05-20T21:15:26Z)
• What goes on inside rumour and non-rumour tweets and their reactions: A Psycholinguistic Analyses [58.75684238003408]
  psycho-linguistics analyses of social media text are vital for drawing meaningful conclusions to mitigate misinformation. This research contributes by performing an in-depth psycholinguistic analysis of rumours related to various kinds of events.
  arXiv  Detail & Related papers  (2021-11-09T07:45:11Z)
• A PubMedBERT-based Classifier with Data Augmentation Strategy for Detecting Medication Mentions in Tweets [2.539568419434224]
  Twitter publishes a large number of user-generated text (tweets) on a daily basis. entity recognition (NER) presents some special challenges for tweet data. In this paper, we explore a PubMedBERT-based classifier trained with a combination of multiple data augmentation approaches. Our method achieved an F1 score of 0.762, which is substantially higher than the mean of all submissions.
  arXiv  Detail & Related papers  (2021-11-03T14:29:24Z)
• You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction [52.442129609979794]
  Recent deep learning approaches for trajectory prediction show promising performance. It remains unclear which features such black-box models actually learn to use for making predictions. This paper proposes a procedure that quantifies the contributions of different cues to model performance.
  arXiv  Detail & Related papers  (2021-10-11T14:24:15Z)
• Misleading the Covid-19 vaccination discourse on Twitter: An exploratory study of infodemic around the pandemic [0.45593531937154413]
  We collect a moderate-sized representative corpus of tweets (200,000 approx.) pertaining to Covid-19 vaccination over a period of seven months (September 2020 - March 2021) Following a Transfer Learning approach, we utilize the pre-trained Transformer-based XLNet model to classify tweets as Misleading or Non-Misleading. We build on this to study and contrast the characteristics of tweets in the corpus that are misleading in nature against non-misleading ones. Several ML models are employed for prediction, with up to 90% accuracy, and the importance of each feature is explained using SHAP Explainable AI (X
  arXiv  Detail & Related papers  (2021-08-16T17:02:18Z)
• Understanding Information Spreading Mechanisms During COVID-19 Pandemic by Analyzing the Impact of Tweet Text and User Features for Retweet Prediction [6.658785818853953]
  COVID-19 has affected the world economy and the daily life routine of almost everyone. Social media platforms enable users to share information with other users who can reshare this information. We propose two CNN and RNN based models and evaluate the performance of these models on a publicly available TweetsCOV19 dataset.
  arXiv  Detail & Related papers  (2021-05-26T15:55:58Z)
• Claim Detection in Biomedical Twitter Posts [11.335643770130238]
  False information on biomedical topics can be particularly dangerous. We aim to fill this research gap and annotate a corpus of 1200 tweets for implicit and explicit biomedical claims. We develop baseline models which detect tweets that contain a claim automatically.
  arXiv  Detail & Related papers  (2021-04-23T14:45:31Z)
• Combining exogenous and endogenous signals with a semi-supervised co-attention network for early detection of COVID-19 fake tweets [14.771202995527315]
  During COVID-19, tweets with misinformation should be flagged and neutralized in their early stages to mitigate the damages. Most of the existing methods for early detection of fake news assume to have enough propagation information for large labeled tweets. We present ENDEMIC, a novel early detection model which leverages endogenous and endogenous signals related to tweets.
  arXiv  Detail & Related papers  (2021-04-12T10:01:44Z)
• Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News [57.9843300852526]
  We introduce the more realistic and challenging task of defending against machine-generated news that also includes images and captions. To identify the possible weaknesses that adversaries can exploit, we create a NeuralNews dataset composed of 4 different types of generated articles. In addition to the valuable insights gleaned from our user study experiments, we provide a relatively effective approach based on detecting visual-semantic inconsistencies.
  arXiv  Detail & Related papers  (2020-09-16T14:13:15Z)
• Exploratory Analysis of Covid-19 Tweets using Topic Modeling, UMAP, and DiGraphs [36.33347149799959]
  This paper illustrates five different techniques to assess the distinctiveness of topics, key terms and features, speed of information dissemination, and network behaviors for Covid19 tweets. One topic specific to U.S. cases would start to uptick immediately after live White House Coronavirus Task Force briefings. One of the simplest highlights of this analysis is that early-stage descriptive methods like regular expressions can successfully identify high-level themes.
  arXiv  Detail & Related papers  (2020-05-06T19:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.