Evaluating AI capabilities in detecting conspiracy theories on YouTube

• URL: http://arxiv.org/abs/2505.23570v2
• Date: Fri, 04 Jul 2025 14:02:59 GMT
• Title: Evaluating AI capabilities in detecting conspiracy theories on YouTube
• Authors: Leonardo La Rocca, Francesco Corso, Francesco Pierri,
• Abstract summary: This study explores the use of open-weight Large Language Models (LLMs), both text-only and multimodal, for identifying conspiracy theory videos on YouTube.<n>We evaluate a variety of LLMs in a zero-shot setting and compare their performance to a fine-tuned RoBERTa baseline.<n>Results show that text-based LLMs achieve high recall but lower precision, leading to increased false positives.<n>Multimodal models lag behind their text-only counterparts, indicating limited benefits from visual data integration.
• Score: 0.1474723404975345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As a leading online platform with a vast global audience, YouTube's extensive reach also makes it susceptible to hosting harmful content, including disinformation and conspiracy theories. This study explores the use of open-weight Large Language Models (LLMs), both text-only and multimodal, for identifying conspiracy theory videos shared on YouTube. Leveraging a labeled dataset of thousands of videos, we evaluate a variety of LLMs in a zero-shot setting and compare their performance to a fine-tuned RoBERTa baseline. Results show that text-based LLMs achieve high recall but lower precision, leading to increased false positives. Multimodal models lag behind their text-only counterparts, indicating limited benefits from visual data integration. To assess real-world applicability, we evaluate the most accurate models on an unlabeled dataset, finding that RoBERTa achieves performance close to LLMs with a larger number of parameters. Our work highlights the strengths and limitations of current LLM-based approaches for online harmful content detection, emphasizing the need for more precise and robust systems.

Related papers

• Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers [59.168391398830515]
  We evaluate 12 pre-trained LLMs and one specialized fact-verifier, using a collection of examples from 14 fact-checking benchmarks.<n>We highlight the importance of addressing annotation errors and ambiguity in datasets.<n> frontier LLMs with few-shot in-context examples, often overlooked in previous works, achieve top-tier performance.
  arXiv  Detail & Related papers  (2025-06-16T10:32:10Z)
• LLM-based Semantic Augmentation for Harmful Content Detection [5.954202581988127]
  This paper introduces an approach that prompts large language models to clean noisy text and provide context-rich explanations.<n>We evaluate on the SemEval 2024 multi-label Persuasive Meme dataset and validate on the Google Jigsaw toxic comments and Facebook hateful memes datasets.<n>Our results reveal that zero-shot LLM classification underperforms on these high-context tasks compared to supervised models.
  arXiv  Detail & Related papers  (2025-04-22T02:59:03Z)
• Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection [22.804501061898616]
  Log Anomaly Detection (LAD) seeks to identify atypical patterns in log data that are crucial to assessing the security and condition of systems.<n>Although Large Language Models (LLMs) have shown tremendous success in various fields, the use of LLMs in enabling the detection of log anomalies is largely unexplored.<n>We explore the use of parameter-efficient fine-tuning techniques (PEFTs) for adapting LLMs to LAD.
  arXiv  Detail & Related papers  (2025-03-11T05:00:19Z)
• Protecting multimodal large language models against misleading visualizations [94.71976205962527]
  We introduce the first inference-time methods to improve performance on misleading visualizations.<n>We find that MLLM question-answering accuracy drops on average to the level of a random baseline.
  arXiv  Detail & Related papers  (2025-02-27T20:22:34Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system.<n>It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting [20.07438999071414]
  Large Language Models (LLMs) are rapidly transforming the landscape of digital content creation.<n>We present FD-Dataset, a comprehensive bilingual fingerprinting benchmark comprising 90,000 text samples from 20 famous proprietary and open-source LLMs.<n>We also present FDLLM, a novel fingerprinting method that leverages parameter-efficient Low-Rank Adaptation (LoRA) to fine-tune a foundation model.
  arXiv  Detail & Related papers  (2025-01-27T13:18:40Z)
• LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms [0.0]
  LLM-SEM (Language Model-Based Student Engagement Metric) is a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement.<n>We generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes.<n>Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels.
  arXiv  Detail & Related papers  (2024-12-18T12:01:53Z)
• Scaling Up Video Summarization Pretraining with Large Language Models [73.74662411006426]
  We introduce an automated and scalable pipeline for generating a large-scale video summarization dataset. We analyze the limitations of existing approaches and propose a new video summarization model that effectively addresses them. Our work also presents a new benchmark dataset that contains 1200 long videos each with high-quality summaries annotated by professionals.
  arXiv  Detail & Related papers  (2024-04-04T11:59:06Z)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance [51.30560006045442]
  Image-gRounded guIdaNcE (MARINE) is a framework that is both training-free and API-free.<n>MARINE effectively and efficiently reduces object hallucinations during inference by introducing image-grounded guidance to LVLMs.<n>Our framework's flexibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance.
  arXiv  Detail & Related papers  (2024-02-13T18:59:05Z)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models [52.98743860365194]
  We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN) At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents.
  arXiv  Detail & Related papers  (2024-01-02T18:53:13Z)
• Video Understanding with Large Language Models: A Survey [97.29126722004949]
  Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding.<n>The emergent capabilities Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity reasoning.<n>This survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs.
  arXiv  Detail & Related papers  (2023-12-29T01:56:17Z)
• Validating Large Language Models with ReLM [11.552979853457117]
  Large language models (LLMs) have been touted for their ability to generate natural-sounding text. There are growing concerns around possible negative effects of LLMs such as data memorization, bias, and inappropriate language. We introduce ReLM, a system for validating and querying LLMs using standard regular expressions.
  arXiv  Detail & Related papers  (2022-11-21T21:40:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.