CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild

• URL: http://arxiv.org/abs/2605.30241v1
• Date: Thu, 28 May 2026 17:09:19 GMT
• Title: CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
• Abstract summary: CommunityFact is a refreshable benchmark for misinformation detection in the wild.<n>This release contains 15,992 standalone claims across five languages and two domains.
• Score: 29.218863878839368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs' source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Unified Hallucination Fuzzing for Multimodal Large Language Models [44.4785399881315]
  Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs)<n>Existing evaluations, predominantly based on static benchmarks, suffer from narrow taxonomical coverage and rapid performance saturation.<n>We present a systematic evaluation framework integrating a comprehensive benchmark with self-evolving stress testing.
  arXiv  Detail & Related papers  (2026-07-15T17:57:37Z)
• What We are Missing in Multimodal LLM Evaluation? [23.144642468756032]
  Multimodal large language models (MLLMs) can process diverse inputs, e.g., text, images, audio, and video, and generate textual responses.<n>Most existing evaluation benchmarks are limited to isolated tasks and reveal little about whether a model integrates information across modalities.<n>We examine current means for evaluating MLLMs and review the existing benchmark taxonomy to identify gaps, including temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention.
  arXiv  Detail & Related papers  (2026-06-24T19:40:53Z)
• DSIPA: Detecting LLM-Generated Texts via Sentiment-Invariant Patterns Divergence Analysis [13.9689104714669]
  DSIPA is a training-free framework that detects large language models (LLMs) content by quantifying sentiment distributional stability under controlled stylistic variation.<n>It operates in a zero-shot, black-box manner, leveraging two unsupervised metrics, sentiment distribution consistency and sentiment distribution preservation.<n>Our framework exhibits superior generalizability across domains and strong resilience to adversarial conditions.
  arXiv  Detail & Related papers  (2026-04-29T06:22:08Z)
• Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval [60.25608870901428]
  Trustworthiness is a core research challenge for agentic AI systems built on Large Language Models (LLMs)<n>We propose the task of fact-checking without retrieval, focusing on the verification of arbitrary natural language claims, independent of their source robustness.
  arXiv  Detail & Related papers  (2026-03-05T18:42:51Z)
• Contradiction to Consensus: Dual Perspective, Multi Source Retrieval Based Claim Verification with Source Level Disagreement using LLM [0.17188280334580197]
  Claim verification, a.k.a. fact-checking, can help identify potential misinformation.<n>Most automated claim verification systems depend on a single knowledge source.<n>We present a novel system for open-domain claim verification.
  arXiv  Detail & Related papers  (2026-02-21T02:21:31Z)
• Multimodal Fact-Level Attribution for Verifiable Reasoning [80.60864342985748]
  Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation.<n>Existing multimodal grounding benchmarks and evaluation methods fail to assess attribution in complex multimodal reasoning.<n>We introduce MuRGAt, a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation.
  arXiv  Detail & Related papers  (2026-02-12T03:10:02Z)
• FedMobile: Enabling Knowledge Contribution-aware Multi-modal Federated Learning with Incomplete Modalities [15.771749047384535]
  Key challenge in mobile sensing systems using multimodal FL is modality incompleteness.<n>Current multimodal FL frameworks typically train multiple unimodal FL subsystems or apply techniques on the node side to approximate missing modalities.<n>We present FedMobile, a new knowledge contribution-aware multimodal FL framework designed for robust learning despite missing modalities.
  arXiv  Detail & Related papers  (2025-02-20T15:10:43Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• Dynamic Analysis and Adaptive Discriminator for Fake News Detection [59.41431561403343]
  We propose a Dynamic Analysis and Adaptive Discriminator (DAAD) approach for fake news detection.<n>For knowledge-based methods, we introduce the Monte Carlo Tree Search algorithm to leverage the self-reflective capabilities of large language models.<n>For semantic-based methods, we define four typical deceit patterns to reveal the mechanisms behind fake news creation.
  arXiv  Detail & Related papers  (2024-08-20T14:13:54Z)
• How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models [95.44559524735308]
  Large language or multimodal model based verification has been proposed to scale up online policing mechanisms for mitigating spread of false and harmful content. We test the limits of improving foundation model performance without continual updating through an initial study of knowledge transfer. Our results on two recent multi-modal fact-checking benchmarks, Mocheg and Fakeddit, indicate that knowledge transfer strategies can improve Fakeddit performance over the state-of-the-art by up to 1.7% and Mocheg performance by up to 2.9%.
  arXiv  Detail & Related papers  (2024-06-29T08:39:07Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.