LLM-based uncertainty assessment of social media situational signals for crisis reporting

• URL: http://arxiv.org/abs/2605.00829v1
• Date: Tue, 17 Mar 2026 17:13:09 GMT
• Title: LLM-based uncertainty assessment of social media situational signals for crisis reporting
• Abstract summary: Social media has become a critical source of situational awareness during disasters.<n>We propose an uncertainty-aware framework for automated situational awareness reporting.<n>We apply this framework to over 200,000 earthquake-related Twitter/X posts.
• Score: 0.9877005520976848
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Social media has become a critical source of situational awareness during disasters, providing real-time insights into evolving impacts and emerging needs. To support crisis response at scale, recent work has increasingly leveraged large language models (LLMs) to automatically classify and summarize situational information from social media streams. However, existing approaches implicitly assume that extracted situational claims are equally plausible, despite information quality varying substantially as a crisis unfolds. In this work, we propose an uncertainty-aware framework for automated situational awareness reporting that explicitly accounts for the plausibility of social media claims. First, we classify social media posts according to an established situational awareness schema. Second, we introduce an uncertainty assessment layer that evaluates whether individual situational claims plausibly reflect real-world conditions when conditioned on external proxy data, while explicitly eliciting the model's confidence in this judgment. Third, we use these uncertainty assessments to generate crisis reports that communicate not only what is being reported, but how certain those reports are. We apply this framework to over 200,000 earthquake-related Twitter/X posts, using impact summaries from the USGS PAGER as a representative external proxy. We argue that explicitly representing uncertainty supports human crisis communicators in prioritizing information under time pressure, and provides a framework for integrating external proxy data into LLM-based situational awareness pipelines.

Related papers

• Large Language Models for Causal Relations Extraction in Social Media: A Validation Framework for Disaster Intelligence [10.83315119677514]
  We examine whether Large Language Models (LLMs) can effectively extract causal relations from disaster-related social media posts.<n>Our findings highlight both the potential and risks of using LLMs for causal relation extraction in disaster decision-support systems.
  arXiv  Detail & Related papers  (2026-05-12T00:07:45Z)
• Agents of Chaos [50.53354213047402]
  We report an exploratory red-teaming study of autonomous language-model-powered agents deployed in a live laboratory environment.<n>Twenty AI researchers interacted with the agents under benign and adversarial conditions.<n>Our findings establish the existence of security-, privacy-, and governance-relevant vulnerabilities in realistic deployment settings.
  arXiv  Detail & Related papers  (2026-02-23T16:28:48Z)
• Situational Awareness as the Imperative Capability for Disaster Resilience in the Era of Complex Hazards and Artificial Intelligence [1.1368819707015188]
  Disasters frequently exceed established hazard models, revealing blind spots where unforeseen impacts and vulnerabilities hamper effective response.<n> situational awareness (SA)-the ability to perceive, interpret, and project dynamic crisis conditions-is an often overlooked yet vital capability for disaster resilience.
  arXiv  Detail & Related papers  (2025-08-21T03:38:47Z)
• Information Retrieval Induced Safety Degradation in AI Agents [52.15553901577888]
  This study investigates how expanding retrieval access affects model reliability, bias propagation, and harmful content generation.<n>Retrieval-enabled agents built on aligned LLMs often behave more unsafely than uncensored models without retrieval.<n>These findings underscore the need for robust mitigation strategies to ensure fairness and reliability in retrieval-enabled and increasingly autonomous AI systems.
  arXiv  Detail & Related papers  (2025-05-20T11:21:40Z)
• Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports [2.230742111425553]
  This paper introduces a dynamic Retrieval-Augmented Generation (RAG) system that autonomously generates situation awareness reports.<n>Our system constructs query-specific knowledge bases on demand, ensuring timely, relevant, and accurate insights.<n>The system is tested across multiple real-world scenarios, demonstrating its effectiveness in producing coherent, insightful, and actionable reports.
  arXiv  Detail & Related papers  (2025-05-14T16:36:30Z)
• MisinfoEval: Generative AI in the Era of "Alternative Facts" [50.069577397751175]
  We introduce a framework for generating and evaluating large language model (LLM) based misinformation interventions. We present (1) an experiment with a simulated social media environment to measure effectiveness of misinformation interventions, and (2) a second experiment with personalized explanations tailored to the demographics and beliefs of users. Our findings confirm that LLM-based interventions are highly effective at correcting user behavior.
  arXiv  Detail & Related papers  (2024-10-13T18:16:50Z)
• Enhancing Trustworthiness and Minimising Bias Issues in Leveraging Social Media Data for Disaster Management Response [0.1499944454332829]
  Leveraging real-time data can significantly deal with data uncertainty and enhance disaster response efforts. Social media appeared as an effective source of real-time data as there has been extensive use of social media during and after the disasters. It also brings forth challenges regarding trustworthiness and bias in these data. We aim to investigate and identify the factors that can be used to enhance trustworthiness and minimize bias.
  arXiv  Detail & Related papers  (2024-08-15T10:59:20Z)
• CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics [50.122541222825156]
  This study introduces a novel approach to disaster text classification by enhancing a pre-trained Large Language Model (LLM)<n>Our methodology involves creating a comprehensive instruction dataset from disaster-related tweets, which is then used to fine-tune an open-source LLM.<n>This fine-tuned model can classify multiple aspects of disaster-related information simultaneously, such as the type of event, informativeness, and involvement of human aid.
  arXiv  Detail & Related papers  (2024-06-16T23:01:10Z)
• From Mean to Extreme: Formal Differential Privacy Bounds on the Success of Real-World Data Reconstruction Attacks [54.25638567385662]
  Differential Privacy in machine learning is often interpreted as guarantees against membership inference.<n> translating DP budgets into quantitative protection against the more damaging threat of data reconstruction remains a challenging open problem.<n>This paper bridges the critical gap by deriving the first formal privacy bounds tailored to the mechanics of demonstrated "from-scratch" attacks.
  arXiv  Detail & Related papers  (2024-02-20T09:52:30Z)
• Adverse Media Mining for KYC and ESG Compliance [2.381399746981591]
  Adverse media or negative news screening is crucial for the identification of such non-financial risks. We present an automated system to conduct both real-time and batch search of adverse media for users' queries. Our scalable, machine-learning driven approach to high-precision, adverse news filtering is based on four perspectives.
  arXiv  Detail & Related papers  (2021-10-22T01:04:16Z)
• Stateful Offline Contextual Policy Evaluation and Learning [88.9134799076718]
  We study off-policy evaluation and learning from sequential data. We formalize the relevant causal structure of problems such as dynamic personalized pricing. We show improved out-of-sample policy performance in this class of relevant problems.
  arXiv  Detail & Related papers  (2021-10-19T16:15:56Z)
• Emergency Incident Detection from Crowdsourced Waze Data using Bayesian Information Fusion [4.039649741925056]
  This paper presents a novel method for emergency incident detection using noisy crowdsourced Waze data. We propose a principled computational framework based on observational theory to model the uncertainty in the reliability of crowd-generated reports.
  arXiv  Detail & Related papers  (2020-11-10T22:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.