CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing

• URL: http://arxiv.org/abs/2004.06774v4
• Date: Sat, 17 Apr 2021 16:10:22 GMT
• Title: CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing
• Authors: Firoj Alam, Hassan Sajjad, Muhammad Imran and Ferda Ofli
• Abstract summary: We consolidate eight human-annotated datasets and provide 166.1k and 141.5k tweets for textitinformativeness and textithumanitarian classification tasks. We provide benchmarks for both binary and multiclass classification tasks using several deep learning architecrures including, CNN, fastText, and transformers.
• Score: 13.11283003017537
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Time-critical analysis of social media streams is important for humanitarian organizations for planing rapid response during disasters. The \textit{crisis informatics} research community has developed several techniques and systems for processing and classifying big crisis-related data posted on social media. However, due to the dispersed nature of the datasets used in the literature (e.g., for training models), it is not possible to compare the results and measure the progress made towards building better models for crisis informatics tasks. In this work, we attempt to bridge this gap by combining various existing crisis-related datasets. We consolidate eight human-annotated datasets and provide 166.1k and 141.5k tweets for \textit{informativeness} and \textit{humanitarian} classification tasks, respectively. We believe that the consolidated dataset will help train more sophisticated models. Moreover, we provide benchmarks for both binary and multiclass classification tasks using several deep learning architecrures including, CNN, fastText, and transformers. We make the dataset and scripts available at: https://crisisnlp.qcri.org/crisis_datasets_benchmarks.html

Related papers

• Zero-Shot Classification of Crisis Tweets Using Instruction-Finetuned Large Language Models [0.0]
  We assess three commercial large language models in zero-shot classification of short social media posts. The models are asked to perform two classification tasks: 1) identify if the post is informative in a humanitarian context; and 2) rank and provide probabilities for the post in relation to 16 possible humanitarian classes. Results are evaluated using macro, weighted, and binary F1-scores.
  arXiv  Detail & Related papers  (2024-09-30T19:33:58Z)
• CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics [49.2719253711215]
  This study introduces a novel approach to disaster text classification by enhancing a pre-trained Large Language Model (LLM) Our methodology involves creating a comprehensive instruction dataset from disaster-related tweets, which is then used to fine-tune an open-source LLM. This fine-tuned model can classify multiple aspects of disaster-related information simultaneously, such as the type of event, informativeness, and involvement of human aid.
  arXiv  Detail & Related papers  (2024-06-16T23:01:10Z)
• CrisisMatch: Semi-Supervised Few-Shot Learning for Fine-Grained Disaster Tweet Classification [51.58605842457186]
  We present a fine-grained disaster tweet classification model under the semi-supervised, few-shot learning setting. Our model, CrisisMatch, effectively classifies tweets into fine-grained classes of interest using few labeled data and large amounts of unlabeled data.
  arXiv  Detail & Related papers  (2023-10-23T07:01:09Z)
• DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank [52.20298962359658]
  In crisis events, people often use social media platforms such as Twitter to disseminate information about the situation, warnings, advice, and support. fully-supervised approaches require annotating vast amounts of data and are impractical due to limited response time. Semi-supervised models can be biased, performing moderately well for certain classes while performing extremely poorly for others. We propose a simple but effective debiasing method, DeCrisisMB, that utilizes a Memory Bank to store and perform equal sampling for generated pseudo-labels from each class at each training.
  arXiv  Detail & Related papers  (2023-10-23T05:25:51Z)
• CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts [3.690904966341072]
  Social media platforms play an essential role in crisis communication, but analyzing crisis-related social media texts is challenging due to their informal nature. This study introduces CrisisTransformers, an ensemble of pre-trained language models and sentence encoders trained on an extensive corpus of over 15 billion word tokens from tweets.
  arXiv  Detail & Related papers  (2023-09-11T14:36:16Z)
• CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and Summarization [62.77066949111921]
  This paper presents CrisisLTLSum, the largest dataset of local crisis event timelines available to date. CrisisLTLSum contains 1,000 crisis event timelines across four domains: wildfires, local fires, traffic, and storms. Our initial experiments indicate a significant gap between the performance of strong baselines compared to the human performance on both tasks.
  arXiv  Detail & Related papers  (2022-10-25T17:32:40Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Kubric: A scalable dataset generator [73.78485189435729]
  Kubric is a Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation.
  arXiv  Detail & Related papers  (2022-03-07T18:13:59Z)
• HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks [5.937482215664902]
  Social media content is often too noisy for direct use in any application. It is important to filter, categorize, and concisely summarize the available content to facilitate effective consumption and decision-making. We present a new large-scale dataset with 77K human-labeled tweets, sampled from a pool of 24 million tweets across 19 disaster events.
  arXiv  Detail & Related papers  (2021-04-07T12:29:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.