TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media

• URL: http://arxiv.org/abs/2209.07216v2
• Date: Fri, 16 Sep 2022 16:54:46 GMT
• Title: TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media
• Authors: Daniel Loureiro, Aminette D'Souza, Areej Nasser Muhajab, Isabella A. White, Gabriel Wong, Luis Espinosa Anke, Leonardo Neves, Francesco Barbieri, Jose Camacho-Collados
• Abstract summary: We present TempoWiC, a new benchmark aimed at accelerating research in social media-based meaning shift. Our results show that TempoWiC is a challenging benchmark, even for recently-released language models specialized in social media.
• Score: 17.840417362892104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language evolves over time, and word meaning changes accordingly. This is especially true in social media, since its dynamic nature leads to faster semantic shifts, making it challenging for NLP models to deal with new content and trends. However, the number of datasets and models that specifically address the dynamic nature of these social platforms is scarce. To bridge this gap, we present TempoWiC, a new benchmark especially aimed at accelerating research in social media-based meaning shift. Our results show that TempoWiC is a challenging benchmark, even for recently-released language models specialized in social media.

Related papers

• A Systematic Analysis on the Temporal Generalization of Language Models in Social Media [12.035331011654078]
  This paper focuses on temporal shifts in social media and, in particular, Twitter. We propose a unified evaluation scheme to assess the performance of language models (LMs) under temporal shift.
  arXiv  Detail & Related papers  (2024-05-15T05:41:06Z)
• SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research [33.698581876383074]
  We introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.
  arXiv  Detail & Related papers  (2023-10-23T09:48:25Z)
• ManiTweet: A New Benchmark for Identifying Manipulation of News on Social Media [74.93847489218008]
  We present a novel task, identifying manipulation of news on social media, which aims to detect manipulation in social media posts and identify manipulated or inserted information. To study this task, we have proposed a data collection schema and curated a dataset called ManiTweet, consisting of 3.6K pairs of tweets and corresponding articles. Our analysis demonstrates that this task is highly challenging, with large language models (LLMs) yielding unsatisfactory performance.
  arXiv  Detail & Related papers  (2023-05-23T16:40:07Z)
• Contextualizing Emerging Trends in Financial News Articles [2.9483477138814287]
  We focus on emerging trends detection in financial news articles about Microsoft, collected before and during the start of the COVID-19 pandemic. We make the dataset accessible and propose a strong baseline for exploring the dynamics of similarities between pairs of keywords. We evaluate against a gold standard (Google Trends) and present noteworthy real-world scenarios regarding the influence of the pandemic on Microsoft.
  arXiv  Detail & Related papers  (2023-01-20T12:56:52Z)
• Time Will Change Things: An Empirical Study on Dynamic Language Understanding in Social Media Classification [5.075802830306718]
  We empirically study social media NLU in a dynamic setup, where models are trained on the past data and test on the future. We show that auto-encoding and pseudo-labeling collaboratively show the best robustness in dynamicity.
  arXiv  Detail & Related papers  (2022-10-06T12:18:28Z)
• SCoT: Sense Clustering over Time: a tool for the analysis of lexical change [79.80787569986283]
  We present Sense Clustering over Time (SCoT), a novel network-based tool for analysing lexical change. SCoT represents the meanings of a word as clusters of similar words. It has been successfully used in a European study on the changing meaning of crisis'
  arXiv  Detail & Related papers  (2022-03-18T12:04:09Z)
• TimeLMs: Diachronic Language Models from Twitter [22.166929118357714]
  We present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models' capacity to deal with future and out-of-distribution tweets.
  arXiv  Detail & Related papers  (2022-02-08T12:47:38Z)
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics [66.96150429230035]
  We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models.
  arXiv  Detail & Related papers  (2021-02-02T18:42:05Z)
• Named Entity Recognition for Social Media Texts with Semantic Augmentation [70.44281443975554]
  Existing approaches for named entity recognition suffer from data sparsity problems when conducted on short and informal texts. We propose a neural-based approach to NER for social media texts where both local (from running text) and augmented semantics are taken into account.
  arXiv  Detail & Related papers  (2020-10-29T10:06:46Z)
• Temporal Common Sense Acquisition with Minimal Supervision [77.8308414884754]
  This work proposes a novel sequence modeling approach that exploits explicit and implicit mentions of temporal common sense. Our method is shown to give quality predictions of various dimensions of temporal common sense. It also produces representations of events for relevant tasks such as duration comparison, parent-child relations, event coreference and temporal QA.
  arXiv  Detail & Related papers  (2020-05-08T22:20:16Z)
• I Know Where You Are Coming From: On the Impact of Social Media Sources on AI Model Performance [79.05613148641018]
  We will study the performance of different machine learning models when being learned on multi-modal data from different social networks. Our initial experimental results reveal that social network choice impacts the performance.
  arXiv  Detail & Related papers  (2020-02-05T11:10:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.