SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages

• URL: http://arxiv.org/abs/2403.18933v5
• Date: Wed, 17 Apr 2024 19:49:12 GMT
• Title: SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages
• Authors: Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, Saif M. Mohammad,
• Abstract summary: We present the first shared task on Semantic Textual Relatedness (STR) We investigate the broader phenomenon of semantic relatedness across 14 languages. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia.
• Score: 39.770050337720676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phenomenon of semantic relatedness across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by the relatively limited availability of NLP resources. Each instance in the datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. Participating systems were asked to rank sentence pairs by their closeness in meaning (i.e., their degree of semantic relatedness) in the 14 languages in three main tracks: (a) supervised, (b) unsupervised, and (c) crosslingual. The task attracted 163 participants. We received 70 submissions in total (across all tasks) from 51 different teams, and 38 system description papers. We report on the best-performing systems as well as the most common and the most effective approaches for the three different tracks.

Related papers

• Wav2Gloss: Generating Interlinear Glossed Text from Speech [78.64412090339044]
  We propose Wav2Gloss, a task in which four linguistic annotation components are extracted automatically from speech. We provide various baselines to lay the groundwork for future research on Interlinear Glossed Text generation from speech.
  arXiv  Detail & Related papers  (2024-03-19T21:45:29Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [53.1913348687902]
  We present ArabicMMLU, the first multi-task language understanding benchmark for Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages [44.017657230247934]
  We present textitSemRel, a new semantic relatedness dataset collection annotated by native speakers across 13 languages. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences.
  arXiv  Detail & Related papers  (2024-02-13T18:04:53Z)
• Mukhyansh: A Headline Generation Dataset for Indic Languages [4.583536403673757]
  Mukhyansh is an extensive multilingual dataset, tailored for Indian language headline generation. Comprising over 3.39 million article-headline pairs, Mukhyansh spans across eight prominent Indian languages. Mukhyansh outperforms all other models, achieving an average ROUGE-L score of 31.43 across all 8 languages.
  arXiv  Detail & Related papers  (2023-11-29T15:49:24Z)
• SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval) [42.140064297754634]
  AfriSenti-SemEval is a sentiment classification challenge in 14 African languages. We present three subtasks: (1) Task A: monolingual classification, which received 44 submissions; (2) Task B: multilingual classification, which received 32 submissions; and (3) Task C: zero-shot classification, which received 34 submissions.
  arXiv  Detail & Related papers  (2023-04-13T22:26:10Z)
• Comprehensive Benchmark Datasets for Amharic Scene Text Detection and Recognition [56.048783994698425]
  Ethiopic/Amharic script is one of the oldest African writing systems, which serves at least 23 languages in East Africa. The Amharic writing system, Abugida, has 282 syllables, 15 punctuation marks, and 20 numerals. We presented the first comprehensive public datasets named HUST-ART, HUST-AST, ABE, and Tana for Amharic script detection and recognition in the natural scene.
  arXiv  Detail & Related papers  (2022-03-23T03:19:35Z)
• Arabic Dialect Identification Using BERT-Based Domain Adaptation [0.0]
  Arabic is one of the most important and growing languages in the world. With the rise of social media platforms such as Twitter, Arabic spoken dialects have become more in use.
  arXiv  Detail & Related papers  (2020-11-13T15:52:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.