SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages

• URL: http://arxiv.org/abs/2403.18933v5
• Date: Wed, 17 Apr 2024 19:49:12 GMT
• Title: SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages
• Authors: Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, Saif M. Mohammad,
• Abstract summary: We present the first shared task on Semantic Textual Relatedness (STR) We investigate the broader phenomenon of semantic relatedness across 14 languages. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia.
• Score: 39.770050337720676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phenomenon of semantic relatedness across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by the relatively limited availability of NLP resources. Each instance in the datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. Participating systems were asked to rank sentence pairs by their closeness in meaning (i.e., their degree of semantic relatedness) in the 14 languages in three main tracks: (a) supervised, (b) unsupervised, and (c) crosslingual. The task attracted 163 participants. We received 70 submissions in total (across all tasks) from 51 different teams, and 38 system description papers. We report on the best-performing systems as well as the most common and the most effective approaches for the three different tracks.

Related papers

• BhasaAnuvaad: A Speech Translation Dataset for 14 Indian Languages [27.273651323572786]
  We evaluate the performance of widely-used Automatic Speech Translation systems on Indian languages. There is a striking absence of systems capable of accurately translating colloquial and informal language. We introduce BhasaAnuvaad, the largest publicly available dataset for AST involving 14 scheduled Indian languages.
  arXiv  Detail & Related papers  (2024-11-07T13:33:34Z)
• SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages [77.75535024869224]
  We present SeaLLMs 3, the latest iteration of the SeaLLMs model family, tailored for Southeast Asian languages. SeaLLMs 3 aims to bridge this gap by covering a comprehensive range of languages spoken in this region, including English, Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao, Tamil, and Javanese. Our model excels in tasks such as world knowledge, mathematical reasoning, translation, and instruction following, achieving state-of-the-art performance among similarly sized models.
  arXiv  Detail & Related papers  (2024-07-29T03:26:22Z)
• Wav2Gloss: Generating Interlinear Glossed Text from Speech [78.64412090339044]
  We propose Wav2Gloss, a task in which four linguistic annotation components are extracted automatically from speech. We provide various baselines to lay the groundwork for future research on Interlinear Glossed Text generation from speech.
  arXiv  Detail & Related papers  (2024-03-19T21:45:29Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages [44.017657230247934]
  We present textitSemRel, a new semantic relatedness dataset collection annotated by native speakers across 13 languages. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences.
  arXiv  Detail & Related papers  (2024-02-13T18:04:53Z)
• Mukhyansh: A Headline Generation Dataset for Indic Languages [4.583536403673757]
  Mukhyansh is an extensive multilingual dataset, tailored for Indian language headline generation. Comprising over 3.39 million article-headline pairs, Mukhyansh spans across eight prominent Indian languages. Mukhyansh outperforms all other models, achieving an average ROUGE-L score of 31.43 across all 8 languages.
  arXiv  Detail & Related papers  (2023-11-29T15:49:24Z)
• SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval) [42.140064297754634]
  AfriSenti-SemEval is a sentiment classification challenge in 14 African languages. We present three subtasks: (1) Task A: monolingual classification, which received 44 submissions; (2) Task B: multilingual classification, which received 32 submissions; and (3) Task C: zero-shot classification, which received 34 submissions.
  arXiv  Detail & Related papers  (2023-04-13T22:26:10Z)
• Comprehensive Benchmark Datasets for Amharic Scene Text Detection and Recognition [56.048783994698425]
  Ethiopic/Amharic script is one of the oldest African writing systems, which serves at least 23 languages in East Africa. The Amharic writing system, Abugida, has 282 syllables, 15 punctuation marks, and 20 numerals. We presented the first comprehensive public datasets named HUST-ART, HUST-AST, ABE, and Tana for Amharic script detection and recognition in the natural scene.
  arXiv  Detail & Related papers  (2022-03-23T03:19:35Z)
• Arabic Dialect Identification Using BERT-Based Domain Adaptation [0.0]
  Arabic is one of the most important and growing languages in the world. With the rise of social media platforms such as Twitter, Arabic spoken dialects have become more in use.
  arXiv  Detail & Related papers  (2020-11-13T15:52:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.