FonMTL: Towards Multitask Learning for the Fon Language

• URL: http://arxiv.org/abs/2308.14280v2
• Date: Mon, 11 Sep 2023 22:51:25 GMT
• Title: FonMTL: Towards Multitask Learning for the Fon Language
• Authors: Bonaventure F. P. Dossou, Iffanice Houndayi, Pamely Zantou, Gilles Hacheme
• Abstract summary: We present the first explorative approach to multitask learning, for model capabilities enhancement in Natural Language Processing for the Fon language. We leverage two language model heads as encoders to build shared representations for the inputs, and we use linear layers blocks for classification relative to each task. Our results on the NER and POS tasks for Fon, show competitive (or better) performances compared to several multilingual pretrained language models finetuned on single tasks.
• Score: 1.9370453715137865
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Fon language, spoken by an average 2 million of people, is a truly low-resourced African language, with a limited online presence, and existing datasets (just to name but a few). Multitask learning is a learning paradigm that aims to improve the generalization capacity of a model by sharing knowledge across different but related tasks: this could be prevalent in very data-scarce scenarios. In this paper, we present the first explorative approach to multitask learning, for model capabilities enhancement in Natural Language Processing for the Fon language. Specifically, we explore the tasks of Named Entity Recognition (NER) and Part of Speech Tagging (POS) for Fon. We leverage two language model heads as encoders to build shared representations for the inputs, and we use linear layers blocks for classification relative to each task. Our results on the NER and POS tasks for Fon, show competitive (or better) performances compared to several multilingual pretrained language models finetuned on single tasks. Additionally, we perform a few ablation studies to leverage the efficiency of two different loss combination strategies and find out that the equal loss weighting approach works best in our case. Our code is open-sourced at https://github.com/bonaventuredossou/multitask_fon.

Related papers

• 2M-NER: Contrastive Learning for Multilingual and Multimodal NER with Language and Modal Fusion [9.038363543966263]
  We construct a large-scale MMNER dataset with four languages (English, French, German and Spanish) and two modalities (text and image) We introduce a new model called 2M-NER, which aligns the text and image representations using contrastive learning and integrates a multimodal collaboration module. Our model achieves the highest F1 score in multilingual and multimodal NER tasks compared to some comparative and representative baselines.
  arXiv  Detail & Related papers  (2024-04-26T02:34:31Z)
• Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages [60.162717568496355]
  Large language models (LLMs) have been pre-trained on multilingual corpora. Their performance still lags behind in most languages compared to a few resource-rich languages.
  arXiv  Detail & Related papers  (2024-02-19T15:07:32Z)
• SkillNet-X: A Multilingual Multitask Model with Sparsely Activated Skills [51.74947795895178]
  This paper proposes a general multilingual multitask model, named SkillNet-X. We define several language-specific skills and task-specific skills, each of which corresponds to a skill module. We evaluate SkillNet-X on eleven natural language understanding datasets in four languages.
  arXiv  Detail & Related papers  (2023-06-28T12:53:30Z)
• Efficient Spoken Language Recognition via Multilabel Classification [53.662747523872305]
  We show that our models obtain competitive results while being orders of magnitude smaller and faster than current state-of-the-art methods. Our multilabel strategy is more robust to unseen non-target languages compared to multiclass classification.
  arXiv  Detail & Related papers  (2023-06-02T23:04:19Z)
• Few-shot Multimodal Multitask Multilingual Learning [0.0]
  We propose few-shot learning for a multimodal multitask multilingual (FM3) setting by adapting pre-trained vision and language models. FM3 learns the most prominent tasks in the vision and language domains along with their intersections.
  arXiv  Detail & Related papers  (2023-02-19T03:48:46Z)
• Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve [78.3500985535601]
  We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations. We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning.
  arXiv  Detail & Related papers  (2022-10-20T22:23:27Z)
• Adaptive Activation Network For Low Resource Multilingual Speech Recognition [30.460501537763736]
  We introduce an adaptive activation network to the upper layers of ASR model. We also proposed two approaches to train the model: (1) cross-lingual learning, replacing the activation function from source language to target language, and (2) multilingual learning. Our experiments on IARPA Babel datasets demonstrated that our approaches outperform the from-scratch training and traditional bottleneck feature based methods.
  arXiv  Detail & Related papers  (2022-05-28T04:02:59Z)
• Are Multilingual Models Effective in Code-Switching? [57.78477547424949]
  We study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching.
  arXiv  Detail & Related papers  (2021-03-24T16:20:02Z)
• Meta-Learning for Effective Multi-task and Multilingual Modelling [23.53779501937046]
  We propose a meta-learning approach to learn the interactions between both tasks and languages. We present experiments on five different tasks and six different languages from the XTREME multilingual benchmark dataset.
  arXiv  Detail & Related papers  (2021-01-25T19:30:26Z)
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP [68.2650714613869]
  We propose a data augmentation framework to generate multi-lingual code-switching data to fine-tune mBERT. Compared with the existing work, our method does not rely on bilingual sentences for training, and requires only one training process for multiple target languages.
  arXiv  Detail & Related papers  (2020-06-11T13:15:59Z)
• Zero-Shot Cross-Lingual Transfer with Meta Learning [45.29398184889296]
  We consider the setting of training models on multiple languages at the same time, when little or no data is available for languages other than English. We show that this challenging setup can be approached using meta-learning. We experiment using standard supervised, zero-shot cross-lingual, as well as few-shot cross-lingual settings for different natural language understanding tasks.
  arXiv  Detail & Related papers  (2020-03-05T16:07:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.