Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities

• URL: http://arxiv.org/abs/2210.16960v1
• Date: Sun, 30 Oct 2022 21:46:01 GMT
• Title: Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities
• Authors: Khyathi Raghavi Chandu, Alborz Geramifard
• Abstract summary: We study the unification of multilingual and multimodal (MultiX) streams. We review the languages studied, gold or silver data with parallel annotations, and understand how these modalities and languages interact in modeling. We present an account of the modeling approaches along with their strengths and weaknesses to better understand what scenarios they can be used reliably.
• Score: 10.721189858694396
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contextualizing language technologies beyond a single language kindled embracing multiple modalities and languages. Individually, each of these directions undoubtedly proliferated into several NLP tasks. Despite this momentum, most of the multimodal research is primarily centered around English and multilingual research is primarily centered around contexts from text modality. Challenging this conventional setup, researchers studied the unification of multilingual and multimodal (MultiX) streams. The main goal of this work is to catalogue and characterize these works by charting out the categories of tasks, datasets and methods to address MultiX scenarios. To this end, we review the languages studied, gold or silver data with parallel annotations, and understand how these modalities and languages interact in modeling. We present an account of the modeling approaches along with their strengths and weaknesses to better understand what scenarios they can be used reliably. Following this, we present the high-level trends in the overall paradigm of the field. Finally, we conclude by presenting a road map of challenges and promising research directions.

Related papers

• Language and Multimodal Models in Sports: A Survey of Datasets and Applications [20.99857526324661]
  Recent integration of Natural Language Processing (NLP) and multimodal models has advanced the field of sports analytics. This survey presents a comprehensive review of the datasets and applications driving these innovations post-2020.
  arXiv  Detail & Related papers  (2024-06-18T03:59:26Z)
• A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers [48.314619377988436]
  The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing. Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient. This survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.
  arXiv  Detail & Related papers  (2024-05-17T17:47:39Z)
• Understanding Cross-Lingual Alignment -- A Survey [52.572071017877704]
  Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field.
  arXiv  Detail & Related papers  (2024-04-09T11:39:53Z)
• Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers [81.47046536073682]
  We present a review and provide a unified perspective to summarize the recent progress as well as emerging trends in multilingual large language models (MLLMs) literature. We hope our work can provide the community with quick access and spur breakthrough research in MLLMs.
  arXiv  Detail & Related papers  (2024-04-07T11:52:44Z)
• M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets [4.478789600295492]
  This paper transforms an existing textual Twitter sentiment dataset into a multimodal format through a straightforward curation process. Our work opens up new avenues for sentiment-related research within the research community.
  arXiv  Detail & Related papers  (2024-04-02T09:11:58Z)
• Multimodal Large Language Models: A Survey [36.06016060015404]
  Multimodal language models integrate multiple data types, such as images, text, language, audio, and other heterogeneity. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. A practical guide is provided, offering insights into the technical aspects of multimodal models. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development.
  arXiv  Detail & Related papers  (2023-11-22T05:15:12Z)
• Cross-lingual Lifelong Learning [53.06904052325966]
  We present a principled Cross-lingual Continual Learning (CCL) evaluation paradigm. We provide insights into what makes multilingual sequential learning particularly challenging. The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata.
  arXiv  Detail & Related papers  (2022-05-23T09:25:43Z)
• Towards Best Practices for Training Multilingual Dense Retrieval Models [54.91016739123398]
  We focus on the task of monolingual retrieval in a variety of typologically diverse languages using one such design. Our study is organized as a "best practices" guide for training multilingual dense retrieval models.
  arXiv  Detail & Related papers  (2022-04-05T17:12:53Z)
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization [128.37244072182506]
  Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
  arXiv  Detail & Related papers  (2020-03-24T19:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.