HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding

• URL: http://arxiv.org/abs/2405.06204v1
• Date: Fri, 10 May 2024 02:40:49 GMT
• Title: HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding
• Authors: Bowen Xing, Ivor W. Tsang,
• Abstract summary: State-of-the-art model for cross-lingual spoken language understanding performs cross-lingual unsupervised contrastive learning. We propose Hybrid and Cooperative Contrastive Learning to address this problem.
• Score: 45.12153788010354
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State-of-the-art model for zero-shot cross-lingual spoken language understanding performs cross-lingual unsupervised contrastive learning to achieve the label-agnostic semantic alignment between each utterance and its code-switched data. However, it ignores the precious intent/slot labels, whose label information is promising to help capture the label-aware semantics structure and then leverage supervised contrastive learning to improve both source and target languages' semantics. In this paper, we propose Hybrid and Cooperative Contrastive Learning to address this problem. Apart from cross-lingual unsupervised contrastive learning, we design a holistic approach that exploits source language supervised contrastive learning, cross-lingual supervised contrastive learning and multilingual supervised contrastive learning to perform label-aware semantics alignments in a comprehensive manner. Each kind of supervised contrastive learning mechanism includes both single-task and joint-task scenarios. In our model, one contrastive learning mechanism's input is enhanced by others. Thus the total four contrastive learning mechanisms are cooperative to learn more consistent and discriminative representations in the virtuous cycle during the training process. Experiments show that our model obtains consistent improvements over 9 languages, achieving new state-of-the-art performance.

Related papers

• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer [4.944761231728674]
  We present a novel framework called "Lexicon-Syntax Enhanced Multilingual BERT" We use Multilingual BERT as the base model and employ two techniques to enhance its learning capabilities. Our experimental results demonstrate this framework can consistently outperform all baselines of zero-shot cross-lingual transfer.
  arXiv  Detail & Related papers  (2024-04-25T14:10:52Z)
• mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning [54.523172171533645]
  Cross-lingual named entity recognition (CrossNER) faces challenges stemming from uneven performance due to the scarcity of multilingual corpora. We propose Multi-view Contrastive Learning for Cross-lingual Named Entity Recognition (mCL-NER) Our experiments on the XTREME benchmark, spanning 40 languages, demonstrate the superiority of mCL-NER over prior data-driven and model-based approaches.
  arXiv  Detail & Related papers  (2023-08-17T16:02:29Z)
• Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning [80.08139343603956]
  In cross-lingual named entity recognition, self-training is commonly used to bridge the linguistic gap. In this work, we aim to improve self-training for cross-lingual NER by combining representation learning and pseudo label refinement. Our proposed method, namely ContProto mainly comprises two components: (1) contrastive self-training and (2) prototype-based pseudo-labeling.
  arXiv  Detail & Related papers  (2023-05-23T02:52:16Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation [8.718749742587857]
  Cross-lingual summarization models rely on the self-attention mechanism to attend among tokens in two languages. We propose a novel Knowledge-Distillation-based framework for Cross-Lingual Summarization. Our method outperforms state-of-the-art models under both high and low-resourced settings.
  arXiv  Detail & Related papers  (2021-12-07T03:45:02Z)
• Robust Mutual Learning for Semi-supervised Semantic Segmentation [23.54885398483688]
  We propose robust mutual learning that improves the prior approach in two aspects. We show that strong data augmentations, model noises and heterogeneous network architectures are essential to alleviate the model coupling. The proposed robust mutual learning demonstrates state-of-the-art performance on semantic segmentation in low-data regime.
  arXiv  Detail & Related papers  (2021-06-01T16:22:01Z)
• Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling [12.497781134446898]
  We address the data scarcity and limitations of linguistic theory by proposing language-agnostic multi-task training methods. First, we introduce a meta-learning-based approach, meta-transfer learning, in which information is judiciously extracted from high-resource monolingual speech data to the code-switching domain. Second, we propose a novel multilingual meta-ems approach to effectively represent code-switching data by acquiring useful knowledge learned in other languages. Third, we introduce multi-task learning to integrate syntactic information as a transfer learning strategy to a language model and learn where to code-switch.
  arXiv  Detail & Related papers  (2021-04-13T14:49:26Z)
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training [135.12061144759517]
  We present an information-theoretic framework that formulates cross-lingual language model pre-training. We propose a new pre-training task based on contrastive learning. By leveraging both monolingual and parallel corpora, we jointly train the pretext to improve the cross-lingual transferability of pre-trained models.
  arXiv  Detail & Related papers  (2020-07-15T16:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.