MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation Learning

• URL: http://arxiv.org/abs/2206.06460v1
• Date: Mon, 13 Jun 2022 20:36:42 GMT
• Title: MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation Learning
• Authors: Weiguo Pian, Hanyu Peng, Xunzhu Tang, Tiezhu Sun, Haoye Tian, Andrew Habib, Jacques Klein, Tegawend\'e F. Bissyand\'e
• Abstract summary: We propose MetaTPTrans, a meta learning approach for multilingual code representation learning. We show that MetaTPTrans improves the F1 score of state-of-the-art approaches significantly.
• Score: 5.434698132994918
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representation learning of source code is essential for applying machine learning to software engineering tasks. Learning code representation across different programming languages has been shown to be more effective than learning from single-language datasets, since more training data from multi-language datasets improves the model's ability to extract language-agnostic information from source code. However, existing multi-language models overlook the language-specific information which is crucial for downstream tasks that is training on multi-language datasets, while only focusing on learning shared parameters among the different languages. To address this problem, we propose MetaTPTrans, a meta learning approach for multilingual code representation learning. MetaTPTrans generates different parameters for the feature extractor according to the specific programming language of the input source code snippet, enabling the model to learn both language-agnostics and language-specific information. Experimental results show that MetaTPTrans improves the F1 score of state-of-the-art approaches significantly by up to 2.40 percentage points for code summarization, a language-agnostic task; and the prediction accuracy of Top-1 (Top-5) by up to 7.32 (13.15) percentage points for code completion, a language-specific task.

Related papers

• IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators [49.903001442804594]
  This work investigates the prospect of leveraging compiler intermediate representations (IR) to improve the multilingual capabilities of Code-LMs. We first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files. Next, we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to learn the IR language. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics.
  arXiv  Detail & Related papers  (2024-03-06T17:52:08Z)
• Learning Transfers over Several Programming Languages [5.350495525141013]
  Cross-lingual transfer uses data from a source language to improve model performance on a target language. This paper reports extensive experiments on four tasks using a transformer-based large language model and 11 to 41 programming languages. We find that learning transfers well across several programming languages.
  arXiv  Detail & Related papers  (2023-10-25T19:04:33Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learning [0.7242530499990028]
  Code-switching is the linguistics phenomenon where in casual settings, multilingual speakers mix words from different languages in one utterance. We propose two novel approaches toward improving language identification accuracy on an English-Mandarin child-directed speech dataset. Our best model achieves a balanced accuracy of 0.781 on a real English-Mandarin code-switching child-directed speech corpus and outperforms the previous baseline by 55.3%.
  arXiv  Detail & Related papers  (2023-05-31T11:43:16Z)
• Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling [12.497781134446898]
  We address the data scarcity and limitations of linguistic theory by proposing language-agnostic multi-task training methods. First, we introduce a meta-learning-based approach, meta-transfer learning, in which information is judiciously extracted from high-resource monolingual speech data to the code-switching domain. Second, we propose a novel multilingual meta-ems approach to effectively represent code-switching data by acquiring useful knowledge learned in other languages. Third, we introduce multi-task learning to integrate syntactic information as a transfer learning strategy to a language model and learn where to code-switch.
  arXiv  Detail & Related papers  (2021-04-13T14:49:26Z)
• X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models [103.75890012041366]
  Language models (LMs) have proven surprisingly successful at capturing factual knowledge. However, studies on LMs' factual representation ability have almost invariably been performed on English. We create a benchmark of cloze-style probes for 23 typologically diverse languages.
  arXiv  Detail & Related papers  (2020-10-13T05:29:56Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)
• Meta-Transfer Learning for Code-Switched Speech Recognition [72.84247387728999]
  We propose a new learning method, meta-transfer learning, to transfer learn on a code-switched speech recognition system in a low-resource setting. Our model learns to recognize individual languages, and transfer them so as to better recognize mixed-language speech by conditioning the optimization on the code-switching data.
  arXiv  Detail & Related papers  (2020-04-29T14:27:19Z)
• Learning to Scale Multilingual Representations for Vision-Language Tasks [51.27839182889422]
  The effectiveness of SMALR is demonstrated with ten diverse languages, over twice the number supported in vision-language tasks to date. We evaluate on multilingual image-sentence retrieval and outperform prior work by 3-4% with less than 1/5th the training parameters compared to other word embedding methods.
  arXiv  Detail & Related papers  (2020-04-09T01:03:44Z)
• Zero-Shot Cross-Lingual Transfer with Meta Learning [45.29398184889296]
  We consider the setting of training models on multiple languages at the same time, when little or no data is available for languages other than English. We show that this challenging setup can be approached using meta-learning. We experiment using standard supervised, zero-shot cross-lingual, as well as few-shot cross-lingual settings for different natural language understanding tasks.
  arXiv  Detail & Related papers  (2020-03-05T16:07:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.