Constructing Multilingual Code Search Dataset Using Neural Machine Translation

• URL: http://arxiv.org/abs/2306.15604v1
• Date: Tue, 27 Jun 2023 16:42:36 GMT
• Title: Constructing Multilingual Code Search Dataset Using Neural Machine Translation
• Authors: Ryo Sekizawa, Nan Duan, Shuai Lu, Hitomi Yanaka
• Abstract summary: We create a multilingual code search dataset in four natural and four programming languages. Our results show that the model pre-trained with all natural and programming language data has performed best in most cases.
• Score: 48.32329232202801
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code search is a task to find programming codes that semantically match the given natural language queries. Even though some of the existing datasets for this task are multilingual on the programming language side, their query data are only in English. In this research, we create a multilingual code search dataset in four natural and four programming languages using a neural machine translation model. Using our dataset, we pre-train and fine-tune the Transformer-based models and then evaluate them on multiple code search test sets. Our results show that the model pre-trained with all natural and programming language data has performed best in most cases. By applying back-translation data filtering to our dataset, we demonstrate that the translation quality affects the model's performance to a certain extent, but the data size matters more.

Related papers

• GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding [5.9535699822923]
  We propose a new benchmark dataset called GenCodeSearchNet (GeCS) to evaluate the programming language understanding capabilities of language models. As part of the full dataset, we introduce a new, manually curated subset StatCodeSearch that focuses on R, a popular but so far underrepresented programming language. For evaluation and comparison, we collect several baseline results using fine-tuned BERT-style models and GPT-style large language models.
  arXiv  Detail & Related papers  (2023-11-16T09:35:00Z)
• Natural Language Models for Data Visualization Utilizing nvBench Dataset [6.996262696260261]
  We build natural language translation models to construct simplified versions of data and visualization queries in a language called Vega Zero. In this paper, we explore the design and performance of these sequences to sequence transformer based machine learning model architectures.
  arXiv  Detail & Related papers  (2023-10-02T00:48:01Z)
• Multi-lingual Evaluation of Code Generation Models [82.7357812992118]
  We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages. We are able to assess the performance of code generation models in a multi-lingual fashion.
  arXiv  Detail & Related papers  (2022-10-26T17:17:06Z)
• Multilingual Multimodal Learning with Machine Translated Text [27.7207234512674]
  We investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning.
  arXiv  Detail & Related papers  (2022-10-24T11:41:20Z)
• Using Document Similarity Methods to create Parallel Datasets for Code Translation [60.36392618065203]
  Translating source code from one programming language to another is a critical, time-consuming task. We propose to use document similarity methods to create noisy parallel datasets of code. We show that these models perform comparably to models trained on ground truth for reasonable levels of noise.
  arXiv  Detail & Related papers  (2021-10-11T17:07:58Z)
• Multilingual Neural Semantic Parsing for Low-Resourced Languages [1.6244541005112747]
  We introduce a new multilingual semantic parsing dataset in English, Italian and Japanese. We show that joint multilingual training with pretrained encoders substantially outperforms our baselines on the TOP dataset. We find that a semantic trained only on English data achieves a zero-shot performance of 44.9% exact-match accuracy on Italian sentences.
  arXiv  Detail & Related papers  (2021-06-07T09:53:02Z)
• CoDesc: A Large Code-Description Parallel Dataset [4.828053113572208]
  We present CoDesc -- a large parallel dataset composed of 4.2 million Java methods and natural language descriptions. With extensive analysis, we identify and remove prevailing noise patterns from the dataset. We show that the dataset helps improve code search by up to 22% and achieves the new state-of-the-art in code summarization.
  arXiv  Detail & Related papers  (2021-05-29T05:40:08Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Code to Comment "Translation": Data, Metrics, Baselining & Evaluation [49.35567240750619]
  We analyze several recent code-comment datasets for this task. We compare them with WMT19, a standard dataset frequently used to train state of the art natural language translators. We find some interesting differences between the code-comment data and the WMT19 natural language data.
  arXiv  Detail & Related papers  (2020-10-03T18:57:26Z)
• A Multi-Perspective Architecture for Semantic Code Search [58.73778219645548]
  We propose a novel multi-perspective cross-lingual neural framework for code--text matching. Our experiments on the CoNaLa dataset show that our proposed model yields better performance than previous approaches.
  arXiv  Detail & Related papers  (2020-05-06T04:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.