Assessing and Improving Syntactic Adversarial Robustness of Pre-trained Models for Code Translation

• URL: http://arxiv.org/abs/2310.18587v1
• Date: Sat, 28 Oct 2023 04:35:24 GMT
• Title: Assessing and Improving Syntactic Adversarial Robustness of Pre-trained Models for Code Translation
• Authors: Guang Yang, Yu Zhou, Xiangyu Zhang, Xiang Chen, Tingting Han, Taolue Chen
• Abstract summary: CoTR consists of two components: CoTR-A and CoTR-D. The effectiveness of CoTR is evaluated through experiments on real world Java to Python datasets.
• Score: 19.186392871168064
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context: Pre-trained models (PTMs) have demonstrated significant potential in automatic code translation. However, the vulnerability of these models in translation tasks, particularly in terms of syntax, has not been extensively investigated. Objective: To fill this gap, our study aims to propose a novel approach CoTR to assess and improve the syntactic adversarial robustness of PTMs in code translation. Method: CoTR consists of two components: CoTR-A and CoTR-D. CoTR-A generates adversarial examples by transforming programs, while CoTR-D proposes a semantic distance-based sampling data augmentation method and adversarial training method to improve the model's robustness and generalization capabilities. The Pass@1 metric is used by CoTR to assess the performance of PTMs, which is more suitable for code translation tasks and offers a more precise evaluation in real world scenarios. Results: The effectiveness of CoTR is evaluated through experiments on real world Java to Python datasets. The results demonstrate that CoTR-A can significantly reduce the performance of existing PTMs, while CoTR-D effectively improves the robustness of PTMs. Conclusion: Our study identifies the limitations of current PTMs, including large language models, in code translation tasks. It highlights the potential of CoTR as an effective solution to enhance the robustness of PTMs for code translation tasks.

Related papers

• Fractional Correspondence Framework in Detection Transformer [13.388933240897492]
  The Detection Transformer (DETR) has significantly simplified the matching process in object detection tasks. This algorithm facilitates optimal one-to-one matching of predicted bounding boxes to ground-truth annotations during training. We propose a flexible matching strategy that captures the cost of aligning predictions with ground truths to find the most accurate correspondences.
  arXiv  Detail & Related papers  (2025-03-06T05:29:20Z)
• CTR-KAN: KAN for Adaptive High-Order Feature Interaction Modeling [37.80127625183842]
  CTR-KAN is an adaptive framework for efficient high-order feature interaction modeling. It builds upon the Kolmogorov-Arnold Network (KAN) paradigm, addressing its limitations in CTR prediction tasks. CTR-KAN achieves state-of-the-art predictive accuracy with significantly lower computational costs.
  arXiv  Detail & Related papers  (2024-08-16T12:51:52Z)
• Patched RTC: evaluating LLMs for diverse software development tasks [1.14219428942199]
  This paper introduces Patched Round-Trip Correctness (Patched RTC), a novel evaluation technique for Large Language Models (LLMs) Patched RTC offers a self-evaluating framework that measures consistency and robustness of model responses without human intervention. Experiments comparing GPT-3.5 and GPT-4 models across different software development tasks reveal that Patched RTC effectively distinguishes model performance and task difficulty.
  arXiv  Detail & Related papers  (2024-07-23T15:12:14Z)
• Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spotter [57.64003871384959]
  This work presents a new approach to fast context-biasing with CTC-based Word Spotter. The proposed method matches CTC log-probabilities against a compact context graph to detect potential context-biasing candidates. The results demonstrate a significant acceleration of the context-biasing recognition with a simultaneous improvement in F-score and WER.
  arXiv  Detail & Related papers  (2024-06-11T09:37:52Z)
• CELA: Cost-Efficient Language Model Alignment for CTR Prediction [71.85120354973073]
  Click-Through Rate (CTR) prediction holds a paramount position in recommender systems. Recent efforts have sought to mitigate these challenges by integrating Pre-trained Language Models (PLMs) We propose textbfCost-textbfEfficient textbfLanguage Model textbfAlignment (textbfCELA) for CTR prediction.
  arXiv  Detail & Related papers  (2024-05-17T07:43:25Z)
• Markovian Transformers for Informative Language Modeling [0.9642500063568188]
  Chain-of-Thought (CoT) reasoning holds great promise for explaining the outputs of language models. Recent studies have highlighted significant challenges in its practical application for interpretability. We propose a technique to factor next-token prediction through intermediate CoT text, ensuring the CoT is causally load-bearing.
  arXiv  Detail & Related papers  (2024-04-29T17:36:58Z)
• ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting [124.69672273754144]
  Chain-of-Thought (CoT) prompting can enhance the reasoning capabilities of large language models (LLMs) Existing CoT approaches usually focus on simpler reasoning tasks and thus result in low-quality and inconsistent CoT prompts. We introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts.
  arXiv  Detail & Related papers  (2024-03-21T11:34:26Z)
• ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction [45.15127775876369]
  Click-through rate (CTR) prediction has become increasingly indispensable for various Internet applications. Traditional CTR models convert the multi-field categorical data into ID features via one-hot encoding, and extract the collaborative signals among features. We propose a novel model-agnostic framework (i.e., ClickPrompt) where we incorporate CTR models to generate interaction-aware soft prompts.
  arXiv  Detail & Related papers  (2023-10-13T16:37:53Z)
• DELTA: Dynamic Embedding Learning with Truncated Conscious Attention for CTR Prediction [61.68415731896613]
  Click-Through Rate (CTR) prediction is a pivotal task in product and content recommendation. We propose a model that enables Dynamic Embedding Learning with Truncated Conscious Attention for CTR prediction.
  arXiv  Detail & Related papers  (2023-05-03T12:34:45Z)
• Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation [64.16077929617119]
  We propose a new criterion for NMT adversarial examples based on the Doubly Round-Trip Translation (DRTT) To enhance the robustness of the NMT model, we introduce the masked language models to construct bilingual adversarial pairs.
  arXiv  Detail & Related papers  (2022-04-19T06:15:27Z)
• Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation [98.11249019844281]
  Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation (NAT) models. We propose reverse KD to rejuvenate more alignments for low-frequency target words. Results demonstrate that the proposed approach can significantly and universally improve translation quality.
  arXiv  Detail & Related papers  (2021-06-02T02:41:40Z)
• Modeling Coverage for Non-Autoregressive Neural Machine Translation [9.173385214565451]
  We propose a novel Coverage-NAT to model the coverage information directly by a token-level coverage iterative refinement mechanism and a sentence-level coverage agreement. Experimental results on WMT14 En-De and WMT16 En-Ro translation tasks show that our method can alleviate those errors and achieve strong improvements over the baseline system.
  arXiv  Detail & Related papers  (2021-04-24T07:33:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.