Patch-CLIP: A Patch-Text Pre-Trained Model

• URL: http://arxiv.org/abs/2310.12753v2
• Date: Sun, 31 Mar 2024 00:59:30 GMT
• Title: Patch-CLIP: A Patch-Text Pre-Trained Model
• Authors: Xunzhu Tang, Zhenghan Chen, Saad Ezzini, Haoye Tian, Jacques Klein, Tegawende F. Bissyande,
• Abstract summary: patch representation learning has emerged as a necessary research direction for exploiting the capabilities of machine learning in software generation. We introduce. theweak-CLIP, a novel pre-training framework for patches and natural language text. We show that. theweak-CLIP sets new state-of-the-art performance, consistently outperforming the state-of-the-art in metrics like BLEU, ROUGE-L, METEOR, and Recall.
• Score: 6.838615442552715
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, patch representation learning has emerged as a necessary research direction for exploiting the capabilities of machine learning in software generation. These representations have driven significant performance enhancements across a variety of tasks involving code changes. While the progress is undeniable, a common limitation among existing models is their specialization: they predominantly excel in either predictive tasks, such as security patch classification, or in generative tasks such as patch description generation. This dichotomy is further exacerbated by a prevalent dependency on potentially noisy data sources. Specifically, many models utilize patches integrated with Abstract Syntax Trees (AST) that, unfortunately, may contain parsing inaccuracies, thus acting as a suboptimal source of supervision. In response to these challenges, we introduce PATCH-CLIP, a novel pre-training framework for patches and natural language text. PATCH-CLIP deploys a triple-loss training strategy for 1) patch-description contrastive learning, which enables to separate patches and descriptions in the embedding space, 2) patch-description matching, which ensures that each patch is associated to its description in the embedding space, and 3) patch-description generation, which ensures that the patch embedding is effective for generation. These losses are implemented for joint learning to achieve good performance in both predictive and generative tasks involving patches. Empirical evaluations focusing on patch description generation, demonstrate that PATCH-CLIP sets new state of the art performance, consistently outperforming the state-of-the-art in metrics like BLEU, ROUGE-L, METEOR, and Recall.

Related papers

• Semi-supervised 3D Object Detection with PatchTeacher and PillarMix [71.4908268136439]
  Current semi-supervised 3D object detection methods typically use a teacher to generate pseudo labels for a student. We propose PatchTeacher, which focuses on partial scene 3D object detection to provide high-quality pseudo labels for the student. We introduce three key techniques, i.e., Patch Normalizer, Quadrant Align, and Fovea Selection, to improve the performance of PatchTeacher.
  arXiv  Detail & Related papers  (2024-07-13T06:58:49Z)
• Automated Description Generation for Software Patches [4.080365651252764]
  We present EXPLAINER, a machine translation approach to generating software patch descriptions. In EXPLAINER, we leverage explicit representations of critical elements, historical context, and syntactic conventions. We show that EXPLAINER consistently outperforms existing methods, with improvements up to 189% in BLEU, 5.7X in Exact Match rate, and 154% in Semantic Similarity.
  arXiv  Detail & Related papers  (2024-02-06T08:46:14Z)
• Learning to Embed Time Series Patches Independently [5.752266579415516]
  Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series. We argue that capturing such patch might not be an optimal strategy for time series representation learning. We propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise reconstruction that embeds each patch independently.
  arXiv  Detail & Related papers  (2023-12-27T06:23:29Z)
• MS-Former: Memory-Supported Transformer for Weakly Supervised Change Detection with Patch-Level Annotations [50.79913333804232]
  We propose a memory-supported transformer (MS-Former) for weakly supervised change detection. MS-Former consists of a bi-directional attention block (BAB) and a patch-level supervision scheme (PSS) Experimental results on three benchmark datasets demonstrate the effectiveness of our proposed method in the change detection task.
  arXiv  Detail & Related papers  (2023-11-16T09:57:29Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• Learning to Represent Patches [7.073203009308308]
  We introduce a novel method, Patcherizer, to bridge the gap between deep learning for patch representation and semantic intent. Patcherizer employs graph convolutional neural networks for structural intention graph representation and transformers for intention sequence representation. Our experiments demonstrate the representation's efficacy across all tasks, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-31T09:34:38Z)
• KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair [33.04645845117822]
  Automated Program Repair (APR) improves software reliability by generating patches for a buggy program automatically. Recent APR techniques leverage deep learning (DL) to build models to learn to generate patches from existing patches and code corpora. We propose a DL-based APR approach, which incorporates domain knowledge to guide patch generation in a direct and comprehensive way.
  arXiv  Detail & Related papers  (2023-02-03T17:02:56Z)
• UIA-ViT: Unsupervised Inconsistency-Aware Method based on Vision Transformer for Face Forgery Detection [52.91782218300844]
  We propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning.
  arXiv  Detail & Related papers  (2022-10-23T15:24:47Z)
• Patch-level Representation Learning for Self-supervised Vision Transformers [68.8862419248863]
  Vision Transformers (ViTs) have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks. Inspired by this, we design a simple yet effective visual pretext task, coined SelfPatch, for learning better patch-level representations. We demonstrate that SelfPatch can significantly improve the performance of existing SSL methods for various visual tasks.
  arXiv  Detail & Related papers  (2022-06-16T08:01:19Z)
• Rethinking Generative Zero-Shot Learning: An Ensemble Learning Perspective for Recognising Visual Patches [52.67723703088284]
  We propose a novel framework called multi-patch generative adversarial nets (MPGAN) MPGAN synthesises local patch features and labels unseen classes with a novel weighted voting strategy. MPGAN has significantly greater accuracy than state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-27T05:49:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.