CoditT5: Pretraining for Source Code and Natural Language Editing

• URL: http://arxiv.org/abs/2208.05446v1
• Date: Wed, 10 Aug 2022 16:59:40 GMT
• Title: CoditT5: Pretraining for Source Code and Natural Language Editing
• Authors: Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, Milos Gligoric
• Abstract summary: CoditT5 is a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments. We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review.
• Score: 34.77621217370665
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretrained language models have been shown to be effective in many software-related generation tasks; however, they are not well-suited for editing tasks as they are not designed to reason about edits. To address this, we propose a novel pretraining objective which explicitly models edits and use it to build CoditT5, a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments. We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review. By outperforming pure generation-based models, we demonstrate the generalizability of our approach and its suitability for editing tasks. We also show how a pure generation model and our edit-based model can complement one another through simple reranking strategies, with which we achieve state-of-the-art performance for the three downstream editing tasks.

Related papers

• A Reinforcement Learning-Based Automatic Video Editing Method Using Pre-trained Vision-Language Model [10.736207095604414]
  We propose a two-stage scheme for general editing. Firstly, unlike previous works that extract scene-specific features, we leverage the pre-trained Vision-Language Model (VLM) We also propose a Reinforcement Learning (RL)-based editing framework to formulate the editing problem and train the virtual editor to make better sequential editing decisions.
  arXiv  Detail & Related papers  (2024-11-07T18:20:28Z)
• Should We Really Edit Language Models? On the Evaluation of Edited Language Models [15.63231238452797]
  Existing editing methods lead to inevitable performance deterioration on general benchmarks. Instruction-tuned models are more robust to editing, showing less performance drop on general knowledge after editing. Our findings indicate that current editing methods are only suitable for small-scale knowledge updates within language models.
  arXiv  Detail & Related papers  (2024-10-24T14:36:48Z)
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
  We evaluate the side effects of model editing on large language models (LLMs) Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively. To mitigate this, a method named RECT is proposed to regularize the edit update weights.
  arXiv  Detail & Related papers  (2024-01-09T18:03:15Z)
• DUnE: Dataset for Unified Editing [3.7346004746366384]
  We introduce DUnE-an editing benchmark where edits are natural language sentences. We show that retrieval-augmented language modeling can outperform specialized editing techniques.
  arXiv  Detail & Related papers  (2023-11-27T18:56:14Z)
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors [53.819805242367345]
  We propose GRACE, a lifelong model editing method, which implements spot-fixes on streaming errors of a deployed model. GRACE writes new mappings into a pre-trained model's latent space, creating a discrete, local codebook of edits without altering model weights. Our experiments on T5, BERT, and GPT models show GRACE's state-of-the-art performance in making and retaining edits, while generalizing to unseen inputs.
  arXiv  Detail & Related papers  (2022-11-20T17:18:22Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)
• Language Anisotropic Cross-Lingual Model Editing [61.51863835749279]
  Existing work only studies the monolingual scenario, which lacks the cross-lingual transferability to perform editing simultaneously across languages. We propose a framework to naturally adapt monolingual model editing approaches to the cross-lingual scenario using parallel corpus. We empirically demonstrate the failure of monolingual baselines in propagating the edit to multiple languages and the effectiveness of the proposed language anisotropic model editing.
  arXiv  Detail & Related papers  (2022-05-25T11:38:12Z)
• Learning to Model Editing Processes [98.11448946134894]
  We propose modeling editing processes, modeling the whole process of iteratively generating sequences. We form a conceptual framework to describe the likelihood of multi-step edits, and describe neural models that can learn a generative model of sequences based on these multistep edits.
  arXiv  Detail & Related papers  (2022-05-24T21:32:52Z)
• Learning Structural Edits via Incremental Tree Transformations [102.64394890816178]
  We present a generic model for incremental editing of structured data (i.e., "structural edits") Our editor learns to iteratively generate tree edits (e.g., deleting or adding a subtree) and applies them to the partially edited data. We evaluate our proposed editor on two source code edit datasets, where results show that, with the proposed edit encoder, our editor significantly improves accuracy over previous approaches.
  arXiv  Detail & Related papers  (2021-01-28T16:11:32Z)
• A Structural Model for Contextual Code Changes [20.185486717922615]
  Given a code snippet that is partially edited, our goal is to predict a completion of the edit for the rest of the snippet. Our model achieves a 28% relative gain over state-of-the-art sequential models and 2x higher accuracy than syntactic models that learn to generate the edited code.
  arXiv  Detail & Related papers  (2020-05-27T07:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.