Out of Context: How important is Local Context in Neural Program Repair?
- URL: http://arxiv.org/abs/2312.04986v1
- Date: Fri, 8 Dec 2023 11:49:02 GMT
- Title: Out of Context: How important is Local Context in Neural Program Repair?
- Authors: Julian Aron Prenner and Romain Robbes
- Abstract summary: We study the importance of this local context on repair success.
We train and evaluate Transformer models in many different local context configurations.
Our results are not only relevant for researchers working on Transformer-based APR tools but also for benchmark and dataset creators.
- Score: 5.732727528813227
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep learning source code models have been applied very successfully to the
problem of automated program repair. One of the standing issues is the small
input window of current models which often cannot fully fit the context code
required for a bug fix (e.g., method or class declarations of a project).
Instead, input is often restricted to the local context, that is, the lines
below and above the bug location. In this work we study the importance of this
local context on repair success: how much local context is needed?; is context
before or after the bug location more important? how is local context tied to
the bug type? To answer these questions we train and evaluate Transformer
models in many different local context configurations on three datasets and two
programming languages. Our results indicate that overall repair success
increases with the size of the local context (albeit not for all bug types) and
confirm the common practice that roughly 50-60% of the input window should be
used for context leading the bug. Our results are not only relevant for
researchers working on Transformer-based APR tools but also for benchmark and
dataset creators who must decide what and how much context to include in their
datasets.
Related papers
- Improved IR-based Bug Localization with Intelligent Relevance Feedback [2.9312156642007294]
Software bugs pose a significant challenge during development and maintenance, and practitioners spend nearly 50% of their time dealing with bugs.
Many existing techniques adopt Information Retrieval (IR) to localize a reported bug using textual and semantic relevance between bug reports and source code.
We present a novel technique for bug localization - BRaIn - that addresses the contextual gaps by assessing the relevance between bug reports and code.
arXiv Detail & Related papers (2025-01-17T20:29:38Z) - ContextCite: Attributing Model Generation to Context [64.90535024385305]
We introduce the problem of context attribution, pinpointing the parts of the context that led a model to generate a particular statement.
We then present ContextCite, a simple and scalable method for context attribution that can be applied on top of any existing language model.
We showcase ContextCite through three applications: helping verify generated statements, improving response quality, and detecting poisoning attacks.
arXiv Detail & Related papers (2024-09-01T14:36:36Z) - Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts [83.57864140378035]
This paper proposes a method to cover longer contexts in Open-Domain Question-Answering tasks.
It leverages a small encoder language model that effectively encodes contexts, and the encoding applies cross-attention with origin inputs.
After fine-tuning, there is improved performance across two held-in datasets, four held-out datasets, and also in two In Context Learning settings.
arXiv Detail & Related papers (2024-04-02T15:10:11Z) - Making Retrieval-Augmented Language Models Robust to Irrelevant Context [55.564789967211844]
An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant.
Recent work has shown that retrieval augmentation can sometimes have a negative effect on performance.
arXiv Detail & Related papers (2023-10-02T18:52:35Z) - WELL: Applying Bug Detectors to Bug Localization via Weakly Supervised
Learning [37.09621161662761]
This paper proposes a WEakly supervised bug LocaLization (WELL) method to train a bug localization model.
With CodeBERT finetuned on the buggy-or-not binary labeled data, WELL can address bug localization in a weakly supervised manner.
arXiv Detail & Related papers (2023-05-27T06:34:26Z) - Does Localization Inform Editing? Surprising Differences in
Causality-Based Localization vs. Knowledge Editing in Language Models [68.03946716358335]
We find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored.
This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models.
Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained language models work may not always translate to insights about how to best change their behavior.
arXiv Detail & Related papers (2023-01-10T21:26:08Z) - Detect-Localize-Repair: A Unified Framework for Learning to Debug with
CodeT5 [14.712753336831172]
We propose a novel unified emphDetect-Localize-Repair framework based on a pretrained programming language model CodeT5.
Our model significantly outperforms existing baselines from both NLP and software engineering domains.
arXiv Detail & Related papers (2022-11-27T16:11:29Z) - BigIssue: A Realistic Bug Localization Benchmark [89.8240118116093]
BigIssue is a benchmark for realistic bug localization.
We provide a general benchmark with a diversity of real and synthetic Java bugs.
We hope to advance the state of the art in bug localization, in turn improving APR performance and increasing its applicability to the modern development cycle.
arXiv Detail & Related papers (2022-07-21T20:17:53Z) - Local-Global Video-Text Interactions for Temporal Grounding [77.5114709695216]
This paper addresses the problem of text-to-video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query.
We tackle this problem using a novel regression-based model that learns to extract a collection of mid-level features for semantic phrases in a text query.
The proposed method effectively predicts the target time interval by exploiting contextual information from local to global.
arXiv Detail & Related papers (2020-04-16T08:10:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.