Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance

• URL: http://arxiv.org/abs/2404.08817v2
• Date: Mon, 3 Jun 2024 11:56:38 GMT
• Title: Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance
• Authors: Yewei Song, Cedric Lothritz, Daniel Tang, Tegawendé F. Bissyandé, Jacques Klein,
• Abstract summary: We revisit recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance. Our experiments showcase the effectiveness of AST editing distance in capturing intricate code structures, revealing a high correlation with established metrics. We propose, optimize, and publish a metric that demonstrates effectiveness across all tested languages.
• Score: 6.164970071786899
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper revisits recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance in diverse programming languages. In particular, we explore the usefulness of these metrics and compare them to traditional sequence similarity metrics. Our experiments showcase the effectiveness of AST editing distance in capturing intricate code structures, revealing a high correlation with established metrics. Furthermore, we explore the strengths and weaknesses of AST editing distance and prompt-based GPT similarity scores in comparison to BLEU score, execution match, and Jaccard Similarity. We propose, optimize, and publish an adaptable metric that demonstrates effectiveness across all tested languages, representing an enhanced version of Tree Similarity of Edit Distance (TSED).

Related papers

• Using Similarity to Evaluate Factual Consistency in Summaries [2.7595794227140056]
  Abstractive summarisers generate fluent summaries, but the factuality of the generated text is not guaranteed. We propose a new zero-shot factuality evaluation metric, Sentence-BERTScore (SBERTScore), which compares sentences between the summary and the source document. Our experiments indicate that each technique has different strengths, with SBERTScore particularly effective in identifying correct summaries.
  arXiv  Detail & Related papers  (2024-09-23T15:02:38Z)
• Predicting Text Preference Via Structured Comparative Reasoning [110.49560164568791]
  We introduce SC, a prompting approach that predicts text preferences by generating structured intermediate comparisons. We select consistent comparisons with a pairwise consistency comparator that ensures each aspect's comparisons clearly distinguish differences between texts. Our comprehensive evaluations across various NLP tasks, including summarization, retrieval, and automatic rating, demonstrate that SC equips LLMs to achieve state-of-the-art performance in text preference prediction.
  arXiv  Detail & Related papers  (2023-11-14T18:51:38Z)
• Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation [55.92852268168816]
  N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely utilized across a range of natural language generation (NLG) tasks. Recent studies have revealed a weak correlation between these matching-based metrics and human evaluations. We propose to utilize textitmultiple references to enhance the consistency between these metrics and human evaluations.
  arXiv  Detail & Related papers  (2023-08-06T14:49:26Z)
• CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation [91.16551253297588]
  COunterfactual Generation via Retrieval and Editing (CORE) is a retrieval-augmented generation framework for creating diverse counterfactual perturbations for training. CORE first performs a dense retrieval over a task-related unlabeled text corpus using a learned bi-encoder. CORE then incorporates these into prompts to a large language model with few-shot learning capabilities, for counterfactual editing.
  arXiv  Detail & Related papers  (2022-10-10T17:45:38Z)
• Evaluating Table Structure Recognition: A New Perspective [2.1067139116005595]
  Existing metrics used to evaluate table structure recognition algorithms have shortcomings with regard to capturing text and empty cells alignment. In this paper, we propose a new metric - TEDS based IOU similarity (TEDS (IOU)) for table structure recognition which uses bounding boxes instead of text while simultaneously being robust against the above disadvantages.
  arXiv  Detail & Related papers  (2022-07-31T07:48:36Z)
• FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric [48.66580267438049]
  We present FastKASSIM, a metric for utterance- and document-level syntactic similarity. It pairs and averages the most similar dependency parse trees between a pair of documents based on tree kernels. It runs up to to 5.2 times faster than our baseline method over the documents in the r/ChangeMyView corpus.
  arXiv  Detail & Related papers  (2022-03-15T22:33:26Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting [15.28941592388958]
  Abstract Syntax Tree (AST), which depicts the source code's syntactic structure, has been incorporated to guide the generation of code summaries. Existing AST based methods suffer from the difficulty of training and generate inadequate code summaries. We present the Block-wise Abstract Syntax Tree Splitting method (BASTS), which fully utilizes the rich tree-form syntax structure in ASTs.
  arXiv  Detail & Related papers  (2021-03-14T05:04:06Z)
• Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance [24.49032191669509]
  We present two techniques for improving encoding representations for similarity metrics. We show results over various BERT-backbone learned metrics and achieving state of the art correlation with human ratings on several benchmarks.
  arXiv  Detail & Related papers  (2020-10-13T03:46:25Z)
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis [57.87741831987889]
  In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy. We introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow.
  arXiv  Detail & Related papers  (2020-09-22T03:10:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.