Understanding Code Semantics: An Evaluation of Transformer Models in
Summarization
- URL: http://arxiv.org/abs/2310.16314v2
- Date: Fri, 27 Oct 2023 01:22:52 GMT
- Title: Understanding Code Semantics: An Evaluation of Transformer Models in
Summarization
- Authors: Debanjan Mondal, Abhilasha Lodha, Ankita Sahoo, Beena Kumari
- Abstract summary: We evaluate the efficacy of code summarization by altering function and variable names.
We introduce adversaries like dead code and commented code across three programming languages.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper delves into the intricacies of code summarization using advanced
transformer-based language models. Through empirical studies, we evaluate the
efficacy of code summarization by altering function and variable names to
explore whether models truly understand code semantics or merely rely on
textual cues. We have also introduced adversaries like dead code and commented
code across three programming languages (Python, Javascript, and Java) to
further scrutinize the model's understanding. Ultimately, our research aims to
offer valuable insights into the inner workings of transformer-based LMs,
enhancing their ability to understand code and contributing to more efficient
software development practices and maintenance workflows.
Related papers
- An Empirical Study on Capability of Large Language Models in Understanding Code Semantics [4.638578225024275]
Large Language Models for Code (code LLMs) have demonstrated remarkable performance across various software engineering (SE) tasks.
This paper introduces EMPICA, a framework designed to evaluate the capabilities of code LLMs in understanding code semantics.
arXiv Detail & Related papers (2024-07-04T03:40:58Z) - ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization [21.886950861445122]
Code summarization aims to automatically generate succinct natural language summaries for given code snippets.
This paper proposes a novel approach to improve code summarization based on summary-focused tasks.
arXiv Detail & Related papers (2024-07-01T03:06:51Z) - How Far Have We Gone in Binary Code Understanding Using Large Language Models [51.527805834378974]
We propose a benchmark to evaluate the effectiveness of Large Language Models (LLMs) in binary code understanding.
Our evaluations reveal that existing LLMs can understand binary code to a certain extent, thereby improving the efficiency of binary code analysis.
arXiv Detail & Related papers (2024-04-15T14:44:08Z) - Code Needs Comments: Enhancing Code LLMs with Comment Augmentation [91.52444946362547]
We introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language.
We conducted experiments on three code-focused Large Language Models and observed consistent improvements in performance on two widely-used programming skill benchmarks.
arXiv Detail & Related papers (2024-02-20T13:56:38Z) - Code Execution with Pre-trained Language Models [88.04688617516827]
Most pre-trained models for code intelligence ignore the execution trace and only rely on source code and syntactic structures.
We develop a mutation-based data augmentation technique to create a large-scale and realistic Python dataset and task for code execution.
We then present CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning to enhance its semantic comprehension.
arXiv Detail & Related papers (2023-05-08T10:00:05Z) - Enhancing Semantic Code Search with Multimodal Contrastive Learning and
Soft Data Augmentation [50.14232079160476]
We propose a new approach with multimodal contrastive learning and soft data augmentation for code search.
We conduct extensive experiments to evaluate the effectiveness of our approach on a large-scale dataset with six programming languages.
arXiv Detail & Related papers (2022-04-07T08:49:27Z) - ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval.
We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
arXiv Detail & Related papers (2022-03-15T08:25:08Z) - Empirical Study of Transformers for Source Code [14.904366372190943]
We study the capabilities of Transformers to utilize syntactic information in different tasks.
We show that Transformers are able to make meaningful predictions based purely on syntactic information.
arXiv Detail & Related papers (2020-10-15T19:09:15Z) - A Transformer-based Approach for Source Code Summarization [86.08359401867577]
We learn code representation for summarization by modeling the pairwise relationship between code tokens.
We show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin.
arXiv Detail & Related papers (2020-05-01T23:29:36Z) - Leveraging Code Generation to Improve Code Retrieval and Summarization
via Dual Learning [18.354352985591305]
Code summarization generates brief natural language description given a source code snippet, while code retrieval fetches relevant source code given a natural language query.
Recent studies have combined these two tasks to improve their performance.
We propose a novel end-to-end model for the two tasks by introducing an additional code generation task.
arXiv Detail & Related papers (2020-02-24T12:26:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.