Do Code LLMs Understand Design Patterns?

• URL: http://arxiv.org/abs/2501.04835v1
• Date: Wed, 08 Jan 2025 20:39:45 GMT
• Title: Do Code LLMs Understand Design Patterns?
• Authors: Zhenyu Pan, Xuefeng Song, Yunkun Wang, Rongyu Cao, Binhua Li, Yongbin Li, Han Liu,
• Abstract summary: We empirically investigate the biases of Code LLMs in software development. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.
• Score: 45.89136944351375
• License:
• Abstract: Code Large Language Models (LLMs) demonstrate great versatility in adapting to various downstream tasks, including code generation and completion, as well as bug detection and fixing. However, Code LLMs often fail to capture existing coding standards, leading to the generation of code that conflicts with the required design patterns for a given project. As a result, developers must post-process to adapt the generated code to the project's design norms. In this work, we empirically investigate the biases of Code LLMs in software development. Through carefully designed experiments, we assess the models' understanding of design patterns across recognition, comprehension, and generation. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.

Related papers

• Case2Code: Scalable Synthetic Data for Code Generation [105.89741089673575]
  Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. Recent work improves code LLMs by training on synthetic data generated by some powerful LLMs. We propose a textbfCase2Code task by exploiting the expressiveness and correctness of programs.
  arXiv  Detail & Related papers  (2024-07-17T11:35:00Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• An Empirical Study on Capability of Large Language Models in Understanding Code Semantics [4.638578225024275]
  Large Language Models for Code (code LLMs) have demonstrated remarkable performance across various software engineering (SE) tasks. This paper introduces EMPICA, a framework designed to evaluate the capabilities of code LLMs in understanding code semantics.
  arXiv  Detail & Related papers  (2024-07-04T03:40:58Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• A Survey on Large Language Models for Code Generation [9.555952109820392]
  Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks. This survey aims to bridge the gap between academia and practical development by providing a comprehensive and up-to-date literature review.
  arXiv  Detail & Related papers  (2024-06-01T17:48:15Z)
• AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data [64.69872638349922]
  We present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. We propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review.
  arXiv  Detail & Related papers  (2024-05-29T16:57:33Z)
• A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actions [13.58143103712]
  GitHub Copilot is a large language model (LLM)-powered code generation tool. This paper investigates how developers validate and repair code generated by Copilot. Being aware of the code's provenance led to improved performance, increased search efforts, more frequent Copilot usage, and higher cognitive workload.
  arXiv  Detail & Related papers  (2024-05-25T06:20:01Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code? [10.249771123421432]
  We investigate whether Large Language Models (LLMs) attend to the same parts of a task description as human programmers during code generation. We manually analyzed 211 incorrect code snippets and found five attention patterns that can be used to explain many code generation errors. Our findings highlight the need for human-aligned LLMs for better interpretability and programmer trust.
  arXiv  Detail & Related papers  (2023-06-02T00:57:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.