A Comparative Study on Code Generation with Transformers

• URL: http://arxiv.org/abs/2412.05749v1
• Date: Sat, 07 Dec 2024 21:18:23 GMT
• Title: A Comparative Study on Code Generation with Transformers
• Authors: Namrata Das, Rakshya Panta, Neelam Karki, Ruchi Manandhar, Dinesh Baniya Kshatri,
• Abstract summary: This paper introduces the concept of a "A Comparative Study on Code Generation with Transformers" A model based on Transformer architecture, and NLP methodologies to automatically generate C++ source code for different varieties of problems.
• Score: 0.0
• License:
• Abstract: In an era of widespread influence of Natural Language Processing (NLP), there have been multiple research efforts to supplant traditional manual coding techniques with automated systems capable of generating solutions autonomously. With rapid research for code generation and a sole focus on large language models, there emerges a need to compare and evaluate the performance of transformer architectures based on several complexities of the model. This paper introduces the concept of a "A Comparative Study on Code Generation with Transformers," a model based on Transformer architecture, and NLP methodologies to automatically generate C++ source code for different varieties of problems. Here, a comparative study is performed to evaluate the robustness of transformer-based models on the basis of their architecture complexities and their capability to handle diverse problem sets, from basic arithmetic to complex computations.

Related papers

• A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis [0.8049701904919515]
  This paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences.
  arXiv  Detail & Related papers  (2024-12-10T05:33:09Z)
• STAR: Synthesis of Tailored Architectures [61.080157488857516]
  We propose a new approach for the synthesis of tailored architectures (STAR) Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.
  arXiv  Detail & Related papers  (2024-11-26T18:42:42Z)
• Transformers are Efficient Compilers, Provably [11.459397066286822]
  Transformer-based large language models (LLMs) have demonstrated surprisingly robust performance across a wide range of language-related tasks. In this paper, we take the first steps towards a formal investigation of using transformers as compilers from an expressive power perspective. We introduce a representative programming language, Mini-Husky, which encapsulates key features of modern C-like languages.
  arXiv  Detail & Related papers  (2024-10-07T20:31:13Z)
• Learning to Estimate System Specifications in Linear Temporal Logic using Transformers and Mamba [6.991281327290525]
  specification mining involves extracting temporal logic formulae from system traces. We introduce autore models that can generate linear temporal logic formulae from traces. We devise a metric for the distinctiveness of the generated formulae and an algorithm to enforce the syntax constraints.
  arXiv  Detail & Related papers  (2024-05-31T15:21:53Z)
• Looped Transformers are Better at Learning Learning Algorithms [16.98720552888865]
  We propose the utilization of looped transformer architecture and its associated training methodology. Experimental results suggest that the looped transformer achieves performance comparable to the standard transformer.
  arXiv  Detail & Related papers  (2023-11-21T08:32:38Z)
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection [88.23337313766353]
  This work first provides a comprehensive statistical theory for transformers to perform ICL. We show that transformers can implement a broad class of standard machine learning algorithms in context. A emphsingle transformer can adaptively select different base ICL algorithms.
  arXiv  Detail & Related papers  (2023-06-07T17:59:31Z)
• Full Stack Optimization of Transformer Inference: a Survey [58.55475772110702]
  Transformer models achieve superior accuracy across a wide range of applications. The amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate. There has been an increased focus on making Transformer models more efficient.
  arXiv  Detail & Related papers  (2023-02-27T18:18:13Z)
• Redefining Neural Architecture Search of Heterogeneous Multi-Network Models by Characterizing Variation Operators and Model Components [71.03032589756434]
  We investigate the effect of different variation operators in a complex domain, that of multi-network heterogeneous neural models. We characterize both the variation operators, according to their effect on the complexity and performance of the model; and the models, relying on diverse metrics which estimate the quality of the different parts composing it.
  arXiv  Detail & Related papers  (2021-06-16T17:12:26Z)
• Thinking Like Transformers [64.96770952820691]
  We propose a computational model for the transformer-encoder in the form of a programming language. We show how RASP can be used to program solutions to tasks that could conceivably be learned by a Transformer. We provide RASP programs for histograms, sorting, and Dyck-languages.
  arXiv  Detail & Related papers  (2021-06-13T13:04:46Z)
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures [57.46093180685175]
  We demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. We add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales.
  arXiv  Detail & Related papers  (2021-06-10T15:41:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.