LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation

• URL: http://arxiv.org/abs/2412.04478v1
• Date: Tue, 19 Nov 2024 21:52:23 GMT
• Title: LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation
• Authors: Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishna Ramanathan, Xiaofei Ma, Anoop Deoras,
• Abstract summary: We introduce LibEvolutionEval, a study requiring an understanding of library evolution to perform in-line code completion accurately. We evaluate popular public models and find that public library evolution significantly influences model performance. We explore mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling fast-evolving packages.
• Score: 40.87656746406113
• License:
• Abstract: Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.

Related papers

• Deep Fast Machine Learning Utils: A Python Library for Streamlined Machine Learning Prototyping [0.0]
  The Deep Fast Machine Learning Utils (DFMLU) library provides tools designed to automate and enhance aspects of machine learning processes. DFMLU offers functionalities that support model development and data handling. This manuscript presents an overview of DFMLU's functionalities, providing Python examples for each tool.
  arXiv  Detail & Related papers  (2024-09-14T21:39:17Z)
• LibCity: A Unified Library Towards Efficient and Comprehensive Urban Spatial-Temporal Prediction [74.08181247675095]
  There are limitations in the existing field, including open-source data being in various formats and difficult to use. We propose LibCity, an open-source library that offers researchers a credible experimental tool and a convenient development framework.
  arXiv  Detail & Related papers  (2023-04-27T17:19:26Z)
• SequeL: A Continual Learning Library in PyTorch and JAX [50.33956216274694]
  SequeL is a library for Continual Learning that supports both PyTorch and JAX frameworks. It provides a unified interface for a wide range of Continual Learning algorithms, including regularization-based approaches, replay-based approaches, and hybrid approaches. We release SequeL as an open-source library, enabling researchers and developers to easily experiment and extend the library for their own purposes.
  arXiv  Detail & Related papers  (2023-04-21T10:00:22Z)
• An Empirical Study of Library Usage and Dependency in Deep Learning Frameworks [12.624032509149869]
  pytorch, Caffe, and Scikit-learn are the most frequent combination in 18% and 14% of the projects. The developer uses two or three dl libraries in the same projects and tends to use different multiple dl libraries in both the same function and the same files.
  arXiv  Detail & Related papers  (2022-11-28T19:31:56Z)
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case [0.0]
  We present Pythae, a versatile open-source Python library providing straightforward, reproducible and reliable use of generative autoencoder models. We present and compare 19 generative autoencoder models representative of some of the main improvements on downstream tasks.
  arXiv  Detail & Related papers  (2022-06-16T17:11:41Z)
• Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code [74.28810048824519]
  Repro is an open-source library which aims at improving the usability of research code. It provides a lightweight Python API for running software released by researchers within Docker containers.
  arXiv  Detail & Related papers  (2022-04-29T01:54:54Z)
• SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics [74.28810048824519]
  SacreROUGE is an open-source library for using and developing summarization evaluation metrics. The library provides Python wrappers around the official implementations of existing evaluation metrics. It provides functionality to evaluate how well any metric implemented in the library correlates to human-annotated judgments.
  arXiv  Detail & Related papers  (2020-07-10T13:26:37Z)
• Picasso: A Sparse Learning Library for High Dimensional Data Analysis in R and Python [77.33905890197269]
  We describe a new library which implements a unified pathwise coordinate optimization for a variety of sparse learning problems. The library is coded in R++ and has user-friendly sparse experiments.
  arXiv  Detail & Related papers  (2020-06-27T02:39:24Z)
• fastai: A Layered API for Deep Learning [1.7223564681760164]
  fastai is a deep learning library which provides practitioners with high-level components. It provides researchers with low-level components that can be mixed and matched to build new approaches.
  arXiv  Detail & Related papers  (2020-02-11T21:16:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.