Code Llama: Open Foundation Models for Code

• URL: http://arxiv.org/abs/2308.12950v3
• Date: Wed, 31 Jan 2024 19:47:26 GMT
• Title: Code Llama: Open Foundation Models for Code
• Authors: Baptiste Rozi\`ere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, J\'er\'emy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D\'efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve
• Abstract summary: We release Code Llama, a family of large language models for code based on Llama 2. Code Llama reaches state-of-the-art performance among open models on several code benchmarks. We release Code Llama under a permissive license that allows for both research and commercial use.
• Score: 93.30115424203868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.

Related papers

• OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models [70.72097493954067]
  Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems. While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs remain limited. We introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community.
  arXiv  Detail & Related papers  (2024-11-07T17:47:25Z)
• AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data [64.69872638349922]
  We present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. We propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review.
  arXiv  Detail & Related papers  (2024-05-29T16:57:33Z)
• Granite Code Models: A Family of Open Foundation Models for Code Intelligence [37.946802472358996]
  Large Language Models (LLMs) trained on code are revolutionizing the software development process. LLMs are being integrated into software development environments to improve the productivity of human programmers. We introduce the Granite series of decoder-only code models for code generative tasks.
  arXiv  Detail & Related papers  (2024-05-07T13:50:40Z)
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence [42.517055368627226]
  We introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. Our evaluations demonstrate that DeepSeek-Coder achieves state-of-the-art performance among open-source code models across multiple benchmarks. DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
  arXiv  Detail & Related papers  (2024-01-25T14:17:53Z)
• PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback [5.459517921633247]
  We propose a novel RRTF (Rank Responses to align Test&Teacher Feedback) framework, which can effectively and efficiently boost pre-trained large language models for code generation. Under this framework, we present PanGu-Coder2, which achieves 62.20% pass@1 on the OpenAI HumanEval benchmark.
  arXiv  Detail & Related papers  (2023-07-27T15:28:29Z)
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct [67.24653703564492]
  We introduce WizardCoder, which empowers Code LLMs with complex instruction fine-tuning. Our model surpasses all other open-source Code LLMs by a substantial margin.
  arXiv  Detail & Related papers  (2023-06-14T15:18:48Z)
• StarCoder: may the source be with you! [79.93915935620798]
  The BigCode community introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories.
  arXiv  Detail & Related papers  (2023-05-09T08:16:42Z)
• A Systematic Evaluation of Large Language Models of Code [88.34057460577957]
  Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. The current state-of-the-art code LMs are not publicly available, leaving many questions about their model and data design decisions. Although Codex is not open-source, we find that existing open-source models do achieve close results in some programming languages. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, which was trained on 249GB of code across 12 programming languages on a single machine.
  arXiv  Detail & Related papers  (2022-02-26T15:53:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.