Contextualized Code Pretraining for Code Generation

• URL: http://arxiv.org/abs/2605.17957v1
• Date: Mon, 18 May 2026 07:12:14 GMT
• Title: Contextualized Code Pretraining for Code Generation
• Abstract summary: We propose code pretraining, an invocation-aware framework that integrates calling context into both the training and evaluation of code models.<n>We train CallerGen, the first code models pretrained with invocation-aware objectives spanning multiple sizes, and evaluate them on CallerEval, a new benchmark featuring realistic scenarios.
• Score: 9.886764308766926
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As code generation becomes increasingly central to improving software development efficiency, modern code models are largely trained and evaluated on code with natural-language descriptions. In real projects, developers often implement missing functions under limited project-specific artifacts, while the local call-site context is already available in the surrounding code. This usage context provides actionable cues about expected behavior, but existing models are not explicitly optimized to leverage it reliably, leading to implementations that may not integrate smoothly with surrounding usage in repository settings. In this work, we propose contextualized code pretraining, an invocation-aware framework that integrates calling context into both the training and evaluation of code models. Using static analysis, we automatically extract large-scale caller-callee pairs from real repositories to construct pretraining tasks and benchmarks that condition generation on the calling context. We train CallerGen, the first code models pretrained with invocation-aware objectives spanning multiple sizes, and evaluate them on CallerEval, a new benchmark featuring realistic scenarios. Experiments show that CallerGen outperforms comparable-scale models and remains competitive with larger ones across two benchmarks. Our 220M and 0.5B models achieve 16.58% and 22.81@% pass1, surpassing baselines on CallerEval. These results highlight the importance of calling context in realistic code generation.

Related papers

• CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues [6.475060769782328]
  Large Language Models (LLMs) are increasingly used in software engineering to generate and refine code.<n>Existing benchmarks generally omit this multi-turn code refinement dialogue setting.<n>We introduce CodeChat-Eval, an evaluation framework that constructs evaluation sessions from multi-turn code refinement dialogues.
  arXiv  Detail & Related papers  (2026-06-24T12:16:04Z)
• SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion [7.668823606571788]
  Code completion is a prominent application of Large Language Models (LLMs) in software engineering.<n>This paper proposes SynthCoder, a model that integrates leading industry practices to achieve state-of-the-art on the Fill-in-the-Middle (FIM) code completion task.
  arXiv  Detail & Related papers  (2025-08-21T12:23:49Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• RepoMasterEval: Evaluating Code Completion via Real-World Repositories [14.744942194298673]
  RepoMasterEval is a novel benchmark for evaluating code completion models constructed from real-world repositories.<n>Each benchmark is generated by masking a code snippet from one source code file with existing test suites.
  arXiv  Detail & Related papers  (2024-08-07T03:06:57Z)
• On the Impacts of Contexts on Repository-Level Code Generation [5.641402231731082]
  We present RepoExec, a novel benchmark designed to evaluate repository-level code generation.<n>We focus on three key aspects: executability, functional correctness through comprehensive test case generation, and accurate utilization of cross-file contexts.
  arXiv  Detail & Related papers  (2024-06-17T10:45:22Z)
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models [20.169432642273524]
  We propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions.
  arXiv  Detail & Related papers  (2023-02-01T07:39:28Z)
• CodeExp: Explanatory Code Document Generation [94.43677536210465]
  Existing code-to-text generation models produce only high-level summaries of code. We conduct a human study to identify the criteria for high-quality explanatory docstring for code. We present a multi-stage fine-tuning strategy and baseline models for the task.
  arXiv  Detail & Related papers  (2022-11-25T18:05:44Z)
• Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation [10.75138604869187]
  In some domain-specific scenarios, building such a large paired corpus for code generation is difficult because there is no directly available pairing data. We propose a task augmentation method that incorporates domain knowledge into code generation models through auxiliary tasks and a Subtoken-TranX model. Our experimental results demonstrate that the subtoken-level TranX model outperforms the original TranX model and the Transformer model on our dataset.
  arXiv  Detail & Related papers  (2022-08-22T06:57:51Z)
• InCoder: A Generative Model for Code Infilling and Synthesis [88.46061996766348]
  We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) and editing (via infilling) InCoder is trained to generate code files from a large corpus of permissively licensed code. Our model is the first generative model that is able to directly perform zero-shot code infilling.
  arXiv  Detail & Related papers  (2022-04-12T16:25:26Z)
• ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
  We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
  arXiv  Detail & Related papers  (2022-03-15T08:25:08Z)
• CodeRetriever: Unimodal and Bimodal Contrastive Learning [128.06072658302165]
  We propose the CodeRetriever model, which combines the unimodal and bimodal contrastive learning to train function-level code semantic representations. For unimodal contrastive learning, we design a semantic-guided method to build positive code pairs based on the documentation and function name. For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build text-code pairs.
  arXiv  Detail & Related papers  (2022-01-26T10:54:30Z)
• CoreGen: Contextualized Code Representation Learning for Commit Message Generation [39.383390029545865]
  We propose a novel Contextualized code representation learning strategy for commit message Generation (CoreGen) Experiments on the benchmark dataset demonstrate the superior effectiveness of our model over the baseline models with at least 28.18% improvement in terms of BLEU-4 score.
  arXiv  Detail & Related papers  (2020-07-14T09:43:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.