LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

• URL: http://arxiv.org/abs/2606.25402v1
• Date: Wed, 24 Jun 2026 04:58:28 GMT
• Title: LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
• Abstract summary: Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases.<n>This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one.<n>We introduce LibEvoBench, a benchmark spanning multiple versions of Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models' consistency when working with evolving APIs.
• Score: 12.441288958152931
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors - calling APIs as they exist in a different library version. To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models' consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models' accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation.

Related papers

• Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs [1.4228421209315671]
  CURE is a contrastive unlearning approach that shifts unlearning from purely suppressing outdated knowledge to explicitly promoting correct API replacements.<n>Experiments show that CURE not only reduces deprecated API usage but also improves correct API replacement, while preserving general code generation performance.
  arXiv  Detail & Related papers  (2026-06-29T18:34:55Z)
• Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition [2.392878902712915]
  NovelAPIBench is a dynamic benchmark for any base model and target library.<n>We compare knowledge injected through retrieval with knowledge internalized through parametric adaptation.<n>Retrieval supplies volatile API content, while tuning improves procedural integration.
  arXiv  Detail & Related papers  (2026-06-02T13:46:04Z)
• Test-Time Learning with an Evolving Library [76.87197412229318]
  EvoLib is a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge.<n>Instead of adapting model parameters, our approach maintains a shared library of knowledge abstractions.
  arXiv  Detail & Related papers  (2026-05-14T07:18:12Z)
• When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation [12.67668690329239]
  "context-memory conflict" arises when external instructions contradict a model's internal parametric knowledge.<n>This paper presents a benchmark of 270 real-world updates from eight Python libraries.
  arXiv  Detail & Related papers  (2026-04-10T17:37:26Z)
• KCoEvo: A Knowledge Graph Augmented Framework for Evolutionary Code Generation [25.38402999279553]
  We propose a knowledge graph-augmented framework that decomposes the migration task into two synergistic stages: evolution path retrieval and path-informed code generation.<n>Our approach constructs static and dynamic API graphs to model intra-version structures and cross-version transitions, enabling structured reasoning over API evolution.
  arXiv  Detail & Related papers  (2026-03-08T10:39:48Z)
• Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations [15.586818028794942]
  Pre-trained Large Language Models (LLMs) have demonstrated strong performance in code completion tasks.<n>LLMs frequently generate deprecated APIs that will no longer be supported in future versions of third-party libraries.<n>We propose AdaLoRA-L, which defines "Common API Layers" (layers with high importance across all APIs, storing general knowledge and excluded from editing) and restricts edits exclusively to "Specific API Layers"<n> Experimental results demonstrate that AdaLoRA-L significantly improves Specificity while maintaining comparable performance across other evaluation metrics.
  arXiv  Detail & Related papers  (2025-11-26T03:36:34Z)
• Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models [49.214291813478695]
  Deep learning (DL) libraries, widely used in AI applications, often contain vulnerabilities like overflows and use buffer-free errors.<n>Traditional fuzzing struggles with the complexity and API diversity of DL libraries.<n>We propose DFUZZ, an LLM-driven fuzzing approach for DL libraries.
  arXiv  Detail & Related papers  (2025-01-08T07:07:22Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution.<n> Experimental results demonstrate that ExploraCoder significantly improves performance for models lacking prior API knowledge.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation [40.87656746406113]
  We introduce LibEvolutionEval, a study requiring an understanding of library evolution to perform in-line code completion accurately.<n>We evaluate popular public models and find that public library evolution significantly influences model performance.<n>We explore mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling fast-evolving packages.
  arXiv  Detail & Related papers  (2024-11-19T21:52:23Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• Private-Library-Oriented Code Generation with Large Language Models [52.73999698194344]
  This paper focuses on utilizing large language models (LLMs) for code generation in private libraries. We propose a novel framework that emulates the process of programmers writing private code. We create four private library benchmarks, including TorchDataEval, TorchDataComplexEval, MonkeyEval, and BeatNumEval.
  arXiv  Detail & Related papers  (2023-07-28T07:43:13Z)
• SequeL: A Continual Learning Library in PyTorch and JAX [50.33956216274694]
  SequeL is a library for Continual Learning that supports both PyTorch and JAX frameworks. It provides a unified interface for a wide range of Continual Learning algorithms, including regularization-based approaches, replay-based approaches, and hybrid approaches. We release SequeL as an open-source library, enabling researchers and developers to easily experiment and extend the library for their own purposes.
  arXiv  Detail & Related papers  (2023-04-21T10:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.