RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation

• URL: http://arxiv.org/abs/2503.16922v1
• Date: Fri, 21 Mar 2025 07:33:59 GMT
• Title: RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation
• Abstract summary: RustEvo is a framework for evaluating the ability of large language models to adapt to evolving Rust APIs.<n>It automates dataset creation by synthesizing 588 API changes into programming tasks mirroring real-world challenges.<n>Experiments on state-of-the-art (SOTA) LLMs reveal significant performance variations.
• Score: 28.156862491709237
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Large Language Models (LLMs) have become pivotal tools for automating code generation in software development. However, these models face significant challenges in producing version-aware code for rapidly evolving languages like Rust, where frequent Application Programming Interfaces (API) changes across versions lead to compatibility issues and correctness errors. Existing benchmarks lack systematic evaluation of how models navigate API transitions, relying on labor-intensive manual curation and offering limited version-specific insights. To address this gap, we present RustEvo, a novel framework for constructing dynamic benchmarks that evaluate the ability of LLMs to adapt to evolving Rust APIs. RustEvo automates dataset creation by synthesizing 588 API changes (380 from Rust standard libraries, 208 from 15 third-party crates) into programming tasks mirroring real-world challenges. These tasks cover four API evolution categories: Stabilizations, Signature Changes, Behavioral Changes, and Deprecations, reflecting their actual distribution in the Rust ecosystem. Experiments on state-of-the-art (SOTA) LLMs reveal significant performance variations: models achieve a 65.8% average success rate on stabilized APIs but only 38.0% on behavioral changes, highlighting difficulties in detecting semantic shifts without signature alterations. Knowledge cutoff dates strongly influence performance, with models scoring 56.1% on before-cutoff APIs versus 32.5% on after-cutoff tasks. Retrieval-Augmented Generation (RAG) mitigates this gap, improving success rates by 13.5% on average for APIs released after model training. Our findings underscore the necessity of our evolution-aware benchmarks to advance the adaptability of LLMs in fast-paced software ecosystems. The framework and the benchmarks are publicly released at https://github.com/SYSUSELab/RustEvo.

Related papers

• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python [2.7324157162184157]
  We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust objective into Python.<n>We demonstrate that the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), confirming near-parity on real-world agentic tasks.<n>Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost.
  arXiv  Detail & Related papers  (2026-04-13T14:21:44Z)
• When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation [12.67668690329239]
  "context-memory conflict" arises when external instructions contradict a model's internal parametric knowledge.<n>This paper presents a benchmark of 270 real-world updates from eight Python libraries.
  arXiv  Detail & Related papers  (2026-04-10T17:37:26Z)
• KCoEvo: A Knowledge Graph Augmented Framework for Evolutionary Code Generation [25.38402999279553]
  We propose a knowledge graph-augmented framework that decomposes the migration task into two synergistic stages: evolution path retrieval and path-informed code generation.<n>Our approach constructs static and dynamic API graphs to model intra-version structures and cross-version transitions, enabling structured reasoning over API evolution.
  arXiv  Detail & Related papers  (2026-03-08T10:39:48Z)
• Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents [7.282281491277909]
  Rust programming language presents a steep learning curve and significant coding challenges.<n>Recently, LLM-powered code agents have shown remarkable success in resolving complex software engineering tasks.<n>RUSTFORGER is a novel agentic approach that integrates an automated test environment setup with a Rust metaprogramming-driven dynamic tracing strategy.
  arXiv  Detail & Related papers  (2026-02-26T08:54:09Z)
• Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation [95.89924101984566]
  We introduce OptimusVLA, a dual-memory VLA framework with Global Prior Memory (GPM) and Local Consistency Memory (LCM)<n>GPM replaces Gaussian noise with task-level priors retrieved from semantically similar trajectories.<n>LCM injects a learned consistency constraint that enforces temporal coherence and smoothness of trajectory.
  arXiv  Detail & Related papers  (2026-02-22T15:39:34Z)
• ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design [15.71144418188142]
  Large Language Models (LLMs) show significant potential in hardware engineering.<n>Current benchmarks suffer from saturation and limited task diversity.<n>We propose a comprehensive benchmark for AI-aided chip design.
  arXiv  Detail & Related papers  (2026-01-29T09:26:55Z)
• From Memorization to Creativity: LLM as a Designer of Novel Neural-Architectures [48.83701310501069]
  Large language models (LLMs) excel in program synthesis, yet their ability to autonomously navigate neural architecture design-balancing reliability, performance, and structural novelty--remains underexplored.<n>We address this by placing a code-oriented LLM within a closed-loop synthesis framework, analyzing its evolution over 22 supervised fine-tuning cycles.
  arXiv  Detail & Related papers  (2026-01-06T13:20:28Z)
• Understanding Robustness of Model Editing in Code LLMs: An Empirical Study [1.5624785508022727]
  We present a systematic study of five state-of-the-art model editing methods.<n>We apply these methods to three leading open-source code LLMs, CodeLlama, CodeQwen1.5, and DeepSeek-Coder.<n>Instant edits consistently degrade model performance, with syntactic validity dropping by up to 86 percentage points and functional correctness declining by 45 points even in the best-performing setting.
  arXiv  Detail & Related papers  (2025-11-05T04:58:13Z)
• Beyond Single LLMs: Enhanced Code Generation via Multi-Stage Performance-Guided LLM Orchestration [12.674888937998086]
  Large Language Models (LLMs) have become the predominant paradigm for automated code generation.<n>This paper challenges the single-model convention by introducing a multi-stage, performance-guided orchestration framework.<n>Perch orchestrates top-performing LLMs for each task context through stage-wise validation and rollback mechanisms.
  arXiv  Detail & Related papers  (2025-10-01T19:07:16Z)
• EVOC2RUST: A Skeleton-guided Framework for Project-Level C-to-Rust Translation [16.12483934561206]
  EvoC2Rust is an automated framework for converting entire C projects to equivalent Rust ones.<n>Our evaluation on open-source benchmarks and six industrial projects demonstrates EvoC2Rust's superior performance in project-level C-to-Rust translation.
  arXiv  Detail & Related papers  (2025-08-06T10:31:23Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering [51.7496756448709]
  Language models (LMs) perform well on coding benchmarks but struggle with real-world software engineering tasks.<n>Existing approaches rely on supervised fine-tuning with high-quality data, which is expensive to curate at scale.<n>We propose Test-Time Scaling (EvoScale), a sample-efficient method that treats generation as an evolutionary process.
  arXiv  Detail & Related papers  (2025-05-29T16:15:36Z)
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test [25.703729145091483]
  A growing trend in the LLM community is scaling up training data to improve model intelligence without increasing inference costs.<n>We observe that scaling up data provides limited improvements for the Eagle program.<n>We introduce Eagle-3, which abandons feature prediction in favor of direct token prediction and replaces reliance on top-layer features with multi-layer feature fusion.
  arXiv  Detail & Related papers  (2025-03-03T18:59:04Z)
• Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation [69.62857948698436]
  Recent advances in large language models (LLMs) have improved their performance on coding benchmarks. However, improvement is plateauing due to the exhaustion of readily available high-quality data. We propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity.
  arXiv  Detail & Related papers  (2025-02-20T18:32:19Z)
• How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario [72.02391485962127]
  Speech Self-Supervised Learning (SSL) models achieve impressive performance on Automatic Speech Recognition (ASR) In low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. We extend a conventional efficient fine-tuning scheme based on the adapter to handle these issues.
  arXiv  Detail & Related papers  (2024-11-27T10:51:00Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)
• Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation [6.463959200930805]
  We evaluate new commercial and open models since the release of the open-source VerilogEval benchmark.<n>We find measurable improvements in state-of-the-art models.<n>We find that prompt engineering remains crucial for achieving good pass rates.
  arXiv  Detail & Related papers  (2024-08-20T17:58:56Z)
• Mitigating the Impact of Malware Evolution on API Sequence-based Windows Malware Detector [5.953199557879621]
  Methods based on API sequences play a crucial role in malware prevention. Evolved malware samples often use the API sequences of the pre-evolution samples to achieve similar malicious behaviors. We propose a frame(MME) framework that can enhance existing API sequence-based malware detectors.
  arXiv  Detail & Related papers  (2024-08-03T04:21:24Z)
• Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models [84.8919069953397]
  Self-TAught Recognizer (STAR) is an unsupervised adaptation framework for speech recognition systems. We show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains. STAR exhibits high data efficiency that only requires less than one-hour unlabeled data.
  arXiv  Detail & Related papers  (2024-05-23T04:27:11Z)
• RoBERTa-Augmented Synthesis for Detecting Malicious API Requests [9.035212370386846]
  We introduce a GAN-inspired learning framework that extends limited API traffic datasets through targeted, domain-aware synthesis.<n>We evaluate our framework on two benchmark datasets, CSIC 2010 and ATRDF 2023, and compare it with a previous data augmentation technique.<n>Our method achieves up to a 4.94% increase in F1 score on CSIC 2010 and up to 21.10% on ATRDF 2023.
  arXiv  Detail & Related papers  (2024-05-18T11:10:45Z)
• Octopus: On-device language model for function calling of software APIs [9.78611123915888]
  Large Language Models (LLMs) play a crucial role due to their advanced text processing and generation abilities. This study introduces a new strategy aimed at harnessing on-device LLMs in invoking software APIs.
  arXiv  Detail & Related papers  (2024-04-02T01:29:28Z)
• StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models [74.88844320554284]
  We introduce StableToolBench, a benchmark evolving from ToolBench.<n>The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status.<n>The stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation.
  arXiv  Detail & Related papers  (2024-03-12T14:57:40Z)
• GEVO-ML: Optimizing Machine Learning Code with Evolutionary Computation [6.525197444717069]
  GEVO-ML is a tool for discovering optimization opportunities and tuning the performance of Machine Learning kernels. We demonstrate GEVO-ML on two different ML workloads for both model training and prediction. GEVO-ML finds significant improvements for these models, achieving 90.43% performance improvement when model accuracy is relaxed by 2%.
  arXiv  Detail & Related papers  (2023-10-16T09:24:20Z)
• Multi-Granularity Detector for Vulnerability Fixes [13.653249890867222]
  We propose MiDas (Multi-Granularity Detector for Vulnerability Fixes) to identify vulnerability-fixing commits. MiDas constructs different neural networks for each level of code change granularity, corresponding to commit-level, file-level, hunk-level, and line-level. MiDas outperforms the current state-of-the-art baseline in terms of AUC by 4.9% and 13.7% on Java and Python-based datasets.
  arXiv  Detail & Related papers  (2023-05-23T10:06:28Z)
• Confident Adaptive Language Modeling [95.45272377648773]
  CALM is a framework for dynamically allocating different amounts of compute per input and generation timestep. We demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $times 3$ -- while provably maintaining high performance.
  arXiv  Detail & Related papers  (2022-07-14T17:00:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.