BackportBench: A Multilingual Benchmark for Automated Backporting of Patches

• URL: http://arxiv.org/abs/2512.01396v1
• Date: Mon, 01 Dec 2025 08:16:43 GMT
• Title: BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
• Abstract summary: Software developers can mitigate this problem by backporting security patches to older releases.<n>BackportBench is the first comprehensive benchmark suite for patch backporting problem.<n>The results show that the agentic method has outperformed traditional patch porting methods.
• Score: 15.032583823113514
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many modern software projects evolve rapidly to incorporate new features and security patches. It is important for users to update their dependencies to safer versions, but many still use older, vulnerable package versions because upgrading can be difficult and may break their existing codebase. Software developers can mitigate this problem by backporting security patches to older releases. However, manually backporting is time-consuming and error-prone. The effectiveness of existing automated backporting techniques on general software remains unclear since they typically target only code-hunk or function-level patch porting scenarios and are evaluated with imperfect metrics. To facilitate the development and evaluation of automated backporting techniques, we introduce BackportBench, the first comprehensive benchmark suite for patch backporting problem. BackportBench is a multilingual benchmark that contains 202 patch backporting problems from PyPI, Maven, and npm, each with executable Docker environments and relevant test cases. We evaluated existing patch porting methods and LLM-based techniques that have the potential to adapt to this task using BackportBench. The results show that the agentic method has outperformed traditional patch porting methods, especially on cases that require logical and structural changes. However, the performance varies across different programming languages. Based on the findings, we draw several implications for researchers and software practitioners in future work on automated backporting.

Related papers

• VeriPort: Automated and Verified Patch Backporting at Scale [41.76637014069122]
  We present VeriPort, an end-to-end agentic system that backports a patch for a given vulnerability advisory to every affected version of the package.<n>For each backport, VeriPort builds a chain of evidence to confirm that the patch blocks exploitation and preserves intended behavior.<n>We deployed VeriPort on 169 high- and critical-severity CVEs and have generated over 5,000 verified backported patches.
  arXiv  Detail & Related papers  (2026-06-21T22:45:33Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks [12.573674060643787]
  TimeMachine-bench is a benchmark designed to evaluate software migration in real-world Python projects.<n>Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates.
  arXiv  Detail & Related papers  (2026-01-30T05:42:45Z)
• A Benchmark for Language Models in Real-World System Building [56.549267258789904]
  Cross-ISA software package repair is a critical task for ensuring the reliability of software deployment and the stability of modern operating systems.<n>We introduce a new benchmark designed for software package build repair across diverse architectures and languages.<n>We evaluate six state-of-the-art LLMs on the benchmark, and the results show that cross-ISA software package repair remains difficult and requires further advances.
  arXiv  Detail & Related papers  (2026-01-19T10:30:46Z)
• What a diff makes: automating code migration with large language models [0.15293427903448018]
  We show that contexts containing diffs can significantly improve performance against out of the box LLMs.<n>We provide a dataset to assist in further development of this problem area, as well as an open-source Python package, AIMigrate, that can be used to assist with migrating code bases.<n>In a real-world migration of TYPHOIDSIM between STARSIM versions, AIMigrate correctly identified 65% of required changes in a single run, increasing to 80% with multiple runs, with 47% of changes generated perfectly.
  arXiv  Detail & Related papers  (2025-10-31T18:08:52Z)
• PortGPT: Towards Automated Backporting Using Large Language Models [16.654479541363802]
  We introduce PORTGPT, an agent for end-to-end automation of patch backporting in real-world scenarios.<n> PORTGPT enhances an LLM with tools to access code on-demand, summarize Git history, and revise patches autonomously based on feedback.<n>We contributed 9 backported patches from PORTGPT to the Linux kernel community and all patches are now merged.
  arXiv  Detail & Related papers  (2025-10-25T18:46:04Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions [53.811953357289866]
  Large language models (LLMs) have shown remarkable progress across various domains.<n>LLMs struggle with incomplete code context understanding and inaccurate migration point identification.<n>MigGPT is a framework that employs a novel code fingerprint structure to retain code snippet information.
  arXiv  Detail & Related papers  (2025-04-13T08:08:37Z)
• SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents [49.73885480071402]
  We introduce SWE-PolyBench, a new benchmark for repository-level, execution-based evaluation of coding agents.<n>SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code.<n>Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks.
  arXiv  Detail & Related papers  (2025-04-11T17:08:02Z)
• MultiMend: Multilingual Program Repair with Context Augmentation and Multi-Hunk Patch Generation [2.7036595757881323]
  MultiMend is a learning-based APR approach designed to improve repair performance on multiple programming languages.<n>It embeds source code lines and applies retrieval-augmented generation to augment the buggy context with relevant lines during patch generation.<n>We evaluate MultiMend on four benchmarks with four programming languages and compare it with state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-27T13:37:43Z)
• SWE-bench-java: A GitHub Issue Resolving Benchmark for Java [27.226354754864783]
  SWE-bench has been released to evaluate issue resolving capabilities of large language models (LLMs) As a first step toward multilingual support, we have developed a Java version of SWE-bench, called SWE-bench-java. To verify the reliability of SWE-bench-java, we implement a classic method SWE-agent and test several powerful LLMs on it.
  arXiv  Detail & Related papers  (2024-08-26T15:30:05Z)
• Automating Zero-Shot Patch Porting for Hard Forks [12.622409703778604]
  We propose a large language model (LLM) based approach to automatically port patches for hard forks on a function-wise basis. PPatHF can correctly port 131 (42.3%) patches and automate 57% of the manual edits required for the developer to port the patch.
  arXiv  Detail & Related papers  (2024-04-27T17:40:39Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.