DRS-OSS: LLM-Driven Diff Risk Scoring Tool for PR Risk Prediction

• URL: http://arxiv.org/abs/2511.21964v1
• Date: Wed, 26 Nov 2025 22:52:18 GMT
• Title: DRS-OSS: LLM-Driven Diff Risk Scoring Tool for PR Risk Prediction
• Abstract summary: Diff Risk Scoring (DRS) estimates the likelihood that a diff will introduce a defect, enabling better review prioritization, test planning, and CI/CD gating.<n>DRS-OSS is an open-source DRS system equipped with a public API, web UI, and GitHub plugin.
• Score: 3.17083738531489
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In large-scale open-source projects, hundreds of pull requests land daily, each a potential source of regressions. Diff Risk Scoring (DRS) estimates the likelihood that a diff will introduce a defect, enabling better review prioritization, test planning, and CI/CD gating. We present DRS-OSS, an open-source DRS system equipped with a public API, web UI, and GitHub plugin. DRS-OSS uses a fine-tuned Llama 3.1 8B sequence classifier trained on the ApacheJIT dataset, consuming long-context representations that combine commit messages, structured diffs, and change metrics. Through parameter-efficient adaptation, 4-bit QLoRA, and DeepSpeed ZeRO-3 CPU offloading, we train 22k-token contexts on a single 20 GB GPU. On the ApacheJIT benchmark, DRS-OSS achieves state-of-the-art performance (F1 = 0.64, ROC-AUC = 0.89). Simulations show that gating only the riskiest 30% of commits can prevent up to 86.4% of defect-inducing changes. The system integrates with developer workflows through an API gateway, a React dashboard, and a GitHub App that posts risk labels on pull requests. We release the full replication package, fine-tuning scripts, deployment artifacts, code, demo video, and public website.

Related papers

• MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves [65.60481414910728]
  Recursive self-improvement lets a system improve the model-building machinery from its own failures, so every later model inherits the gain.<n>Yet RSI has been validated almost exclusively on coding and formal benchmarks such as science QA and mathematics.<n>We argue RSI must next operate across real, diverse scientific, engineering, and meta-scientific domains, not where formal evaluation is merely tractable.<n>We present MetaRSI-v1, where improvement is the scheduled composition of three typed operators over one unified paradigm.
  arXiv  Detail & Related papers  (2026-09-06T05:15:31Z)
• VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies [5.733713042530412]
  VAKRA is a benchmark of over $8,000$ executable APIs across $62$ domains.<n>It tests diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints.<n>Using a fixed ReAct harness, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4% on single-hop endpoint-style tasks.
  arXiv  Detail & Related papers  (2026-08-12T17:27:27Z)
• Taming the Drift: Context-aware Repair of Dockerfile Drift during Software Evolution [52.50730821321986]
  Cadre is a context-aware framework for automated Dockerfile drift repair.<n>CDG maps each Dockerfile instruction to its file-level and inter-instruction dependencies.<n>DodeX mines real-world Dockerfile drift instances from GitHub Actions CI logs.
  arXiv  Detail & Related papers  (2026-07-14T09:12:26Z)
• Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs [0.0]
  We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation.<n>Phoenix decomposes the work across six specialized agents.<n>A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation.
  arXiv  Detail & Related papers  (2026-06-18T13:56:12Z)
• Pull Requests as a Training Signal for Repo-Level Code Editing [49.82435173554125]
  Clean Pull Request (Clean-PR) is a mid-training paradigm that leverages real-world GitHub pull requests as a training signal for repository-level editing.<n>We introduce a scalable pipeline that converts noisy pull request diffs into Search/Replace edit blocks through reconstruction and validation.<n>On SWE-bench, our model significantly outperforms the instruction-tuned baseline, achieving absolute improvements of 13.6% on SWE-bench Lite and 12.3% on SWE-bench Verified.
  arXiv  Detail & Related papers  (2026-02-07T09:22:25Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation [8.176905459241047]
  We pioneer the exploration of attack surfaces in Retrieval-Augmented Code Generation.<n>We show how poisoned documentation containing hidden malicious dependencies can subvert RACG.<n>We propose ImportSnare, a novel attack framework employing two synergistic strategies.
  arXiv  Detail & Related papers  (2025-09-09T17:21:20Z)
• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights [75.83625828306839]
  textbfDrag-and-Drop LLMs (textitDnD) eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates.<n>A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices.
  arXiv  Detail & Related papers  (2025-06-19T15:38:21Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution [56.9361004704428]
  Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks.<n>SWE-Fixer is a novel open-source framework designed to effectively and efficiently resolve GitHub issues.<n>We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving competitive performance among open-source models.
  arXiv  Detail & Related papers  (2025-01-09T07:54:24Z)
• ModSec-Learn: Boosting ModSecurity with Machine Learning [14.392409275321528]
  ModSecurity is widely recognized as the standard open-source Web Application Firewall (WAF) We propose a machine-learning model that uses the Core Rule Set (CRS) rules as input features. ModSec-Learn is able to tune the contribution of each CRS rule to predictions, thus adapting the severity level to the web applications to protect.
  arXiv  Detail & Related papers  (2024-06-19T13:32:47Z)
• VulLibGen: Generating Names of Vulnerability-Affected Packages via a Large Language Model [13.96251273677855]
  VulLibGen is a method to directly generate affected packages. It has an average accuracy of 0.806 for identifying vulnerable packages. We have submitted 60 vulnerability, affected package> pairs to GitHub Advisory.
  arXiv  Detail & Related papers  (2023-08-09T02:02:46Z)
• Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attacks [64.03012884804458]
  We propose a versatile framework based on random search, Sparse-RS, for sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on substitute models and achieves state-of-the-art success rate and query efficiency for multiple sparse attack models.
  arXiv  Detail & Related papers  (2020-06-23T08:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.