JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

• URL: http://arxiv.org/abs/2607.02825v1
• Date: Thu, 02 Jul 2026 23:32:43 GMT
• Title: JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode
• Abstract summary: textscVulBench is a benchmark and evaluation harness for Java vulnerability evaluation.<n>The dataset contains $sim,600 Java methods spanning 1-14, 700+ projects, labelled at both line and publication dates.<n>The harness provides a single ttexttLlm schema across backend families.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains $\sim$30{,}600 Java methods spanning 1{,}740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single \texttt{LlmPrediction} schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet~4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model so users can separate genuinely unseen test CVEs from potentially memorised ones. Data, code, and fine-tuned checkpoints are archived on Zenodo [31] and short demonstration video is available on YouTube (https://www.youtube.com/watch?v=nMTX\_hqkuoM) https://www.youtube.com/watch?v=nMTX_hqkuoM.

Related papers

• Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations [0.0]
  LLM-based code generation is now embedded in mission-critical pipelines.<n>We test if an LLM reads a piece of C/C++ code as context, do its hidden activations already carry a signal about that code's vulnerability status?<n>This is early evidence that a coding LLM's own representation of arbitrary code is informative about that code's vulnerability status.
  arXiv  Detail & Related papers  (2026-08-17T08:52:22Z)
• Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents [2.417342411475111]
  Nautilus Compass is a black-box drift detector and agent memory layer for production coding agents.<n>The system ships as a Claude Code plugin, an MCP 20245 A2A server, a CLI, and an API on one daemon.
  arXiv  Detail & Related papers  (2026-05-11T01:49:17Z)
• An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning [10.815335026032377]
  TraceEval is the first execution-verified, multi-language benchmark for code semantic reasoning.<n>TraceEval consists of 10,583 real-world programs extracted from 1,600+ open-source repositories across Python, JavaScript, and Java.<n>The strongest model, Claude-Opus-4.6, reaches an average F1 of 72.9% across the three languages.
  arXiv  Detail & Related papers  (2026-05-10T07:17:00Z)
• LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests [0.0]
  Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed.<n>We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests.<n>We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API.
  arXiv  Detail & Related papers  (2026-04-13T21:05:42Z)
• Bridging Code Property Graphs and Language Models for Program Analysis [1.7158296436650338]
  Large Language Models (LLMs) face critical challenges when analyzing security vulnerabilities in real worlds.<n> token limits prevent loading entire repositories, code embeddings fail to capture inter procedural data flows, and LLMs struggle to generate complex static analysis queries.<n>We introduce codebadger, an open source Model Context Protocol (MCP) server that integrates Joern's Code Property Graph engine with LLMs.
  arXiv  Detail & Related papers  (2026-03-25T22:08:04Z)
• vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models [58.633451339058986]
  VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository.<n>We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution.<n>A complete evaluation requires only two commands: vla eval serve and vla eval run.
  arXiv  Detail & Related papers  (2026-03-14T14:38:53Z)
• Extracting books from production language models [65.85348210518937]
  It remains an open question if similar extraction is feasible for production LLMs.<n>In some cases, jailbroken Claude 3.7 Sonnet outputs entire books near-verbatim.<n>Even with model- and system-level safeguards, extraction of (in-copyright) training data remains a risk for production LLMs.
  arXiv  Detail & Related papers  (2026-01-06T03:01:27Z)
• I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution [0.0580448704422069]
  This paper presents the first systematic study of authorship attribution for C programs.<n>We release CodeT5-Authorship, a novel model that uses only the encoder layers from the original CodeT5 encoder-decoder architecture.<n>Our model achieves 97.56% accuracy in distinguishing C programs generated by closely related models.
  arXiv  Detail & Related papers  (2025-06-18T19:49:41Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• CLEVER: A Curated Benchmark for Formally Verified Code Generation [53.5486188696892]
  $rm Csmall LEVER$ is a high-quality, curated benchmark of 161 problems for end-to-end verified code generation in Lean.<n>Each problem consists of (1) the task of generating a specification that matches a held-out ground-truth specification, and (2) the task of generating a Lean implementation that provably satisfies this specification.
  arXiv  Detail & Related papers  (2025-05-20T05:15:47Z)
• DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories [83.5195424237358]
  Existing benchmarks are poorly aligned with real-world code repositories. We propose a new benchmark named DevEval, which has three advances. DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains.
  arXiv  Detail & Related papers  (2024-05-30T09:03:42Z)
• Private-Library-Oriented Code Generation with Large Language Models [52.73999698194344]
  This paper focuses on utilizing large language models (LLMs) for code generation in private libraries. We propose a novel framework that emulates the process of programmers writing private code. We create four private library benchmarks, including TorchDataEval, TorchDataComplexEval, MonkeyEval, and BeatNumEval.
  arXiv  Detail & Related papers  (2023-07-28T07:43:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.