An Empirical Analysis of CodeQL False Positives and Query Refinements for Java Vulnerabilities

• URL: http://arxiv.org/abs/2609.04535v1
• Date: Thu, 03 Sep 2026 22:48:21 GMT
• Title: An Empirical Analysis of CodeQL False Positives and Query Refinements for Java Vulnerabilities
• Abstract summary: We run CodeQL's Java security query suite on 167 CVE instances from 110 projects.<n>We manually review 500 sampled false positive paths and locations and construct a source-level taxonomy.<n>We implement CodeQL refinements that detect and filter recurring false positive patterns at the query level.
• Score: 7.697021102902879
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Static application security testing (SAST) tools help developers find vulnerabilities before deployment, but false positives create substantial triage effort. We study whether CodeQL false positives in Java security analysis form recurring, explainable patterns that can be reduced by refining the analysis. We run CodeQL's Java security query suite on 167 CVE instances from 110 projects, focusing on the ten queries with the highest false positive rates. We manually review 500 sampled false positive paths and locations and construct a source-level taxonomy. The five categories are Missed Path Constraint or Sanitization (36.6%), Benign Execution Context (29.4%), Missing Trust Boundary Modeling (27.6%), Imprecise Concurrency Modeling (5%), and Imprecise Sink Modeling (1.4%). Guided by these findings, we implement CodeQL refinements that detect and filter recurring false positive patterns at the query level. The refinements remove 81.8% of reviewed false positives. Across the full selected-query dataset, they remove 15.8% of reported paths and locations while retaining 7 of 8 true positives. This shows that many false positives can be reduced in the analysis, although fixed refinements often depend on project-specific context. To address this generalization gap, we evaluate whether agentic coding tools can adapt refinement patterns to new projects. Given our patterns as templates, the two tools succeed on 56% and 62% of tasks, with query compile-pass rates above 90%. Without this guidance, both succeed on only 28%, while compile rates fall to 30-36%. These results support a refinement-oriented SAST workflow in which recurring false positives are modeled in CodeQL queries and automatically adapted to different project contexts, reducing repeated triage.

Related papers

• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation [1.8589311604031529]
  We present CodeAssay, a taxonomy-first benchmark of 185 Python tasks across ten software-engineering categories.<n>It combines audited ground truth, public tests for generation and repair, hidden tests for grading, mutation-based test-suite validation, and selected code-property measures.
  arXiv  Detail & Related papers  (2026-08-04T12:15:17Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testing [0.0]
  We present QASecClaw, a multi agent approach that combines Static Application Security Testing with coding specialized Large Language Model based contextual code review.<n>A SAST engine first reports candidate vulnerabilities, and a Large Language Model based SAST Filter Agent then reviews each finding with source code context to decide whether it is likely to be a true positive or a false positive.
  arXiv  Detail & Related papers  (2026-05-03T14:05:52Z)
• LLM-Guided Issue Generation from Uncovered Code Segments [2.9610847900970856]
  IssueSpecter finds bugs in uncovered code segments and automatically generates prioritized, actionable issue reports.<n>We evaluate IssueSpecter on 13 actively maintained Python projects, generating 10,467 issue reports.
  arXiv  Detail & Related papers  (2026-04-28T21:10:53Z)
• Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL [1.5773713958458309]
  No open, execution-based benchmark exists for mapping natural language to Jira Query Language (JQL)<n>We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues.
  arXiv  Detail & Related papers  (2026-04-10T16:27:31Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• Evaluating NL2SQL via SQL2NL [45.88028371034407]
  New framework generates semantically equivalent, lexically diverse queries.<n>State-of-the-art models are far more brittle than standard benchmarks suggest.
  arXiv  Detail & Related papers  (2025-09-04T21:03:59Z)
• CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification [71.34070740261072]
  This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases.<n>The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
  arXiv  Detail & Related papers  (2025-02-12T21:42:56Z)
• ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis [50.020850767257095]
  We propose ProofAug, a procedure that equips LLMs with automation methods at various granularities.<n>Our method is validated on the miniF2F benchmark using the open-source deep-math-7b-base model and the Isabelle proof assistant.<n>We also implement a Lean 4 version of ProofAug that can improve the pass@1 performance of Kimina-Prover-seek-Distill-1.5B from 44.3% to 50.4%.
  arXiv  Detail & Related papers  (2025-01-30T12:37:06Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• Finding 709 Defects in 258 Projects: An Experience Report on Applying CodeQL to Open-Source Embedded Software (Experience Paper) -- Extended Report [5.1138836487878825]
  SAST tools are rarely used -- only 3% of projects go beyond trivial compiler analyses.<n>Developers cited the perception of ineffectiveness and false positives as reasons for limited adoption.<n>Across the 258 projects, CodeQL reported 709 true defects with a false positive rate of 34%.<n>There were 535 (75%) likely security vulnerabilities, including in major projects maintained by Microsoft, Amazon, and the Apache Foundation.
  arXiv  Detail & Related papers  (2023-09-30T00:49:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.