From Reviewers' Lens: Understanding Bug Bounty Report Invalid Reasons with LLMs

• URL: http://arxiv.org/abs/2511.18608v1
• Date: Sun, 23 Nov 2025 20:27:54 GMT
• Title: From Reviewers' Lens: Understanding Bug Bounty Report Invalid Reasons with LLMs
• Abstract summary: We collect a dataset of 9,942 disclosed bug bounty reports, including 1,400 invalid reports.<n>We evaluate whether state-of-the-art large language models can identify invalid reports.<n>We build a taxonomy of reasons for Information Disclosure vulnerabilities and incorporate it into a retrieval-augmented generation framework.
• Score: 1.8652189113472575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bug bounty platforms (e.g., HackerOne, BugCrowd) leverage crowd-sourced vulnerability discovery to improve continuous coverage, reduce the cost of discovery, and serve as an integral complement to internal red teams. With the rise of AI-generated bug reports, little work exists to help bug hunters understand why these reports are labeled as invalid. To improve report quality and reduce reviewers' burden, it is critical to predict invalid reports and interpret invalid reasons. In this work, we conduct an empirical study with the purpose of helping bug hunters understand the validity of reports. We collect a dataset of 9,942 disclosed bug bounty reports, including 1,400 invalid reports, and evaluate whether state-of-the-art large language models can identify invalid reports. While models such as GPT-5, DeepSeek, and a fine-tuned RoBERTa achieve strong overall accuracy, they consistently struggle to detect invalid cases, showing a tendency to over-accept reports. To improve invalidity detection, we build a taxonomy of rejection reasons for Information Disclosure vulnerabilities and incorporate it into a retrieval-augmented generation (RAG) framework. This approach substantially improves classification consistency and reduces bias. We also examine whether reviewer decisions may be influenced by factors beyond the content of the report. Our analysis shows that reporters with higher reputations tend to receive more favorable outcomes in borderline cases, suggesting that perceived expertise can influence review judgments. Overall, our findings highlight the challenges of invalid report identification and show that combining LLMs with structured reviewer knowledge can support more transparent and consistent vulnerability report review.

Related papers

• Characterizing and Mitigating False-Positive Bug Reports in the Linux Kernel [14.987479226824023]
  We present the first comprehensive empirical study of false-positive bug reports in the Linux kernel.<n>We manually construct a dataset of 2,006 bug reports comprising 1,509 genuine bugs and 497 false positives collected from Bugzilla and Syzkaller.<n>Among various prompting strategies, retrieval-augmented generation (RAG) performs best, achieving 91% recall and an F1 score of 88%.
  arXiv  Detail & Related papers  (2026-05-08T12:48:36Z)
• LLM-Guided Issue Generation from Uncovered Code Segments [2.9610847900970856]
  IssueSpecter finds bugs in uncovered code segments and automatically generates prioritized, actionable issue reports.<n>We evaluate IssueSpecter on 13 actively maintained Python projects, generating 10,467 issue reports.
  arXiv  Detail & Related papers  (2026-04-28T21:10:53Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review [54.141490756509306]
  We introduce PaperAudit-Bench, which consists of two components: PaperAudit-Dataset, an error dataset, and PaperAudit-Review, an automated review framework.<n>Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths.<n>We show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost.
  arXiv  Detail & Related papers  (2026-01-07T04:26:12Z)
• Community Notes are Vulnerable to Rater Bias and Manipulation [75.34858521118305]
  We evaluate the Community Notes algorithm using simulated data that models realistic rater and note behaviors.<n>We find that the algorithm suppresses a substantial fraction of genuinely helpful notes and is highly sensitive to rater biases.
  arXiv  Detail & Related papers  (2025-11-04T14:39:34Z)
• COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations [89.37527535663433]
  We present a large-scale dataset of 104k posts with user-provided notes and helpfulness labels.<n>We propose a framework that automatically generates and improves reason definitions via automatic prompt optimization.<n>Our experiments show that the optimized definitions can improve both helpfulness and reason prediction.
  arXiv  Detail & Related papers  (2025-10-28T05:28:47Z)
• Crash Report Enhancement with Large Language Models: An Empirical Study [21.842377922082104]
  We examine whether large language models can enhance crash reports by adding fault locations, root-cause explanations, and repair suggestions.<n>On a dataset of 492 real-world crash reports, LLM-enhanced reports improve Top-1 problem-localization accuracy from 10.6% to 40.2-43.1%.<n>Both our manual evaluations and LLM-as-a-judge assessment show that Agentic-LLM delivers stronger root-cause explanations and more actionable repair guidance.
  arXiv  Detail & Related papers  (2025-09-16T21:02:57Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers [59.168391398830515]
  We evaluate 12 pre-trained LLMs and one specialized fact-verifier, using a collection of examples from 14 fact-checking benchmarks.<n>We highlight the importance of addressing annotation errors and ambiguity in datasets.<n> frontier LLMs with few-shot in-context examples, often overlooked in previous works, achieve top-tier performance.
  arXiv  Detail & Related papers  (2025-06-16T10:32:10Z)
• The Impact Of Bug Localization Based on Crash Report Mining: A Developers' Perspective [7.952391285456257]
  We report our experience of using an approach for grouping crash reports and finding buggy code on a weekly basis for 18 months. The approach investigated in this study correctly suggested the buggy file most of the time -- the approach's precision was around 80%.
  arXiv  Detail & Related papers  (2024-03-16T01:23:01Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• Recommending Bug Assignment Approaches for Individual Bug Reports: An Empirical Investigation [8.186068333538893]
  Multiple approaches have been proposed to automatically recommend potential developers who can address bug reports. These approaches are typically designed to work for any bug report submitted to any software project. We conducted an empirical study to validate this conjecture, using three bug assignment approaches applied on 2,249 bug reports from two open source systems.
  arXiv  Detail & Related papers  (2023-05-29T23:02:56Z)
• Auto-labelling of Bug Report using Natural Language Processing [0.0]
  Rule and Query-based solutions recommend a long list of potential similar bug reports with no clear ranking. In this paper, we have proposed a solution using a combination of NLP techniques. It uses a custom data transformer, a deep neural network, and a non-generalizing machine learning method to retrieve existing identical bug reports.
  arXiv  Detail & Related papers  (2022-12-13T02:32:42Z)
• Early Detection of Security-Relevant Bug Reports using Machine Learning: How Far Are We? [6.438136820117887]
  In a typical maintenance scenario, security-relevant bug reports are prioritised by the development team when preparing corrective patches. Open security-relevant bug reports can become a critical leak of sensitive information that attackers can leverage to perform zero-day attacks. In recent years, approaches for the detection of security-relevant bug reports based on machine learning have been reported with promising performance.
  arXiv  Detail & Related papers  (2021-12-19T11:30:29Z)
• ScoreGAN: A Fraud Review Detector based on Multi Task Learning of Regulated GAN with Data Augmentation [50.779498955162644]
  We propose ScoreGAN for fraud review detection that makes use of both review text and review rating scores in the generation and detection process. Results show that the proposed framework outperformed the existing state-of-the-art framework, namely FakeGAN, in terms of AP by 7%, and 5% on the Yelp and TripAdvisor datasets.
  arXiv  Detail & Related papers  (2020-06-11T16:15:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.