Answer Is Cheap, Show Me the Evidence! Augmenting Automated Vulnerability Assessment with Evidence

• URL: http://arxiv.org/abs/2608.25905v1
• Date: Wed, 26 Aug 2026 15:23:36 GMT
• Title: Answer Is Cheap, Show Me the Evidence! Augmenting Automated Vulnerability Assessment with Evidence
• Abstract summary: We propose EAVA, a framework that uses large language models to assess software vulnerability reports.<n>EAVA employs specialized LLM agents to process rich-text content and project information.<n>EAVA outperforms the strongest baseline by 5.3 to 35.2 percent across multiple metrics.
• Score: 10.654006827286912
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software vulnerability (SV) assessment helps prioritize remediation by characterizing reported vulnerabilities. Existing automated methods predict assessment results from SV reports (SVRs), but often overlook information in rich text, such as screenshots and code snippets, as well as contextual information about vulnerable projects. They also focus on prediction accuracy without providing explanations or supporting evidence, limiting their practical use when analysts must validate imperfect predictions. We propose EAVA, a framework that uses large language models (LLMs) to assess SVs and provide supporting evidence. EAVA employs specialized LLM agents to process rich-text content and project information, and builds a dedicated assessment model through a two-stage training pipeline. It first uses supervised instruction tuning on automatically annotated reasoning trajectories to inject domain knowledge, and then applies reinforcement learning to improve intrinsic reasoning. EAVA also retrieves similar historical vulnerabilities as supplementary evidence. Experiments on a newly collected SVR dataset show that EAVA outperforms the strongest baseline by 5.3 to 35.2 percent across multiple metrics. Ablation studies confirm the effectiveness of assessment-specific model training and information enrichment. A user study with security experts further demonstrates that the evidence provided by EAVA is useful and practical for real-world SV assessment.

Related papers

• When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards [49.25249414962884]
  We propose Reinforcement Learning with Active Verifiable Rewards (RLAVR)<n>RLAVR actively acquires ground-truth labels for a small set of selected samples and integrates them with pseudo-labels.<n>Building on this, we introduce Correction-Aware Reliability Estimation for RLAVR (CARE)
  arXiv  Detail & Related papers  (2026-05-25T13:55:12Z)
• ReVul-CoT: Towards Effective Software Vulnerability Assessment with Retrieval-Augmented Generation and Chain-of-Thought Prompting [9.735224996021591]
  We propose a novel framework that integrates Retrieval-Augmented Generation (RAG) with Chain-of-Thought (COT) prompting.<n>In ReVul-CoT, the RAG module dynamically retrieves contextually relevant information from a constructed local knowledge base.<n>Building on DeepSeek-V3.1, CoT prompting guides the LLM to perform step-by-step reasoning over exploitability, impact scope, and related factors.
  arXiv  Detail & Related papers  (2025-11-21T08:01:49Z)
• Look As You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning [55.232400251303794]
  Look As You Think (LAT) is a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution.<n>LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in IoU@0.5.
  arXiv  Detail & Related papers  (2025-11-15T02:50:23Z)
• Towards Real-Time Fake News Detection under Evidence Scarcity [66.58597356379907]
  We propose Evaluation-Aware Selection of Experts (EASE), a novel framework for real-time fake news detection.<n>EASE adapts its decision-making process according to the assessed sufficiency of available evidence.<n>We introduce RealTimeNews-25, a new benchmark for evaluating model generalization on emerging news with limited evidence.
  arXiv  Detail & Related papers  (2025-10-13T11:11:46Z)
• Weaponizing Language Models for Cybersecurity Offensive Operations: Automating Vulnerability Assessment Report Validation; A Review Paper [0.0]
  Large Language Models (LLMs) have emerged as a highly promising tool for defensive and offensive cybersecurity-related strategies.<n>This paper investigates the capabilities of LLMs in automating and improving the validation process of the Vulnerability Assessment (VA) report.
  arXiv  Detail & Related papers  (2025-05-07T09:14:55Z)
• R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation [11.15622721122057]
  Large language models (LLMs) have shown promising performance in software vulnerability detection, yet their reasoning capabilities remain unreliable.<n>We propose R2Vul, a method that combines reinforcement learning from AI feedback (RLAIF) and structured reasoning distillation.<n>We evaluate R2Vul across five programming languages and against four static analysis tools.
  arXiv  Detail & Related papers  (2025-04-07T03:04:16Z)
• EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment [17.74561647070259]
  We introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of software vulnerability (SV) assessment.<n>EvalSVA offers a human-like process and generates both reason and answer for SV assessment.
  arXiv  Detail & Related papers  (2024-12-11T08:00:50Z)
• A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedback [7.742213291781287]
  We present VRpilot, a vulnerability repair technique based on reasoning and patch validation feedback. Our results show that VRpilot generates, on average, 14% and 7.6% more correct patches than the baseline techniques on C and Java.
  arXiv  Detail & Related papers  (2024-05-24T16:29:48Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Context-faithful Prompting for Large Language Models [51.194410884263135]
  Large language models (LLMs) encode parametric knowledge about world facts. Their reliance on parametric knowledge may cause them to overlook contextual cues, leading to incorrect predictions in context-sensitive NLP tasks. We assess and enhance LLMs' contextual faithfulness in two aspects: knowledge conflict and prediction with abstention.
  arXiv  Detail & Related papers  (2023-03-20T17:54:58Z)
• Improving the Adversarial Robustness of NLP Models by Information Bottleneck [112.44039792098579]
  Non-robust features can be easily manipulated by adversaries to fool NLP models. In this study, we explore the feasibility of capturing task-specific robust features, while eliminating the non-robust ones by using the information bottleneck theory. We show that the models trained with our information bottleneck-based method are able to achieve a significant improvement in robust accuracy.
  arXiv  Detail & Related papers  (2022-06-11T12:12:20Z)
• Design Guidelines for Inclusive Speaker Verification Evaluation Datasets [0.6015898117103067]
  Speaker verification (SV) provides billions of voice-enabled devices with access control, and ensures the security of voice-driven technologies. Current SV evaluation practices are insufficient for evaluating bias: they are over-simplified and aggregate users, not representative of real-life usage scenarios. This paper proposes design guidelines for constructing SV evaluation datasets that address these short-comings.
  arXiv  Detail & Related papers  (2022-04-05T15:28:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.