Who Said CVE? How Vulnerability Identifiers Are Mentioned by Humans, Bots, and Agents in Pull Requests

• URL: http://arxiv.org/abs/2601.19636v1
• Date: Tue, 27 Jan 2026 14:13:08 GMT
• Title: Who Said CVE? How Vulnerability Identifiers Are Mentioned by Humans, Bots, and Agents in Pull Requests
• Abstract summary: This paper compares vulnerability ID use in GitHub pull requests authored by autonomous agents, bots, and human developers.<n>Bots account for around 69.1% of all mentions, usually adding few identifiers in pull request descriptions.<n>Humans and agents use them to support fixes, maintenance, and discussion.
• Score: 8.26054481096776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vulnerability identifiers such as CVE, CWE, and GHSA are standardised references to known software security issues, yet their use in practice is not well understood. This paper compares vulnerability ID use in GitHub pull requests authored by autonomous agents, bots, and human developers. Using the AIDev pop dataset and an augmented set of pull requests from the same repositories, we analyse who mentions vulnerability identifiers and where they appear. Bots account for around 69.1% of all mentions, usually adding few identifiers in pull request descriptions, while human and agent mentions are rarer but span more locations. Qualitative analysis shows that bots mainly reference identifiers in automated dependency updates and audits, whereas humans and agents use them to support fixes, maintenance, and discussion.

Related papers

• Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents [2.657356138730743]
  We formalize agent reconnaissance by modeling the process and identifying the knowledge assets it seeks to extract.<n>We instantiate these insights in Know Your Agent (KYA), a framework that automates black-box, reconnaissance-driven pentesting.
  arXiv  Detail & Related papers  (2026-07-22T07:20:27Z)
• How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests [8.26054481096776]
  We describe our planned study of vulnerability communication in pull requests by humans, bots, and coding agents.<n>We analyze explicit vulnerability references and implicit security-related signals across pull request titles, descriptions, review comments, commit messages, and timeline discussions.
  arXiv  Detail & Related papers  (2026-06-26T14:26:13Z)
• SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents [21.409263140916327]
  Large language model (LLM) reviewers are increasingly used in pull-request (PR)<n>This raises a question that benchmarks for static vulnerability detection or code generation do not address.<n>We introduce SEVRA-BENCH, a benchmark that measures how often an automated reviewer approves such pull requests.
  arXiv  Detail & Related papers  (2026-06-11T16:35:56Z)
• These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests [0.0]
  We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset.<n>We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans.
  arXiv  Detail & Related papers  (2026-05-04T06:32:50Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics [9.25340189071758]
  We present iAgentBench, a dynamic ODQA benchmark for cross-source sensemaking.<n>iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions.<n>Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks.
  arXiv  Detail & Related papers  (2026-03-04T22:40:08Z)
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• A Trajectory-Based Safety Audit of Clawdbot (OpenClaw) [77.1549110891026]
  We present a trajectory-centric evaluation of Clawdbot across six risk dimensions.<n>We log complete interaction trajectories (messages, actions, tool-call arguments/outputs) and assess safety using both an automated trajectory judge and human review.
  arXiv  Detail & Related papers  (2026-02-16T00:33:02Z)
• Favia: Forensic Agent for Vulnerability-fix Identification and Analysis [5.43098755190303]
  We propose Favia, a forensic, agent-based framework for vulnerability-fix identification.<n>Favia combines scalable candidate ranking with deep and iterative semantic reasoning.<n>We evaluate Favia on CVEVC, a large-scale dataset we made that comprises over 8 million commits from 3,708 real-world repositories.
  arXiv  Detail & Related papers  (2026-02-13T00:51:22Z)
• Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub [4.409447722044799]
  This study aims to characterize how autonomous coding agents contribute to software security in practice.<n>We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset.<n>We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes.
  arXiv  Detail & Related papers  (2026-01-01T21:14:11Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Attacks by Content: Automated Fact-checking is an AI Security Issue [1.5770741892463622]
  Previous research has studied indirect prompt injection, where the attacker injects malicious instructions.<n>We argue that injection of instructions is not necessary to manipulate agents - attackers could instead supply biased, misleading, or false information.<n>Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content.
  arXiv  Detail & Related papers  (2025-10-13T10:18:48Z)
• On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub [6.7302091035327285]
  Large language models (LLMs) are increasingly being integrated into software development processes.<n>The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice.<n>We empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 open-source projects.
  arXiv  Detail & Related papers  (2025-09-18T08:48:32Z)
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents [60.78202583483591]
  We introduce OS-Harm, a new benchmark for measuring safety of computer use agents.<n> OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior.<n>We evaluate computer use agents based on a range of frontier models and provide insights into their safety.
  arXiv  Detail & Related papers  (2025-06-17T17:59:31Z)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents [74.6761188527948]
  Computer-Use Agents (CUAs) with full system access pose significant security and privacy risks.<n>We investigate Visual Prompt Injection (VPI) attacks, where malicious instructions are visually embedded within rendered user interfaces.<n>Our empirical study shows that current CUAs and BUAs can be deceived at rates of up to 51% and 100%, respectively, on certain platforms.
  arXiv  Detail & Related papers  (2025-06-03T05:21:50Z)
• Reducing Alert Fatigue via AI-Assisted Negotiation: A Case for Dependabot [2.9497620776025046]
  This paper presents a position on using Artificial Intelligence (AI) agents as dependency negotiators.<n>We examine specific use cases where AI agents can facilitate dependency negotiations.<n>Our goal is to reduce alert fatigue to an extent that maintainers no longer feel overwhelmed and welcome pull requests.
  arXiv  Detail & Related papers  (2025-02-10T05:58:21Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• My Brother Helps Me: Node Injection Based Adversarial Attack on Social Bot Detection [69.99192868521564]
  Social platforms such as Twitter are under siege from a multitude of fraudulent users. Due to the structure of social networks, the majority of methods are based on the graph neural network(GNN), which is susceptible to attacks. We propose a node injection-based adversarial attack method designed to deceive bot detection models.
  arXiv  Detail & Related papers  (2023-10-11T03:09:48Z)
• A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments [70.1864008701113]
  Bots are used in Github repositories to automate repetitive activities that are part of the distributed software development process. This paper proposes a ground-truth dataset, based on a manual analysis with high interrater agreement, of pull request and issue comments in 5,000 distinct Github accounts. We propose an automated classification model to detect bots, taking as main features the number of empty and non-empty comments of each account, the number of comment patterns, and the inequality between comments within comment patterns.
  arXiv  Detail & Related papers  (2020-10-07T09:30:52Z)
• Detecting and Characterizing Bots that Commit Code [16.10540443996897]
  We propose a systematic approach to detect bots using author names, commit messages, files modified by the commit, and projects associated with the ommits. We have compiled a shareable dataset containing detailed information about 461 bots we found (all of whom have more than 1000 commits) and 13,762,430 commits they created.
  arXiv  Detail & Related papers  (2020-03-02T21:54:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.