Who Said CVE? How Vulnerability Identifiers Are Mentioned by Humans, Bots, and Agents in Pull Requests
In this paper, we study how vulnerability IDs (like CVE, CWE, and GHSA) are mentioned in Github pull requests. By examining who mentions these IDs, we can better assess whether autonomous agents are truly participating in security maintenance tasks. The highlights of our work include:- Augmented Dataset: We are sharing an augmented dataset with additional pull requests from the same repositories found in the AIDev-pop data, a dataset designed to capture the activity of AI agents in real-world GitHub repositories.
- Quantitative Analysis: Our data shows that bots account for 69.1% of all mentions. While humans and agents mention IDs less frequently, their mentions appear across titles and commit messages, whereas bots primarily mention vulnerabilities in PR descriptions.
- Qualitative Analysis: Our manual review reveals that bots primarily use these IDs in automated dependency updates, while humans and agents use them to provide context for fixes.
Link to arXiv