JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode
arXiv:2607.02825v1 Announce Type: cross Abstract: We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains $\sim$30{,}600 Java methods spanning 1{,}740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single \texttt{LlmPrediction} schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-
The proliferation of AI models, especially large language models (LLMs), has accelerated the need for robust and standardized vulnerability detection benchmarks, particularly for widely used languages like Java.
A sophisticated reader should care because this benchmark directly addresses critical cybersecurity risks in software development, enabling more effective and reliable AI-powered vulnerability detection and enhancing overall software supply chain security.
The introduction of JavaVulBench provides a standardized, realistic, and leakage-aware evaluation framework for AI models detecting Java vulnerabilities, moving beyond previous limited or biased datasets.
- · Cybersecurity researchers
- · Software developers
- · AI model developers
- · Organizations using Java applications
- · Cyber attackers exploiting Java vulnerabilities
Improved accuracy and reliability of AI-driven vulnerability scanners for Java applications.
Reduced incidence of critical security flaws in new and existing Java-based software, leading to fewer breaches.
Enhanced trust in AI-assisted code security, potentially accelerating the adoption of similar benchmarks for other programming languages and critical infrastructure.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI