R+R: Reassessing Java Security API Misuse in Current LLMs: A Replication on JCA and JSSE APIs with External Security Knowledge

• URL: http://arxiv.org/abs/2605.31135v1
• Date: Fri, 29 May 2026 10:46:20 GMT
• Title: R+R: Reassessing Java Security API Misuse in Current LLMs: A Replication on JCA and JSSE APIs with External Security Knowledge
• Abstract summary: This paper replicates Mousavi et al.'s study on the Java Cryptography Architecture (JCA) and Java Secure Socket Extension (JSSE) APIs.<n>We focus on two complementary settings: GPT-5.5 as a frontier proprietary coding model, and Llama-3.3-70B-Instruct as a strong open-weight model relevant to self-hosted deployment.
• Score: 2.255599116578844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The misuse of Java security APIs is a serious security problem in software development. Research in 2024 has shown that this problem is widespread in LLM-generated code. However, it remains unclear whether this phenomenon persists in current models and how external security knowledge affects it. This paper presents a scoped replication and extension of Mousavi et al.'s study on the Java Cryptography Architecture (JCA) and Java Secure Socket Extension (JSSE) APIs. We focus on two complementary settings: GPT-5.5 as a frontier proprietary coding model, and Llama-3.3-70B-Instruct as a strong open-weight model relevant to self-hosted deployment. The results show that although newer LLMs perform better in using Java security APIs, the problem of Java security API misuse has not been eliminated. External security knowledge substantially improves the measured outcome, but its effect is model-dependent. For Llama-3.3-70B-Instruct, secure code examples are the most effective single knowledge type. For GPT-5.5, explicit misuse patterns eliminate all detected security API misuses among valid programs in our benchmark, although some outputs remain invalid due to compilation errors or target-API mismatches. In addition, developer-guide knowledge becomes much more effective, and secure prompting also provides large gains for GPT-5.5. Overall, these findings confirm the Java security API misuse risk identified in the original study and show that the benefits of retrieval-augmented knowledge depend not only on the knowledge itself and retrieval behavior, but also on model capability.

Related papers

• SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
  Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
  arXiv  Detail & Related papers  (2026-04-04T04:29:11Z)
• Contextualizing Sink Knowledge for Java Vulnerability Discovery [7.342448037592857]
  GONDAR is a sink-centric fuzzing framework that systematically leverages sink API semantics for targeted vulnerability discovery.<n>GONDAR first identifies reachable and exploitable sink call sites through CWE-specific scanning.<n>It then deploys two specialized agents that work collaboratively with a coverage-guided fuzzer.
  arXiv  Detail & Related papers  (2026-04-02T05:43:25Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• Jailbreaking in the Haystack [64.82605038772253]
  We introduce NINJA (short for Needle-in-haystack jailbreak attack), a method that jailbreaks aligned LMs by appending benign, model-generated content to harmful user goals.<n>We show that NINJA significantly increases attack success rates across state-of-the-art open and proprietary models, including LLaMA, Qwen, Mistral, and Gemini.<n>These findings reveal that even benign long contexts -- when crafted with careful goal positioning -- introduce fundamental vulnerabilities in modern LMs.
  arXiv  Detail & Related papers  (2025-11-05T01:12:50Z)
• Are Sparse Autoencoders Useful for Java Function Bug Detection? [5.119371135458389]
  Software vulnerabilities are a major source of security breaches.<n>Traditional methods for vulnerability detection are limited by high false positive rates, scalability issues, and reliance on manual effort.<n>Sparse Autoencoder offer a promising solution to this problem.
  arXiv  Detail & Related papers  (2025-05-15T14:59:17Z)
• Mining for Lags in Updating Critical Security Threats: A Case Study of Log4j Library [2.593806238402966]
  Delays in applying patch updates can leave client systems exposed to exploitation.<n>We identify factors influencing update lags and categorize them based on version classification.<n>Results show that lags exist, but projects with higher release cycle rates tend to address severe security issues more swiftly.
  arXiv  Detail & Related papers  (2025-04-14T03:02:16Z)
• SQL Injection Jailbreak: A Structural Disaster of Large Language Models [71.55108680517422]
  Large Language Models (LLMs) are susceptible to jailbreak attacks that can induce them to generate harmful content.<n>In this paper, we introduce a novel jailbreak method, which targets the external properties of LLMs.<n>By injecting jailbreak information into user prompts, SIJ successfully induces the model to output harmful content.
  arXiv  Detail & Related papers  (2024-11-03T13:36:34Z)
• Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities [50.980446687774645]
  We introduce ADV-LLM, an iterative self-tuning process that crafts adversarial LLMs with enhanced jailbreak ability.<n>Our framework significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100% ASR on various open-source LLMs.<n>It exhibits strong attack transferability to closed-source models, achieving 99% ASR on GPT-3.5 and 49% ASR on GPT-4, despite being optimized solely on Llama3.
  arXiv  Detail & Related papers  (2024-10-24T06:36:12Z)
• What's in a Package? Getting Visibility Into Dependencies Using Security-Sensitive API Calls [3.6525326603691504]
  We present a novel methodology to construct a security-sensitive API list for an ecosystem.<n>We then compare the prevalence of security-sensitive APIs in functionally similar package groups.<n>More than half of the developers would use security-sensitive API information in the dependency selection process if available.
  arXiv  Detail & Related papers  (2024-08-05T22:01:18Z)
• An Investigation into Misuse of Java Security APIs by Large Language Models [9.453671056356837]
  This paper systematically assesses ChatGPT's trustworthiness in code generation for security API use cases in Java. Around 70% of the code instances across 30 attempts per task contain security API misuse, with 20 distinct misuse types identified. For roughly half of the tasks, this rate reaches 100%, indicating that there is a long way to go before developers can rely on ChatGPT to securely implement security API code.
  arXiv  Detail & Related papers  (2024-04-04T22:52:41Z)
• How Effective Are Neural Networks for Fixing Security Vulnerabilities [25.050519427592253]
  This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data.
  arXiv  Detail & Related papers  (2023-05-29T20:50:27Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.