Related papers: APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls

APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls

URL: http://arxiv.org/abs/2409.16526v1
Date: Wed, 25 Sep 2024 00:37:40 GMT
Title: APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls
Authors: Weiheng Bai, Keyang Xuan, Pengxiang Huang, Qiushi Wu, Jianing Wen, Jingjing Wu, Kangjie Lu,
Abstract summary: APILOT maintains a realtime, quickly updatable dataset of outdated APIs. It uses an augmented generation method to navigate LLMs in generating secure, version-aware code. It can reduce outdated code recommendations by 89.42% on average with limited performance overhead.
Score: 15.865915079829943
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: With the rapid development of large language models (LLMs), their applications have expanded into diverse fields, such as code assistance. However, the substantial size of LLMs makes their training highly resource- and time-intensive, rendering frequent retraining or updates impractical. Consequently, time-sensitive data can become outdated, potentially misleading LLMs in time-aware tasks. For example, new vulnerabilities are discovered in various programs every day. Without updating their knowledge, LLMs may inadvertently generate code that includes these newly discovered vulnerabilities. Current strategies, such as prompt engineering and fine-tuning, do not effectively address this issue. To address this issue, we propose solution, named APILOT, which maintains a realtime, quickly updatable dataset of outdated APIs. Additionally, APILOT utilizes an augmented generation method that leverages this dataset to navigate LLMs in generating secure, version-aware code. We conducted a comprehensive evaluation to measure the effectiveness of APILOT in reducing the incidence of outdated API recommendations across seven different state-of-the-art LLMs. The evaluation results indicate that APILOT can reduce outdated code recommendations by 89.42% on average with limited performance overhead. Interestingly, while enhancing security, APILOT also improves the usability of the code generated by LLMs, showing an average increase of 27.54% in usability. This underscores APILOT's dual capability to enhance both the safety and practical utility of code suggestions in contemporary software development environments.