Logo image
Back
Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions
Journal article   Open access   Peer reviewed

Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions

Amirali Sajadi, Binh Le, Anh Nguyen, Kostadin Damevski and Preetha Chatterjee
Empirical software engineering, v 30(3), 101
11 Apr 2025
DOI: https://doi.org/10.1007/s10664-025-10658-6
Featured in Collection :   Research Supported by Drexel Libraries' OA Programs
url
https://doi.org/10.1007/s10664-025-10658-6View
Published, Version of Record (VoR)Open Access via Drexel Libraries Read and Publish Program 2025CC BY V4.0,  Open

Abstract

Computer Science - Artificial Intelligence Computer Science - Computation and Language Computer Science - Learning Computer Science - Software Engineering Artificial Intelligence or Cybernetics
The widespread adoption of conversational LLMs for software development has raised new security concerns regarding the safety of LLM-generated content. Our motivational study outlines ChatGPT’s potential in volunteering context-specific information to the developers, promoting safe coding practices. Motivated by this finding, we conduct a study to evaluate the degree of security awareness exhibited by three prominent LLMs: Claude 3, GPT-4, and Llama 3. We prompt these LLMs with Stack Overflow questions that contain vulnerable code to evaluate whether they merely provide answers to the questions or if they also warn users about the insecure code, thereby demonstrating a degree of security awareness. Further, we assess whether LLM responses provide information about the causes, exploits, and the potential fixes of the vulnerability, to help raise users’ awareness. Our findings show that all three models struggle to accurately detect and warn users about vulnerabilities, achieving a detection rate of only 12.6% to 40% across our datasets. We also observe that the LLMs tend to identify certain types of vulnerabilities related to sensitive information exposure and improper input neutralization much more frequently than other types, such as those involving external control of file names or paths. Furthermore, when LLMs do issue security warnings, they often provide more information on the causes, exploits, and fixes of vulnerabilities compared to Stack Overflow responses. Finally, we provide an in-depth discussion on the implications of our findings, and demonstrated a CLI-based prompting tool that can be used to produce more secure LLM responses.

Metrics

12 Record Views
1 citations in Web of Science

Details

InCites Highlights

Data related to this publication, from InCites Benchmarking & Analytics tool:

Collaboration types
Domestic collaboration
Web of Science research areas
Computer Science, Software Engineering
Logo image