Vibe Coding and Web Application Security: A Twin-Prompt Study

Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.

The Anatomy of a Prompt Injection: A Component Model for Structured Analysis

This paper formalizes the structure of prompt-injection artifacts, enabling defenders, red teamers, and cyber threat intelligence (CTI) teams to label, compare, and mutate attacks without relying on fragile string matching.

Multi-SALLM: a multilingual security assessment of generated code

As Large Language Models (LLMs) become increasingly integrated into software engineers’ daily workflows, it is critical to ensure the code they generate is not just functionally correct but also secure. While LLMs can boost developer productivity, prior empirical studies have shown that they often produce insecure code. This issue stems from two key factors. First, the datasets commonly used to evaluate LLMs don’t accurately reflect real-world software engineering tasks where security is a concern. Instead, they tend to focus on competitive programming problems or classroom-style exercises, which lack the complexity and security risks of production code integrated into larger systems. Second, current evaluation metrics mostly emphasize functional correctness and overlook security aspects altogether. To address these gaps, we introduce Multi-SALLM, a benchmarking framework designed to systematically evaluate LLMs’ ability to generate secure code. The framework includes three main components: (1) a novel dataset of security-focused Python, Java, and C++ prompts translated into 23 natural languages, (2) automated assessment techniques for analyzing generated code, and (3) new metrics that assess models from the perspective of secure code generation. Our empirical evaluation of four state-of-the-art LLMs (StarCoder2, Qwen2.5-Coder, GPT-4o-Mini, Gemini-2.5-Flash) reveals three key findings. First, functional correctness and security are closely related but not equivalent. GPT-4o-Mini achieves the highest pass@k and also exhibits high vulnerable@k, largely because it produces more compilable and analyzable outputs; in contrast, models that appear safer often do so due to lower functional yield rather than consistently secure generation. Second, programming language has a stronger impact than natural language: performance is broadly stable across the 23 natural languages and does not alter relative model rankings, whereas the target programming language introduces substantial variation, with Java consistently lagging behind Python and C++. Third, sampling strategy is a critical risk factor: increasing temperature and k increase the likelihood of obtaining a correct solution but also increase vulnerable@k and sharply reduce security@k, indicating that broader exploration systematically surfaces more insecure variants.

Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code

Large Language Models (LLMs) increasingly generate code from natural-language prompts, making prompt engineering a key mechanism for shaping the security of generated software. Structured and security-oriented prompts are widely used to encourage safer code, yet their effects extend beyond whether detected weaknesses are simply present or absent. Using 424 security-sensitive Python tasks, we generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants that progressively add structural and security guidance, and evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE distributions. Structured prompting substantially reduces refusals (e.g., GPT-4o invalid outputs drop from 338 of 424 to 37-52), enabling large-scale analysis, but security-oriented refinements do not consistently reduce overall weakness prevalence. For GPT-4o, stronger prompts primarily redistribute risk: high-severity findings fall (20.8% to 13.6%) while low-severity findings rise (32% to 43.5%); LLaMA shows weaker, less consistent shifts. We also observe security-driven semantic drift, where stricter prompts silently remove or rewrite explicitly requested unsafe constructs. Overall, prompt structure improves compliance but is an unreliable substitute for robust security controls in LLM-assisted development.

Distilled documentation for dialect-specific SQL generation

Large Language Model Vulnerabilities