SecureBreak: A Dataset towards Safe and Secure Models

The 2nd SeT-LLM Workshop on Secure and Trustworthy Large Language Models

Large language models (LLMs) are increasingly embedded as core components of data-centric systems, supporting analytical decision making, and automated reasoning over large-scale, heterogeneous datasets. Yet their deployment in open-world environments raises fundamental challenges to security and trustworthiness: LLMs can leak sensitive data, fall prey to prompt injection and jailbreaks, generate misinformation, and behave unpredictably under adversarial inputs, failures that propagate through data pipelines and affect downstream decisions. The rise of LLM-based agents further amplifies these risks through unsafe tool use and autonomous decision-making. The 2nd SeT-LLM Workshop on Secure and Trustworthy Large Language Models brings together researchers and practitioners from data mining, machine learning, security, and responsible AI to address these issues from a data-centric, system-level perspective, spanning robust defenses, trustworthy evaluation, privacy and copyright protection, robustness, alignment and safety, agent security, and high-stakes applications. Through invited talks, contributed papers, a poster session, and a panel discussion, the workshop prioritizes early-stage ideas, system experiences, and open problems across the lifecycle of LLM-based systems.

Generating Attacks for LLMs with GFlowNets

This study proposes an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another, and introduces a model capable of generating attack inputs in the Turkish language.

Generating Attacks for LLM with GFlowNets

The rapid advancement of Large Language Models (LLMs) has facilitated their ubiquitous integration into various domains, leading to widespread adoption. However, this escalating trend has introduced significant security vulnerabilities, necessitating the identification and mitigation of flaws arising from malicious exploitation. Red teaming assessments, conducted to evaluate model robustness through diverse adversarial inputs, are essential for exposing security risks and implementing countermeasures. Currently, red teaming is performed either manually by experts or automatically using predefined attack datasets. Nevertheless, manual testing remains time-consuming, while existing automated methods suffer from limited creativity due to their inherent dependency on fixed datasets. In this study, we propose an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.

DT-GenShield: A Digital Twin-Driven Runtime Security Architecture for Protecting Large Language Models Against Indirect Prompt Injection

DT-GenShield, a Digital Twin-driven runtime security architecture that integrates semantic threat detection, operational state representation, policy-guided mediation, and runtime logging to protect LLM-based systems before model inference, is proposed.

Large Language Model Vulnerabilities

Multi-SALLM: a multilingual security assessment of generated code

As Large Language Models (LLMs) become increasingly integrated into software engineers’ daily workflows, it is critical to ensure the code they generate is not just functionally correct but also secure. While LLMs can boost developer productivity, prior empirical studies have shown that they often produce insecure code. This issue stems from two key factors. First, the datasets commonly used to evaluate LLMs don’t accurately reflect real-world software engineering tasks where security is a concern. Instead, they tend to focus on competitive programming problems or classroom-style exercises, which lack the complexity and security risks of production code integrated into larger systems. Second, current evaluation metrics mostly emphasize functional correctness and overlook security aspects altogether. To address these gaps, we introduce Multi-SALLM, a benchmarking framework designed to systematically evaluate LLMs’ ability to generate secure code. The framework includes three main components: (1) a novel dataset of security-focused Python, Java, and C++ prompts translated into 23 natural languages, (2) automated assessment techniques for analyzing generated code, and (3) new metrics that assess models from the perspective of secure code generation. Our empirical evaluation of four state-of-the-art LLMs (StarCoder2, Qwen2.5-Coder, GPT-4o-Mini, Gemini-2.5-Flash) reveals three key findings. First, functional correctness and security are closely related but not equivalent. GPT-4o-Mini achieves the highest pass@k and also exhibits high vulnerable@k, largely because it produces more compilable and analyzable outputs; in contrast, models that appear safer often do so due to lower functional yield rather than consistently secure generation. Second, programming language has a stronger impact than natural language: performance is broadly stable across the 23 natural languages and does not alter relative model rankings, whereas the target programming language introduces substantial variation, with Java consistently lagging behind Python and C++. Third, sampling strategy is a critical risk factor: increasing temperature and k increase the likelihood of obtaining a correct solution but also increase vulnerable@k and sharply reduce security@k, indicating that broader exploration systematically surfaces more insecure variants.