Jul 2026· The Scholar Journal for Sciences & Technology· Vol 4· 0 citations
TL;DR
A prototype framework for an effective LLM API designed to mimic digital banking assistant responses and evaluate against synthesized and real-world banking dialogues is introduced and how the results dovetail with the growing regulatory landscape for AI in financial services is explored.
Abstract
Abstract
But now, given the AI revolution and increased interest in bringing virtual agents and assistants to life banks too are testing LLM-powered AI agents that may assist customers, explain and customize products as well as simplify operational work done by bank employees in the background. But similar systems are susceptible to prompt injection, insecure output handling, and other LLM-specific threats that had only become more prevalent since these publications. Existing surveys and frameworks survey the generic security space of LLMs but do not propose reach an end-to-end, banking-specific threat model nor deployable defense architecture for assistants in line with systems from core into the edge. We also present a data-privacy-aware threat model and a learnable multi-layer defense framework for LLM-based banking assistants. We showcase a reference architecture for an omnichannel banking chatbots built using an instruction-tuned, medium-scale commercial LLM and also access to tools for accessing customer data, payments and knowledge bases. Assets, trust boundaries, adversary capabilities, and a banking-related threat taxonomy based on OWASP LLM guidance in combination with the NIST AI Risk Management Framework and recent sector-wide reports are defined. We then present a defense-in-depth framework with which we employ: governing inputs, prompt level controls, policy enforcement in the orchestration layer, safety guards at the tool-layer, structured output filtering along with response-risk scoring and constant monitoring and governance. We introduce a prototype framework for an effective LLM API designed to mimic digital banking assistant responses and evaluate against synthesized and real-world banking dialogues. This analysis will comprise 48 attack models based on the use of SMS injection techniques, leaking data or other misuses or abuses of authenticated tools. The framework decreases the overall success rates of attacks from 68–79% to between 14–24% across three common styles of attack, while introducing an average response time overhead of less than 11.8%, a modest increase in token consumption and maintaining false alarm rates below 7%. We, too, explore how the results dovetail with the growing regulatory landscape for AI in financial services and how banks can choose to tailor the framework according to their risk appetite and legacy infrastructure constraints
Keywords:Artificial intelligence, machine learning, deep learning, natural language processing systems, data privacy, network security
Large language model (LLM) applications now combine hidden prompts, retrieval pipelines, memory stores, content filters, tool calls, delegated identities, and downstream automation. Security reviewers are increasingly asked to assess such systems without access to source code, model weights, prompt templates, vector-store configuration, or internal logs. This paper presents D-RELLM, a defensive reverse-engineering framework for black-box security assessment of deployed LLM applications. The framework maps observable application behavior to an attack-surface graph, executes bounded and non-destructive probes, converts evidence into reproducible risk scores, and links each finding to mitigation and re-test criteria. Unlike model-only jailbreak evaluation, D-RELLM treats the deployed application as a socio-technical system whose risk depends on instruction hierarchy, retrieval trust, authorization, tool agency, output handling, monitoring, and operational controls. The paper defines a threat model, a probe taxonomy, a weighted scoring equation, a confidence equation, an application-level aggregation method, an evidence schema, and a mitigation playbook. A synthetic pilot across three representative archetypes—chatbot, retrieval-augmented generation (RAG) assistant, and tool-using agent—illustrates how direct prompt injection, indirect prompt injection, retrieval poisoning exposure, excessive agency, output-handling weaknesses, and sensitive-information disclosure can be prioritized before and after remediation. The result is a practical, auditable, and publication-ready method for defenders who need to quantify residual LLM application risk from black-box evidence while avoiding unsafe exploitation.
Bhavesh B. Prajapati, Bhavya Shah· International journal of com...· 0 citations
Chatbots powered by LLMs have become popular due to their scalability and instant reply feature. There can be many security issues associated with LLM-based chatbots, which emerge in the interaction between the LLM-based model and other software components. One major problem is that chatbots can be attacked using prompt injection, jailbreaks, and harmful queries. These attacks can change the chatbot behavior and may lead to wrong answers or leakage of sensitive information. This makes security very important in such systems. Existing methods improve the chatbot performance using techniques like modular design, Retrieval-Augmented Generation (RAG) and input filtering for security. However, most of these methods use a single model and do not have strong real-time protection, making them weak against advanced attacks. Hence, this work proposes a Dual-LLM architecture where security is handled as a separate component for protection against real time threats. A classification model is designed to check the user input queries using threat categories and a confidence score is evaluated to block the harmful inputs. Only safe queries are sent to the response model, which uses a controlled RAG system to generate accurate answers. The system is built as a step-by-step pipeline combining classification and response generation. The results show an accuracy of 90% in detecting threats which leads to reduction of unsafe responses.
G.Kavitha, Hishaam Ahamed M, Premnath R· 2026 4th International Confe...· 0 citations
Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them. Interpretability methods can expose model-internal signals along the generation path that could inform enforcement, but these signals are not security controls by themselves. Deployments that adapt them for safety typically couple each signal to its own calibration, policy logic, and intervention code, so each new artifact creates integration work instead of strengthening a shared defense. We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving. In LMSM, a selected security backend exposes calibrated evidence, a versioned policy evaluates active rules over trusted per-request context, and a separate gate authorizes buffered output release. This design separates mediation correctness from policy effectiveness, and it allows backend, rule, or schedule changes without rebuilding request handling or enforcement. Our prototype shows the separation working in practice: with Hugging Face Transformers and continuously batched vLLM, the same substrate hosts artifact-backed sparse autoencoder (SAE) and transcoder deployments and task-fitted dense probes, preserves request-specific decisions under scheduler churn, and selectively enforces and composes multiple rules per request. On Qwen3-4B, LMSM-Checkpoint reduces HarmBench attack success rate from 39.20% to 3.32%, with XSTest false refusals rising from 2.40% to 4.40%, while retaining 98.14% of the throughput of a matched serving path that performs no monitoring work at 32 active sequences. LMSM gives advances in interpretability and model-internal analysis a common path to runtime enforcement.
Xiucheng Zhang, Bonan Ruan, Junfeng Fang et al.· 0 citations
Large language models (LLMs) are increasingly embedded as core components of data-centric systems, supporting analytical decision making, and automated reasoning over large-scale, heterogeneous datasets. Yet their deployment in open-world environments raises fundamental challenges to security and trustworthiness: LLMs can leak sensitive data, fall prey to prompt injection and jailbreaks, generate misinformation, and behave unpredictably under adversarial inputs, failures that propagate through data pipelines and affect downstream decisions. The rise of LLM-based agents further amplifies these risks through unsafe tool use and autonomous decision-making. The 2nd SeT-LLM Workshop on Secure and Trustworthy Large Language Models brings together researchers and practitioners from data mining, machine learning, security, and responsible AI to address these issues from a data-centric, system-level perspective, spanning robust defenses, trustworthy evaluation, privacy and copyright protection, robustness, alignment and safety, agent security, and high-stakes applications. Through invited talks, contributed papers, a poster session, and a panel discussion, the workshop prioritizes early-stage ideas, system experiences, and open problems across the lifecycle of LLM-based systems.
Lu Lin, Jinghui Chen, Ting Wang et al.· Proceedings of the 32nd ACM...· 0 citations
Large Language Models (LLMs) are increasingly being deployed in critical domains such as healthcare, finance, and public infrastructure to support intelligent decision-making and conversational interactions. However, these systems introduce significant challenges related to security, reliability, and trustworthiness. Vulnerabilities such as adversarial prompt injections, behavioral manipulation, and multi-stage attacks can lead to unsafe outputs, privacy risks, and loss of user trust. There is a need for robust approaches that ensure both safe application-level interactions and adaptive system-level defenses against evolving LLM threats. In this thesis, we propose a unified two-layer approach to enhancing the trustworthiness and security of LLM-enabled systems. At the application layer, we develop EmpathAI, a RAG-based mental healthcare chatbot that incorporates source tagging, sentiment-aware context retrieval, and a two-layer defense mechanism using regex filtering and prompt engineering to mitigate prompt injection attacks. Building on this, at the system layer, we introduce the Adaptive LLM Threat Response (ALTR) framework, which integrates behavioral anomaly detection, context-aware prompt classification, and temporal threat memory to identify and mitigate adversarial interactions in real time. We evaluate both layers using conversational datasets and adversarial interaction traces. At the application layer, EmpathAI achieves high semantic alignment (similarity scores >0.80–0.85), with all prompt injection classes successfully mitigated. At the system layer, ALTR attains strong detection performance (accuracy and AUC of 0.961, false-negative rate of 0.9 percent) under low-latency constraints (<20 ms). Together, these results demonstrate that securing LLMs in critical domains requires both application-layer trust and systemlayer defense, and that combining domain-aware RAG systems with adaptive multi-layer security frameworks enables their trustworthy and reliable deployment in high-risk environments.
This paper presents OS-NLI, a privacy-respecting and modular natural-language interface for operating-system tasks. Unlike vendor-bound assistants, OS-NLI separates intent interpretation from command execution through an architecture comprising a web interface, a retrieval-augmented generation (RAG) layer, an API, an encrypted datastore, and a local execution worker. The LLM is treated as an untrusted suggestion engine: only pre-approved command templates with validated parameters may be queued, and execution requires explicit user confirmation. Sensitive data are protected with TLS in transit and AES-GCM at rest, while the threat model explicitly excludes hosts that are fully compromised at runtime. A prototype was implemented across Windows 11 and Fedora Linux. Evaluation on 60 natural-language queries shows strong task performance for application and utility identification, median answer latency of 1.4 s, and median post-confirmation command-execution latency of 285 ms. Preliminary malicious-input checks indicate that non-allowlisted outputs are rejected before execution. The current prototype still relies on vendor-hosted components for the UI host and LLM, so it is not yet end-to-end auditable; however, the architecture isolates these dependencies and provides a practical migration path to self-hosted deployment. These results support the feasibility of a safer, auditable design for naturallanguage OS interaction.
David Lawrence, Xiaoli Huan· International Conference on...· 0 citations