System Prompt Exfiltration via Chained LLM Vulnerabilities: A Case Study on a Production AI Cybersecurity Chatbot
Abstract
Large Language Model (LLM)-powered applications introduce a new class of security vulnerabilities that differ fundamentally from traditional web application flaws. This paper presents a case study of a six-stage attack chain against OSCAR, a production AI support chatbot operated by OffSec at portal.offsec.com. The attack chain demonstrates three independent system prompt exfiltration vectors: direct disclosure without any bypass, a format-injection technique consistent with a conflict between competing LLM training objectives, and a deterministic system role injection through an unauthenticated API endpoint. Findings are mapped to OWASP LLM Top 10 (2025), classified using CVSS v3.1 scoring [9], and analyzed through a three-layer root cause model. Concrete architectural remediations with code-level examples are provided. All findings were responsibly disclosed under OffSec's bug bounty program and have been patched. This research was accepted as a poster presentation at DEF CON AI Village 34.