Using LLM Agents for Security Testing of Compiled Mobile Applications: A Case Study on a Real-World Compiled iOS App

Abstract

Security testing of compiled applications is crucial for organizations to assess the behavior and security of the software they deploy without relying on the vendor. In this paper, we present a case study that investigates the capability of autonomous Large Language Model (LLM) agents to detect security weaknesses and compliance risks in compiled code of a real-world iOS application. These application-layer vulnerabilities are often context-dependent: they arise from app logic patterns that violate security principles in ways that require a human-like understanding to recognize, making them resistant to detection via conventional static analysis methods. We design and perform an evaluation on a real-world iOS app with several confirmed weaknesses that we discovered and reported to the vendor. We derive two task types: (1) directed tasks, where the system is given context on a security issue and must locate the relevant code, and (2) reporting tasks, where the model receives an open-ended task to analyze the app for security issues. We evaluate 20 model configurations across single-agent and multi-agent system architectures using open-weight LLMs. Our evaluation shows that real-world, context‑dependent vulnerabilities and compliance risks, which are difficult to detect using conventional static analysis methods and usually rely on manual assessments, are reliably detectable using LLM agents operating on sufficiently reconstructed pseudo-code. Additionally, we observe that improved decompilation output is crucial to enable models to consistently solve the most complex tasks. We conclude that developers should use LLMs at the source-code level to counter adversaries applying the same techniques on decompiled code and research should advance decompilation support for the growing language diversity in the iOS ecosystem.