Nov 2026· Proceedings of the 19th ACM Workshop on Artificial Intelligence and Security· 0 citations· 8 references
Adversarial Robustness in Machine Learning
Abstract
Large language model (LLM) agents are no longer limited to text generators, but they can also plan, call tools, delegate tasks to other agents, act within external environments, and change states in response to their actions. In Anthropic’s recent report on an AI-orchestrated cyber-espionage campaign involving Claude Code, individual subtasks appeared legitimate when evaluated in isolation but became malicious when combined into an autonomous attack sequence. Consequently, the security of such complex LLM-based agentic systems may fail even when each component appears secure in isolation. This position and open-problem paper argues that LLM agent security should be studied as emergence-driven security: the analysis of security properties that arise from the interactions among agents and its components over time. We define emergence-driven security and provide a conceptual taxonomy to classify emergent security-related phenomena in LLM agents, covering both beneficial and harmful forms of emergence. Based on these insights, we propose EASE, an Emergence-Aware Security Evaluation framework that provides a systematic process for assessing how interactions among agents combine over time to produce harmful or beneficial system-level security outcomes. Finally, we conclude by outlining open problems for studying emergent behavior in LLM agents in the context of security. The paper is intended to provide a foundation for examining how the emerging capabilities and operational needs of LLM agents may shape security evaluations that go beyond static, model-level testing.
The results are packaged in the Greenfield Startup Model (GSM), which explains the priority of startups to release the product as quickly as possible, and the need to shorten time-to-market, by speeding up the development through low-precision engineering activities.
Carmine Giardino, Nicolò Paternoster, M. Unterkalmsteiner et al.· IEEE Transactions on Softwar...· 178 citations· ⚡14
Software startup companies develop innovative, software-intensive products within limited timeframes and with few resources, searching for sustainable and scalable business models.
M. Unterkalmsteiner, P. Abrahamsson, Xiaofeng Wang et al.· e-Informatica Software Engin...· 157 citations· ⚡17
This study conducts a case survey study based on the secondary data of the major pivots happened in 49 software startups, and demonstrates that customer need pivot is the most common among all pivot types.
Sohaib Shahid Bajwa, Xiaofeng Wang, Anh Nguyen-Duc et al.· Empirical Software Engineeri...· 127 citations· ⚡15
The comparison of adopter and non-adopter sample reveals three potential adoption inhibitor, security, data privacy, and portability, which underlines the importance of the technical and security perspectives for research investigating the adoption of technology.
Nattakarn Phaphoom, Xiaofeng Wang, S. Samuel et al.· Journal of Systems and Softw...· 111 citations· ⚡8
The ongoing work building a Raspberry Pi cluster consisting of 300 nodes is presented, with potential use cases being an inexpensive and green test bed for cloud computing research and a robust and mobile data center for operating in adverse environments.
P. Abrahamsson, S. Helmer, Nattakarn Phaphoom et al.· IEEE International Conferenc...· 110 citations· ⚡7
Objective There is dearth of information on the timeliness of antenatal care (ANC) uptake. This study aimed to determine the timely ANC uptake by a medically trained provider (MTP) as per the World Health Organization (WHO) recommendations and the country guideline. Methods Cross-sectional survey was done with 2,731 wo...
B. Sarker, Musfikur Rahman, T. Rahman et al.· PLoS ONE· 89 citations· ⚡7
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Enforcing physics-based constraints prevents AI models from generating unreliable predictions for complex nonlinear systems in weather, climate, and turbulent flows.
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.