From Task to Intentionality Automation: Mitigating the Open-Loop and Metacognitive Gaps in Agentic AI Systems

Abstract

Artificial Intelligence (AI) enables powerful capabilities that are transforming almost all sectors. However, the economic growth driven by AI comes at a cost, and its sociotechnical impacts are fraught with contradictions and paradoxes. As a result, several legal initiatives and risk management frameworks have been introduced to mitigate the various risks associated with AI systems. Agentic AI systems require even closer attention than traditional AI. While traditional AI has a narrow focus and responds to direct commands, Agentic AI emerges from combining multiple types of AI capable of planning, tool use, and multi-step execution. These systems can behave and interact autonomously, making decisions and performing tasks to achieve system objectives with minimal human oversight. Recognizing that Agentic AI represents a paradigm shift, this paper addresses its challenges from a Human-AI Interaction perspective. It examines the root causes and impacts of risks arising from the transition from Task Automation to Intentionality Automation, where the user manages outcomes and constraints rather than individual task steps. Key issues include the Open-Loop Control Gap and the Metacognitive Gap, whose relationship is fundamental to understanding the collapse of human oversight, as they represent two sides of the same coin in the loss of control. By analysing scenarios such as cybersecurity and healthcare, this paper identifies dimensions of user demand and identifies Ecological Interface Design as an ergonomic approach to ensure that as AI gains agency, the human retains authority and situational awareness.