ISCA: A Structured Framework for Assessing and Reporting of Interactive System Complexity

Abstract

Interactive systems are increasingly studied across research domains due to their growing importance in real-world settings, yet their capabilities are often described only at a high level, hindering systematic comparison and assessment across contexts. Existing evaluation approaches are based on subjective assessments or are too vague or broad to enable meaningful comparison. We present a structured complexity framework that provides reproducible complexity assessments for hardware and software. The questionnaire awards up to 50 points: 25 for hardware (degrees of freedom, computing power, sensor diversity, hardware safety, and interfaces) and 25 for software (decision autonomy, adaptability to user behavior, modalities, accessibility and usability, as well as personalization capabilities). It enables consistent classification of system capabilities and transparent comparison between different platforms. We conducted a survey of interactive systems used in the HRI community, collecting 28 system assessments across 25 distinct platforms. The results provide initial evidence for the questionnaire's reliability, content and construct validity, as well as its usability and coherence, based on participant-reported system assessments, usability feedback, and analysis of the collected data.