Trustworthy Human-Machine Systems: Formal Methods for Safety-by-Design : Plenary talk

Abstract

Society repeatedly faces complex challenges where robotic solutions are deployed. In many of these applications, humans and machines work together to accomplish shared tasks – for example, robots that assist patients or support human operators in their work. However, ensuring safety and trustworthiness in such systems cannot rely on learning-based models alone. These models often behave as non-verifiable "black boxes," which makes it difficult to provide rigorous guarantees. As a result, real-world deployment of human robot systems is frequently delayed or restricted: Autonomous driving is a prominent example. In our research group at KIT, we investigate how to develop verifiable mathematical models for human machine systems. Such models can formally describe system behavior and enable safety-by-design, where safety and performance guarantees are built into the system from the beginning. The talk will also show how these mathematical models can be integrated with data-driven learning methods, combining formal guarantees with the flexibility of modern machine learning. Finally, application scenarios will be presented in which the safety of human machine systems is considered essential, since societal acceptance and trust cannot be expected without credible and convincing safety guarantes.