Real-Time Digital Twin Development and Augmented Environment Generation for Multi-Agent Robotic Systems

Abstract

This study focuses on the design and development of a real-time digital twin of a multi-agent robotic system utilizing the NVIDIA Omniverse platform. Multi-agent robotic systems and their Digital Twin systems often struggle to make effective, collaborative decision-making in a dynamic environment. These limitations are mainly rooted in limited sensory input, data inaccessibility from occluded regions, and resource constraints like available numbers and intelligence of robots or robotic agents. Our proposed approach addresses these challenges by establishing collaborative data-sharing networks among multiple robotic agents, enabling enhanced situational awareness and interaction. To this end, with its enhanced data-sharing networks, our system first constructs a high-fidelity virtual environment using multimodal input data from various sensors, such as LiDAR and cameras. A unified virtual map is created by fusing individual sensory data, accurately representing the environment. Second, our approach is powered by generative Artificial Intelligence (GenAI) models. Third, it further enhances the fidelity of the virtual environment by generating the most reliable data for hidden or unobserved areas using its pattern recognition capabilities on available data for visible areas. Fourth, our approach optimizes resource utilization to find a robust balance between system performance and resource availability. Fifth, our digital twin system will leverage the privacy-preserving framework of distributed data networks to improve security measures and maintain data integrity to be extended to real-world industry problems. As a result, the advantages of the proposed approach are expected to support efficient decision-making and navigation for multiple robotic agents in a complex and dynamically changing environment.