MetaGPT: An Immersive Mixed Reality AI Assistant via Offline Edge-Deployed Multi-Agent Vision-Language Inference
TL;DR
To coordinate agents without dense O(N2) messaging overhead, Evolutionary Topology Optimization (ETO), which searches offline over interagent connection graphs, confidence/divergence trigger thresholds, and fusion weights, producing a frozen sparse communication policy shipped at inference.
Abstract
MetaGPT is a fully offline, edge-deployed immersive mixed reality AI assistant, running entirely on a co-located edge device without any cloud dependency. Deployed on Meta Quest 3, it captures egocentric color-and-depth (RGB-D) observations and interaction cues—object selection and voice queries—and offloads vision-language model (VLM) inference to a co-located edge server, returning compact structured semantics rendered as view-consistent mixed-reality feedback. The core challenge is the edge memory budget: high-quality vision-language reasoning demands large models, yet extended reality (XR) edge hardware is tightly constrained in graphics processing unit (GPU) memory. Rather than deploy one large model, MetaGPT runs a pool of role-prompted agents that share a single compact FastVLM-0.5B backbone, occupying the video random access memory (VRAM) footprint of one small model while collectively reasoning at a higher level. To coordinate agents without dense O(N2) messaging overhead, we introduce Evolutionary Topology Optimization (ETO), which searches offline over interagent connection graphs, confidence/divergence trigger thresholds, and fusion weights, producing a frozen sparse communication policy shipped at inference. On Caltech-256, cooperating agents gain up to +4.7 top-1 accuracy points over a single-agent baseline; on MMStar, they raise the visual score from 30.9 to 40.9—matching or exceeding VLMs 4-6× larger—at near-identical peak GPU memory. All inference runs locally: no data leaves the device, no network round-trip to the cloud.
