Real-Time Terrain Recognition for Knee Exoskeletons via Adaptive Vision-IMU Fusion

Abstract

Traditional terrain recognition for lower-limb exoskeletons relies on inertial measurement units (IMUs). These sensors respond only after the user has physically engaged with new terrain, causing inherent assistive torque lag. Visual perception can anticipate terrain ahead, but suffers from lighting sensitivity and prohibitive computational cost on wearable embedded platforms. This paper proposes a lightweight, real-time terrain recognition method that fuses vision and IMU signals through an adaptive mechanism deployed on an ESP32-S3 microcontroller. The visual branch uses a structurally pruned and INT8-quantized YOLOv11n model; the IMU branch employs a CNN-GRU temporal network. A Monte Carlo dropout-based uncertainty estimation dynamically adjusts fusion weights, complemented by Jensen-Shannon divergence conflict detection and a three-level faulttolerance scheme. On a multi-modal dataset with five terrain classes, the system achieves 98.1% accuracy, a model size of 0.9 MB, and end-to-end latency of 101.5ms (78ms visual inference with ESP-NN acceleration). The method offers a practical edge-AI solution for real-time terrain recognition on resource-constrained wearable devices and provides a basis for proactive gait planning in knee exoskeletons.