NeuroVisionArm: Gaze-EEG Hybrid Control for Assistive Robotic Prosthetics

Abstract

Supporting fine motor interaction with upper-limb prostheses demands an interface intended to be intuitive, low-latency, and robust to user and signal variability. This challenge is particularly acute when EMG signals are unreliable, as in conditions such as ALS or high cervical SCI, where EEG-based BCIs offer a potential but challenging alternative due to low SNR and signal drift; gaze-only interfaces, in turn, suffer from the “Midas Touch” problem of unintended activations. Motivated by these target populations, we present NeuroVisionArm, a hybrid gaze–EEG system that couples fixation-gated target selection with lightweight MI–EEG command-state decoding mapped to Pick, Drop, and Idle states for semi-autonomous reaching and grasping in a controlled benchtop setting. In contrast to prior gaze–BCI systems, NeuroVisionArm introduces: 1) a fixation-gated late-fusion policy that explicitly time-aligns stable gaze fixations with MI windows using hysteresis and median filtering to suppress false triggers without long dwells; 2) a calibrated image-to-robot mapping that upgrades 2D gaze into metric 3D poses using MiDaS monocular depth and ArUco-based scale anchoring, while exposing pose uncertainty to the planner; and 3) an embedded EEG pipeline that combines CNN/LSTM/Random Forest models into an ensemble selected on a latency–accuracy Pareto frontier. We integrate these layers in a reproducible ROS/MoveIt stack and use complementary validation streams to separate physical task execution from decoder and fusion characterization. In 36 physical benchtop pick–place episodes, the system achieves a mean gaze localization error of $1.2 \pm 0.4$ cm, a planning success rate of 94%, and an overall grasp success rate of 87%; to keep EEG decoder validation independent of arm-task execution outcomes, the in-house MI–EEG model is reported as a separately controlled offline window-level metric of 92.1%. Additional validation includes external public MI-EEG benchmarking, public ERD/ERS physiology checks, RealSense RGB-D and ChArUco-anchored gaze-depth validation, and simulated fusion-policy sensitivity analysis using real gaze timing with EEG decisions parameterized from public-dataset operating characteristics. While the present study is limited to short sessions with non-clinical participants in a controlled environment, NeuroVisionArm demonstrates the feasibility of a reproducible hybrid architecture for prosthetic command and target selection when EMG signals are compromised, serving as an engineering foundation for future longitudinal and clinical evaluation.