Eye-Movement-Assisted Time–Frequency EEG Decoding for Multimodal Robotic Arm Control

Abstract

Brain–computer interface (BCI) technology has shown potential for future rehabilitation-related and assistive control applications. Nevertheless, single-modality electroencephalography-based motor imagery (EEG-MI) signals are susceptible to interference, whereas existing algorithmic models suffer from limited classification accuracy and insufficient actionable control commands for interactive devices, thereby impeding their practical deployment. To tackle these limitations, this study presents a multimodal human–computer interaction control scheme that integrates eye-movement command encoding with EEG motor imagery decoding. Self-collected EEG-MI and eye-movement datasets were established to support the proposed multimodal control framework. In this framework, eye movements are not used merely as auxiliary inputs, but are encoded as discrete commands for start, stop, grasp, and release, thereby reducing the command burden of EEG-MI decoding. The EEG-TransNet model is enhanced by integrating a time–frequency feature branch and replacing the original convolutional encoder with an adaptive multi-branch EEG feature gating module, strengthening the representation and fusion of multi-domain features. The model yields average classification accuracies of 86.96% and 88.73% on the BCI IV-2a dataset and the self-collected EEG dataset, respectively. Four independent SVM binary classifiers are adopted to identify four eye movement patterns. The EEG and eye movement classification results are binary-encoded to generate hardware-compatible control commands. Robotic-arm grasping experiments with healthy trained participants showed an average task completion time of 17 s, and the repeated grasping success-rate results further provide preliminary evidence for the real-time feasibility of the multimodal control framework under controlled laboratory conditions.