Deep learning-based speller system for decoding user intentions from pupillary light reflex signals using time-division flickering visual stimuli

Abstract

Despite extensive studies on brain–computer interface (BCI)-based speller systems for individuals with motor disabilities as communication aids and, more recently, for healthy users as hands-free input devices, these systems continue to face practical limitations, including calibration and training requirements, high costs, and limited portability. This study proposes a keyboard-type 40-class pupillary light reflex (PLR)-based speller system to interpret user intentions through changes in pupil size induced by brightness variations in visual stimuli. Twelve healthy participants engaged in the experiments using time division-based flickering visual stimuli with different inter-stimulus intervals (ISIs). We then applied a dynamic visibility-based feature extraction strategy and a temporal-spatial hybrid deep learning architecture to decode user intentions. The performance of the proposed PLR-based speller system was evaluated across three ISI conditions (0.2, 0.3, and 0.4 s) using nested cross-validation, leave-one-subject-out cross-validation (LOSO-CV), and real-time online evaluation. Across all three evaluation approaches, the 0.3-s ISI condition consistently yielded the best overall performance, achieving an accuracy of 94.14 ± 1.15% and an ITR of 67.32 ± 0.66 bits/min in nested cross-validation, 90.58 ± 1.36% and 62.80 ± 1.73 bits/min in LOSO-CV, and 93.06 ± 0.99% and 55.12 ± 1.04 bits/min in the real-time online evaluation. The proposed PLR-based speller system outperforms previous PLR-based interface systems. Although a performance gap remains between the proposed approach and state-of-the-art BCI-based speller systems, it provides intuitive training-free interaction without the need for user-specific gaze-to-display calibration. It also enables subject-independent decoding through a non-contact pupil interface using a low-cost sensor.