EyeGest: A Webcam-Only Multimodal Eye-and-Gesture Framework for Zero-UI Assistive Device Control, with Subject-Independent Evaluation
Abstract
People with severe motor impairment need to operate computers and smart-home devices without a keyboard, mouse, or touchscreen. We present EyeGest, a webcam-only assistive framework that recognizes eight control commands from two complementary channels: eye landmarks (gaze direction, blink through the eye-aspect-ratio, and pupil offset) and hand landmarks. It dispatches these commands to a browser and to IoT devices without any special or medical hardware. The system is evaluated under a subject-independent protocol that never allows a test user's data to appear in training. Under this protocol the fused model reaches 81.7% accuracy (F1 0.82), compared with 86.7% when subjects are allowed to leak across the split, which quantifies the optimistic bias that inflates many webcam-HCI accuracy claims. A modality ablation shows that fusion is essential: eyeonly and gesture-only models reach only 42% and 57%, because some commands separate only by gaze and others only by gesture, and only the fused model resolves both. A confidence-based rejection rule reduces the false-trigger rate on no-command (resting) inputs to 6.6% at 99.7% precision, which is important when an accidental command has a real cost. Classification runs in about 0.03 ms, so end-to-end latency is bounded by the camera rather than the model.