Skip to content
Conference Open access

EyeGest: A Webcam-Only Multimodal Eye-and-Gesture Framework for Zero-UI Assistive Device Control, with Subject-Independent Evaluation

Sep 2026 · 2026 7th International Conference on Smart Electronics and Communication (ICOSEC) · pp. 1423-1428 · 0 citations · 17 references

Abstract

People with severe motor impairment need to operate computers and smart-home devices without a keyboard, mouse, or touchscreen. We present EyeGest, a webcam-only assistive framework that recognizes eight control commands from two complementary channels: eye landmarks (gaze direction, blink through the eye-aspect-ratio, and pupil offset) and hand landmarks. It dispatches these commands to a browser and to IoT devices without any special or medical hardware. The system is evaluated under a subject-independent protocol that never allows a test user's data to appear in training. Under this protocol the fused model reaches 81.7% accuracy (F1 0.82), compared with 86.7% when subjects are allowed to leak across the split, which quantifies the optimistic bias that inflates many webcam-HCI accuracy claims. A modality ablation shows that fusion is essential: eyeonly and gesture-only models reach only 42% and 57%, because some commands separate only by gaze and others only by gesture, and only the fused model resolves both. A confidence-based rejection rule reduces the false-trigger rate on no-command (resting) inputs to 6.6% at 99.7% precision, which is important when an accidental command has a real cost. Classification runs in about 0.03 ms, so end-to-end latency is bounded by the camera rather than the model.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.