Gesture Controlled MIDI Interface for Real Time Audio Effects Using Embedded Systems

Abstract

This paper presents a novel two tier hybrid hand-tracking architecture for a real time gesture controlled Musical Instrument Digital Interface (MIDI) system implemented entirely on a Raspberry Pi 4B. Unlike conventional systems that rely continuously on computationally expensive landmark detection, the proposed approach combines per-frame hue-saturation-value (HSV) colour segmentation, spatial-continuity region-of-interest (ROI) tracking, and asynchronously executed MediaPipe landmark estimation. Approximately 85% of frames are resolved through the lightweight ROI-tracking path, while landmark estimation runs in a background thread at one-fifth of the frame rate. The system maps hand position, finger pinch, wrist tilt, and open or closed-hand gestures to MIDI controls. Evaluated over 3000 frames, it achieves 5.43 ms median latency, 9.09 ms p99 latency, and approximately 169 FPS, keeping virtually all frames within the 10 ms real-time bound. The key novelty is its adaptive fusion of lightweight tracking and selective landmark estimation to achieve low-latency gesture control on resource-constrained edge hardware without host-side processing.