MyoSem: Aligning Electromyography with Natural-Language Action Semantics

Abstract

Electromyography (EMG) records muscle activation for hand gesture recognition, prosthetic control, and wearable interaction. Existing EMG methods usually formulate hand action understanding as prediction over fixed gesture labels, leaving EMG signals and action-semantic text unaligned in a common feature space and limiting bidirectional querying between signals and hand-action descriptions. We present MyoSem, an EMG--action semantic alignment framework that maps hand EMG and multi-view natural-language action descriptions into a shared feature space for retrieval. MyoSem builds semantic prototypes from multi-view descriptions, encodes EMG with local activation-aware representations, and uses semantic queries to aggregate local EMG evidence for bidirectional EMG--text retrieval. We evaluate MyoSem under seven protocols on NinaPro DB2, EMG2Pose, and NinaPro DB2$\rightarrow$DB3, covering random splits, user shifts, held-out action classes, and able-bodied-to-amputee transfer. Across this evaluation, MyoSem achieves the strongest overall bidirectional retrieval performance among the evaluated methods. Ablations and representation analyses show how semantic prototypes, activation-aware encoding, and semantic queries shape the learned space. Overall, MyoSem moves EMG hand action understanding from fixed-label recognition toward language-queryable semantic retrieval.