Towards a Multimodal Video-Based Approach for Editable Piano Score Transcription

Abstract

Transcribing piano performances into editable notation remains difficult without dedicated symbolic instruments, controlled recording setups, or substantial manual correction. This paper presents an early prototype for transforming piano performance videos captured from a keyboard-oriented viewpoint into draft MusicXML scores by combining visual analysis of the keyboard with audio-informed temporal interpretation. Rather than treating transcription as a purely audio-based or vision-based task, the system exploits the multimodal structure of piano performance: visible key motion, hand activity, timing cues, and sound. We evaluate the prototype on three short musical excerpts recorded across different lighting conditions. The preliminary results suggest that video-based multimodal transcription can support the creation of correction-ready score drafts, while also revealing key challenges related to illumination, occlusion, rhythmic ambiguity, and incomplete keyboard coverage. Although the current implementation is limited, it offers a practical starting point for accessible video-to-score workflows and motivates future work on richer audio-visual fusion for robust piano transcription.