Contrastive language-music pretraining for cross-modal music retrieval

Abstract

Cross-modal music retrieval allows searching music by natural language queries, thereby resolving the challenge present in conventional tag-based music retrieval systems. This paper presents a contrastive pre-training framework for learning aligned embeddings between textural description and music audio signals. The proposed framework consists of two main components: (1) a music-aware text encoder with an expanded vocabulary using specific musical terms and fine-tuned on music description datasets, and (2) a hierarchical audio encoder with multi-scale temporal patterns by aggregating features across transformer layers. The framework utilizes a symmetric contrastive objective with data augmentation techniques for efficient cross-modal alignment. A two-stage training process is employed with initial training of projection layers with frozen encoders for stable mapping learning and joint fine-tuning of all parameters. Experimental results on the MagnaTagATune and GTZAN datasets show that the proposed approach is competitive on both crossmodal retrieval and zero-shot classification tasks. For text-to-music retrieval, the approach reaches a recall at 10 (R@10) value of 58.6% on the MagnaTagATune dataset, outperforming other models. For zero-shot classification on GTZAN, the approach reaches 71.2% accuracy with zero training on specific classification tasks. Ablation studies verify the effectiveness of both components with a music-aware text encoder showing the largest gain. The approach can be applied for real-world usage such as natural language music searching as well as automatic music tagging for online music streaming services as well as online music repositories.