GenVarLoader: an accelerated dataloader for applying deep learning to personalized genomics
Deep learning models can predict the functional impact of genetic variants. However, training and applying these sequence models to personalized genome sequences are severely constrained by the computational burden of storing and reading massive datasets. We developed GenVarLoader, an accelerated dataloader that creates personalized genomic sequences and functional tracks on-the-fly to overcome these bottlenecks. GenVarLoader stores this personalized genomic data in formats specifically optimized for machine learning. Our tool reads variants up to 26,000 times faster than a pre-subset BCF, loads personalized data up to 130 times faster than tuned FASTA and pyBigWig pipelines, and reduces storage approximately 2,000-fold compared to existing alternatives. By eliminating severe data loading bottlenecks, GenVarLoader ensures efficient model training and inference. It provides a scalable solution for integrating biobank-scale personalized genomic data with deep learning applications.