A hybrid BiLSTM transformer model for drug synergy prediction

Abstract

Accurately predicting synergistic drug combinations remains a critical challenge in computational pharmacology, with important implications for the development of combination therapies in oncology. However, in vitro screening of drug combinations for synergy is both time-consuming and labor-intensive, as the number of possible combinations grows exponentially. Although several computational methods have been proposed to predict synergistic drug pairs, more effective modeling of the complex multidimensional relationships among drug compounds is still needed. In this study, we present BT-Synergy, a task-specific hybrid deep learning model for drug synergy prediction. The model integrates BiLSTM modules within Transformer blocks to capture complementary sequential and contextual patterns from SELFIES-based molecular representations. In this design, bidirectional LSTM layers complement self-attention with sequential inductive biases, enabling the model to capture local molecular patterns as well as long-range contextual dependencies. In parallel, protein features are extracted using pre-trained protein language models and aggregated to form biologically informed cell-line representations. These integrated embeddings, which combine drug, protein, and cell-line information, are then processed by a multi-layer perceptron to predict drug synergy. Experimental results on benchmark datasets show that BT-Synergy effectively predicts synergistic interactions, achieving an accuracy of 0.8458. These findings highlight the effectiveness of the proposed architecture in capturing complex biochemical interactions and improving predictive performance in drug synergy prediction.