HSSynergy: scale-aware hierarchical attention for interpretable drug synergy prediction

Abstract

Combination therapy offers a promising strategy for cancer treatment by reducing toxicity and overcoming drug resistance. However, existing substructure-based prediction methods may struggle to effectively capture the scale-specific features of substructure interactions and often overlook cell-line-specific substructure selection, which can limit mechanistic interpretability. To address this issue, we propose HSSynergy, a hierarchical substructure-aware deep learning framework for predicting anticancer drug synergy. It first employs Graph Attention-Convolution Fusion Module to adaptively extract multi-scale substructure features from molecular graphs. Rather than indiscriminately mixing features, it introduces a Scale-Aware Masked Attention mechanism that enforces precise layer-wise alignment, and utilizes hierarchical grouping with mask constraints to achieves same-scale focusing while shielding against cross-scale noise. Furthermore, shifting away from passive cell line representations, a Cell-Active Cross-Attention mechanism models the active selection of specific substructures by heterogeneous cancer cells, capturing precise drug-cell contexts. Rigorous evaluations on two benchmark datasets show that HSSynergy achieves superior performance compared to state-of-the-art methods with robust generalization to unseen drugs and cell lines. Beyond predictive metrics, it provides mechanistic interpretability insights, revealing the hierarchical emergence of substructures and accurately pinpointing literature-validated functional groups driving synergy in specific drug combinations. Notably, several novel synergistic combinations predicted by HSSynergy are supported by existing literature and clinical evidence, suggesting its potential utility in aiding anticancer drug discovery. HSSynergy (i) Scale-Aware Masked Attention restricts substructure interactions within scale groups to reduce cross-scale noise. (ii) Cell-Active Cross-Attention models dynamic, cell-specific substructure selection instead of static cell-line fusion. (iii)Hierarchical attention links synergy predictions to pharmacologically functional groups.