Self-Architecting Protein Transformers: An Empirical Study

Abstract

Motivation Protein language models (pLMs) such as ESM-2 and ProtBERT rely on pretraining corpora of tens to hundreds of millions of sequences and on encoder architectures whose depth, width and number of attention heads are chosen by the practitioner and never revisited during training. The entry cost of state-of-the-art pLMs is therefore out of reach for laboratories without industrial-scale infrastructure, and the fixed architecture provides no in-training diagnostic of whether the chosen capacity matches the structural complexity of the data. This work asks whether a self-architecting transformer, which grows its own width and depth from quantitative signals derived from the attention matrices, can extract competitive protein representations from a single reference proteome. Results A three-level self-architecting framework, INCRT-geo, is applied to masked-language pretraining on the human Ensembl proteome (approximately twenty thousand sequences). On Pfam-50 family classification, the principal model attains a linear-probe accuracy that exceeds two pretrained baselines, ESM-2 small and ProtBERT, despite a corpus several orders of magnitude smaller. Three single-variable ablations isolate the contributions of one-residue tokenisation, depth growth and an asymmetry-loss regulariser; the regulariser is shown to be necessary for the depth-growth trigger to fire. Scaling pretraining to eight vertebrate proteomes does not improve Pfam accuracy under the available compute budget; the negative result is reported transparently. Architectural diagnostics indicate that the heads allocated by the framework are functionally diverse rather than redundant. Availability Code, notebooks and pretrained checkpoints are released under an open-source license; details in Data Availability.