TPT: a compact CNN-transformer encoder for efficient microbial small protein modeling
TL;DR
TPT provides a compact and computationally efficient encoder with competitive performance for microbial peptide analysis and suggests that effective smORF representation may benefit from pretraining objectives and inductive biases tailored to short, rapidly evolving sequences, rather than from model scale alone.
Abstract
Introduction Microbial small proteins, encoded by small open reading frames (smORFs), play essential roles in antimicrobial activity, metabolic regulation, and signaling pathways. However, their short length and rapid evolutionary rate present significant challenges for computational modeling. Methods We introduce TinyProteinTransformer (TPT), a lightweight CNN-Transformer hybrid encoder pretrained on the Global Microbial smORF Catalog (GMSC; >280 million smORF families). TPT integrates multi-scale convolutional filters to capture local sequence motifs with Transformer layers for broader contextual modeling, and is trained jointly with masked language modeling and contrastive learning to capture both residue-level and sequence-level representations. We evaluated TPT on six downstream peptide/protein classification benchmarks spanning antimicrobial peptides (AMP), toxic peptides (TOX), bacteriocins (BCN), anti-CRISPR proteins (Acr), quorum-sensing peptides (QSP), and cell-penetrating peptides (CPP) using frozen-encoder linear probing. Results On the AMP and TOX tasks, TPT (103 M parameters) matched ESM2-150 M in predictive performance (AUC 0.930 vs. 0.928 for AMP; 0.930 vs. 0.925 for TOX) while achieving 4.3-fold faster inference. Across the evaluated benchmarks, TPT showed competitive performance compared with larger protein language models. Ablation experiments identified the contrastive objective as an important contributor to representation quality: its removal reduced AUC across all six downstream tasks and produced performance patterns consistent with MLM-only pretraining. Conclusion These results suggest that, within the evaluated benchmarks, effective smORF representation may benefit from pretraining objectives and inductive biases tailored to short, rapidly evolving sequences, rather than from model scale alone. TPT therefore provides a compact and computationally efficient encoder with competitive performance for microbial peptide analysis.