Evaluating the effectiveness of embeddings from five Protein Language Models, including ProtBERT-BFD, ESM-2, ProtALBERT, ProLLaMA, and ProtGPT-2, as input features for various machine learning classifiers suggests that while current embeddings offer strong performance, further advancements in feature extraction and model architectures are needed to significantly boost strict accuracy.
Abstract
Determining the subcellular localization of proteins is critical for understanding their functional roles. With the rapid expansion of protein sequence databases, traditional experimental and homology-based approaches for Protein Subcellular Localization (PSCL) prediction are becoming increasingly impractical. Although several computational approaches have been developed, achieving high performance in terms of strict accuracy for multi-label PSCL prediction remains a significant challenge. This study evaluates the effectiveness of embeddings from five Protein Language Models (PLMs), including ProtBERT-BFD, ESM-2, ProtALBERT, ProLLaMA, and ProtGPT-2, as input features for various machine learning classifiers. Through five-fold cross-validation on the Swiss-Prot dataset, the results show that encoder-based PLMs, particularly ESM-2, combined with a Support Vector Machine (SVM) employing a polynomial kernel, consistently achieve the best performance. This configuration demonstrates a modest yet consistent improvement in strict-accuracy metrics compared to other model combinations. The proposed work offers a comprehensive evaluation of different PLM architectures and classifier complexities, achieving a strict accuracy of 0.58 while providing valuable insights into their performance trade-offs. This represents a 3 percentage point improvement over existing state-of-the-art models like DeepLoc 2.0. These results suggest that while current embeddings offer strong performance, further advancements in feature extraction and model architectures are needed to significantly boost strict accuracy.
Predicting protein subcellular locations computationally is crucial for analyzing large protein datasets. A key issue is that similar sequences in training and test sets artificially inflate accuracy estimates. This study investigates whether Protein Language Model (PLM) features alone can achieve strong predictions using simple classifiers instead of complex architectures. We developed a streamlined deep learning framework combining pre-trained ESM-2 embeddings with an attention-enhanced Bi-LSTM network, deployed as a 3-fold ensemble with soft voting. Training used eukaryotic sequences with ≤40% similarity to ensure rigorous evaluation. The model achieved 86.81% accuracy (MCC = 0.825) on test data—a +22.27% improvement over an SVM baseline (64.54%, MCC = 0.530). On 86 newly released 2024 proteins, the system reached 88.37% accuracy (MCC = 0.827), surpassing DeepLoc 2.1 (80.23%, MCC = 0.714, p=0.007) and MULocDeep (77.91%, MCC = 0.682, p=0.019). However, the small validation set (N=86) and limited representation in categories like Mitochondrion (N=5) require cautious interpretation. The method only handles single-location assignments across four compartments, excluding multi-location proteins. Attention weight analysis shows the model identifies biologically relevant signals, including C-terminal membrane regions and N-terminal mitochondrial sequences, confirming that ESM-2 embeddings enable effective performance with simplified architectures.
Johaimen Omar· International journal of lif...· 0 citations
Applications to thioredoxins, visual opsins, and Tara Oceans environmental diatom cold-shock proteins show that PLMView can move from interpretable residue-level determinants in well-studied protein families to large-scale environmental functional discovery, linking molecular specialization to ecological distribution and transcriptional deployment across the global ocean.
Accurate enzyme annotation remains a major bottleneck in translating rapidly growing protein sequence data into biological knowledge. Enzyme Commission (EC) prediction is particularly challenging because enzyme functions are organized hierarchically, annotations are often imbalanced across classes, and sequence similarity alone may be insufficient to resolve functional differences. To address these challenges, we developed ESM-ECForest, a two-stage framework that combines protein embeddings generated by the pretrained language model ESM-2 (Evolutionary Scale Modeling 2) with Random Forest classifiers. The first stage distinguishes enzymes from non-enzymes, whereas the second assigns one or more EC numbers to proteins predicted to be enzymatic. On an external benchmark comprising 25,778 protein sequences, ESM-ECForest achieved the highest weighted F1 score among the evaluated methods at all four EC levels, decreasing from 0.94 at Level 1 to 0.90 at Level 4. The largest relative improvements were observed for lyases (EC 4), ligases (EC 6), and translocases (EC 7), although EC 6 and EC 7 remained the most difficult classes internally. Visualization of the ESM-2 embedding space using Uniform Manifold Approximation and Projection (UMAP) revealed clustering patterns consistent with enzyme functional relationships, indicating that biologically relevant information is retained in the pretrained representations prior to supervised classification. These results support the use of pretrained protein language model embeddings as an effective foundation for enzyme annotation. By combining large-scale sequence representations with a lightweight supervised classifier, ESM-ECForest provides a scalable approach for EC prediction and may facilitate functional annotation of protein sequences derived from large genomic and metagenomic datasets.
A neural network-based pipeline that integrates amino acid sequences with structural features is developed and provides a modular prototype for follow-up, more extensive protein modeling, including larger proteins and sequence of variable sizes.
Carl David Jasper Causin, M. Fyta· APL Machine Learning· 0 citations
Disordered proteins (IDPs) and disordered protein regions (IDRs) have
important roles in cellular signalling and regulation and in disease development. However, their
flexible conformations create challenges in annotating them through computational methods. Recent
advances in IDP disorder predictive methods based on deep learning models (e.g., SPOTDisorder,
AUCpreD, IDP-Fusion) have improved the accuracy of IDP disorder prediction; however,
all current methods still require heavy computational resources and are lacking in the ability
to interpret their results.
A new lightweight method for predicting IDRs called IDP-T5CKNN, which uses embeddings
produced by ProtT5-XL-UniRef50 and a cosine similarity k-nearest neighbour (KNN) classifier
to predict the residue-level disorder in an input protein sequence, is proposed in this study.
The residue level embeddings have been normalised and class-balanced, and evaluated using
standard binary classification metrics. This study also estimated the computational costs for the
IDP-T5CKNN method using formal Big-O notation.
For the independent MXD494 dataset, the IDP-T5CKNN method had a maximum correlation
coefficient (MCC) of 0.5459 and a balanced accuracy coefficient (BAC) of 0.7906, outperforming
all other currently available IDP disorder predictors. For the sample from the fiDPnn
Test176 dataset, the IDP-T5CKNN method produced an MCC of 0.3942 and a BAC of 0.7163, with
approximately equal sensitivity and specificity, while also achieving a comparable performance to
deep neural networks without requiring iterative training.
The results of this study demonstrate that embeddings generated by protein language
models contain disorder-relevant information and that classification methods based on similarities
to other proteins can achieve similar performance levels as deep artificial neural networks, while
also improving our ability to understand the meaning of the outputs and reducing the amount of
computational resources needed to produce the desired results.
Overall, the IDP-T5CKNN method provides a low-cost, scalable, and interpretable
method for making residue-level disorder predictions for entire proteomes.
Deepak Chaurasiya· Current Computer Science· 0 citations
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By a common consensus, embeddings from the model's last layer are used, and the model's internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs from 11 datasets to investigate the informativeness of embeddings created in intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and computed characteristics of the latent spaces they span to estimate the information they contain, and found that the last layers of PLMs rarely contained embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between DTs and the distribution across PLMs'layers of the relevant information to predict that task. For example, similarity between the pre-training objective and the objective of predicting properties of individual residues leads to a steady increase in understanding of such tasks across the layers of PLMs. On the other hand, for whole-protein tasks, we observe that the dataset, rather than the task itself, defines PLMs'ability to perform well on a DT. Embeddings from shallow layers of PLMs perform better for datasets that contain deep mutational scan (DMS) data, while datasets containing diverse natural proteins find most useful embeddings in the models'deeper layers. Additionally, we discover that the performance of PLMs drops significantly when tasks are introduced for artificial proteins.
R. Joeres, Ilya S. Senatorov, A. Kolchina et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.