It is established that shared structural regularities between natural language and biological sequences do not imply symmetric representational transfer, revealing an intrinsic directionality in cross-domain foundation-model learning.
Abstract
Cross-domain transfer is a defining property of foundation models, yet whether such transfer is symmetric across domains remains unknown. Prior work has reported a striking transfer from natural language to biological sequences: language models fine-tuned only on English structural tasks acquire zero-shot protein-homology discrimination. Here we ask the converse and general question—is structural transfer between language and biology directional? —and answer it systematically. We first reproduce forward transfer (language →biology) under controlled conditions, then evaluate the reverse direction (biology →language) across fine-tuning, iso-token continued pretraining, model scaling, multiple biological foundation-model families (ESM-2, ProtBERT), and adversarial synthetic structure tasks. Reverse transfer is consistently weak: it does not exceed matched-token controls, does not scale, and does not generalize. In an architecture-matched 2 × 2 analysis on models with known training data—which eliminates the pretraining-contamination confound that clouds large-model studies—a language model retains far more competence when moved to biology (off-domain drop 0.08) than a biological model retains when moved to language (drop 0.36). Scaling widens rather than closes this gap: language →biology transfer strengthens with size while biology →language transfer decays toward chance, a pattern shared by two independent protein-model families. Our findings establish that shared structural regularities between natural language and biological sequences do not imply symmetric representational transfer, revealing an intrinsic directionality in cross-domain foundation-model learning.
The factors underlying performance differences across matcher architectures are clarified and motivate future research and benchmark designs that better disentangle architectural choices from model-level factors while explicitly evaluating distribution shift and cross-dataset transferability.
Zeyu Zhang, Xue Li, Iacer Calixto et al.· 0 citations
This paper introduces Caduceus, a family of MoE-enhanced foundation models built with a hierarchical pre-training paradigm to jointly integrate biological and natural language, and incorporates a multi-task instruction tuning phase, enabling robust protein parsing and natural language question answering.
Mingze Yin, Yiheng Zhu, Jialu Wu et al.· Proceedings of the 32nd ACM...· 0 citations
This work combines learnability-based approaches with a reconstruction framework, fine-tuning GPT-2 models pre-trained on impossible languages to reconstruct natural English from three perturbation types and shows that the recovered structures exhibit shorter dependency lengths, mirroring the locality preference observed in unconstrained language model generation.
Amir H Mohammadi, Laurence E. Frank, Albert Gatt et al.· 0 citations
End-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal, and it is concluded that end-to-end activation-state transfer between LLMs is architecture-dependent rather than universal.
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Conventional protein language models are pretrained on full-length sequences that interleave multiple domains with linkers and disordered regions, diluting fold-specific signals. Our approach pretrains masked language models on structurally-defined domain segments from The Encyclopedia of Domains. TEDlm learns from domain sequences alone with a standard MLM objective, while its variant TEDlm3D adds a Cα distance-guided contact loss that supervises the attention maps. On CATH S40 remote-homology detection (<40% identity), the domain-centric pretraining has a bigger effect than model scale: at the final layer, a 650M-parameter TEDlm achieves an AUROC1 of 0.28 compared to 0.22 for ESM2 3B, whereas TEDlm3D reaches 0.50, approaching the structure-based search tool Foldseek (0.53) from sequence alone at inference. Attention-map and categorical Jacobian probes show that the contact signal is encoded in the model representations themselves, not only in a trained output head. TEDlm variants also substantially improve zero-shot Molecular Function prediction over ESM2, while matching it on various biophysical property tasks, indicating that signals are largely domain-intrinsic. Together, these results position domain-centric pretraining as a route to compact, structurally informed protein language models.
Tiejun Wei, S. Kandathil, Daniel W. A. Buchan et al.· bioRxiv· 0 citations