EHRAdapt, an adapter that maps tuples directly into a frozen language model's embedding space, outperforms text-based LLM and count-based baselines, and both pathways improve rare-disease discrimination.
Abstract
Electronic health records (EHRs) encode clinical histories as (time, modality, code) tuples, whereas pretrained language models expect text tokens. Serializing tuples as text inflates sequence length and repeatedly encodes the same structure. We introduce EHRAdapt, an adapter that maps tuples directly into a frozen language model's embedding space. Modality and time gaps enter through a learned embedding and attention biases, and event codes receive dedicated vectors. Learning them is challenging because clinical vocabularies are long-tailed, leaving rare events with few observations. EHRAdapt therefore represents each event vector as the sum of a semantic prior and an evidence residual. The prior is a frozen, ontology-trained biomedical embedding of the event's clinical description, mapped into the input space by a shared learned projection, so it supplies clinical meaning even when observations are scarce. The residual is a learned low-rank, event-specific correction that refines it as evidence accumulates. We perform continued pretraining on records from about 4 million patients with three frozen LLM backbones, OLMo2 1B, Llama3.2 1B, and OLMo2 7B, training only the adapter (0.1--0.6% of all parameters). The full adapter outperforms all ablations in held-out next-event prediction on every backbone. Removing the semantic pathway hurts rare events most, over ten times more than the most frequent ones, whereas removing the residual hurts overall prediction but improves it for the rarest events. On reportable infectious-disease and syndromic classification, EHRAdapt outperforms text-based LLM and count-based baselines and the same LLMs adapted to the same records through next-event prediction, and both pathways improve rare-disease discrimination. The two pathways therefore play complementary roles, visible only when results are broken down by event frequency.
Structured electronic health records (EHRs) contain a patient's clinical trajectory as a sequence of clinical codes. Answering clinical questions from such records requires both the context of the whole trajectory and the specific events that support the answer. We introduce GLoC-EHR, a multimodal language model that r...
Interpretability analysis shows that demographics, organ system labs, drug ingredient features, and first-level ontology disease categories drive prediction, while deeper hierarchy levels contribute negligibly.
Mohamad Najafi, Hong-Yun Fu, M. Brochhausen et al.· 0 citations
BERT-LER is presented, a BERT-style model for coded EHR timelines pretrained and fine-tuned from a de-identified EHR dataset of 75 million patients, that encodes laboratory test results as discrete tokens while retaining graded information through percentile-based binning, paired with Integrated Gradients for token-lev...
Jun-Ni Du, Lukas Adamek, Maxim A Kryukov et al.· 0 citations
INTERVenE is presented, a family of Transformer architectures whose input is an interval-based, knowledge-based temporal abstraction (KBTA), a token stream of named clinical concepts drawn from a curated medical ontology, rather than an unnamed bin index or a raw measurement triplet.
Combining domain-specific embedding-based retrieval with LLM-based reranking substantially improves MCN performance compared to a standalone LLM approach, and supports robust semantic matching between German clinical expressions and standardized terminology concepts, facilitating structured representation and enabling...
Helena Adam, Akhila Abdulnazar, Roland Roller et al.· Studies in Health Technology...· 0 citations
Cross-system transfer of longitudinal Electronic Health Record (EHR) representations is challenging because clinical coding, patient populations, and healthcare workflows differ substantially across institutions and countries. We introduce Asymmetric Supervised Contrastive Learning (Asymmetric SupCon), a task-specific...