This work presents a scalable, reproducible framework for evaluating, optimizing, and interpreting LLMs for biomedical knowledge extraction, with a focus on gene–gene regulatory relation prediction, pathway component recognition, multimodal pathway figure understanding, and automated prompt optimization.
Abstract
Biomedical knowledge extraction is fundamental to understanding complex biological systems, yet they remain challenging due to the rapid growth of biomedical literature, the heterogeneity of data sources, and the limitations of traditional rule-based and supervised learning approaches. Large language models (LLMs) have shown considerable promise for biomedical knowledge retrieval and discovery by demonstrating strong capacities for contextual reasoning and information extraction; however, their reliability, robustness, and interpretability in biomedical applications remain insufficiently understood. In particular, systematic evaluations against curated biological ground truth, multimodal pathway analysis, and scalable prompt optimization have received limited attention. This work presents a comprehensive investigation of LLMs for biomedical knowledge extraction, with a focus on gene–gene regulatory relation prediction, pathway component recognition, multimodal pathway figure understanding, and automated prompt optimization. First, a large-scale benchmarking study evaluates multiple proprietary and open-source LLMs using curated ground-truth data from the Kyoto Encyclopedia of Genes and Genomes (KEGG), which reveals substantial variability across models and persistent challenges in recall, directional inference, and hallucination control. Second, this work extends biomedical knowledge extraction beyond text by introducing a Genetic Prompt Vision framework for multimodal pathway image analysis. This framework employs evolutionary optimization to automatically generate effective multimodal prompts for extracting gene–gene interactions from biological pathway figures. Experimental results demonstrate that vision–language models enhanced through automated prompt optimization outperform traditional OCR-based pipelines in extraction accuracy and biological interpretability. Third, this study proposes a Bayesian Optimization Prompt framework for automated prompt optimization in biomedical relation extraction. This model-agnostic approach identifies robust prompts of ASCII strings without accessing model internals or performing fine-tuning. Across multiple LLMs, automated prompt optimization improves predictive performance while enhancing internal representation quality, as evidenced by improved semantic clustering and representation stability. Overall, this work establishes a scalable, reproducible framework for evaluating, optimizing, and interpreting LLMs for biomedical knowledge extraction. The findings provide practical insights into the strengths and limitations of LLMs in biomedical contexts and contribute methodological advances toward a reliable, interpretable, and multimodal LLM-based bioinformatics system.
Summary Biomedical information extraction (BIE) is fundamental for transforming unstructured biomedical text into structured, computable knowledge, yet the effectiveness of large language models (LLMs) remains limited by dataset heterogeneity and lack of unified benchmarks. We present InfoFlowEX, a unified framework for generalizable biomedical knowledge extraction with LLMs. InfoFlowEX incorporates an automated data integration pipeline using ontology-guided alignment to construct BIE-Corpus, a large-scale multi-domain benchmark unifying 40 public datasets for named entity recognition and relation extraction. We further introduce a task-conditioned schema instruction tuning strategy encoding 28 biomedical entity and relation types into a schema codebase, enabling LLMs to align heterogeneous annotations and generalize across settings. Finally, we evaluated InfoFlowEX in diverse applications, including evidence retrieval for question-answering, clinical diagnosis from electronic health records, and knowledge graph expansion. Results demonstrate that InfoFlowEX equips LLMs with robust adaptability, achieving consistent gains over baselines with minimal task-specific customization, highlighting InfoFlowEX for real-world biomedical applications.
Wuyang Lan, Siqi Zhang, Wenzheng Wang et al.· Cell Reports Medicine· 0 citations
The growth of biomedical literature poses significant challenges for researchers conducting systematic and scoping reviews. In fields such as the use of digital biomarkers for the treatment of heart failure and cardiovascular disease, manually screening thousands of papers is time-consuming and not scalable. To address this problem, we developed AI-assisted tools for large scale analysis and structured knowledge extraction from biomedical research articles. Our approach emphasizes cross-graph analytics, facilitating the exploration of relationships among key biomedical concepts, including digital biomarkers and digital health technologies. By automating the extraction and structuring of knowledge from unstructured text, our system aims to accelerate evidence of synthesis and support more comprehensive and up-to-date reviews in rapidly evolving biomedical domains.We developed a knowledge graph generation pipeline that extracts subject-predicate-object triplets representing scientific claims from research articles. To address redundancy caused by linguistic variation across documents, we implemented a multi-stage consolidation process focused on normalizing entities and relations. This process begins by validating and filtering extracted triplets, then applies lexical normalization to unify entity representations by removing stop words, resolving variants, and merging acronyms with their full forms. Entity types and relations are similarly standardized to ensure uniformity and clarity. The pipeline is designed to be modular and extensible, allowing for the integration of additional normalization strategies or domain-specific ontologies as needed.To further consolidate equivalent triplets, we leverage embedding-based semantic similarity, enabling the merging of semantically similar entities and relationships even when expressed differently across sources. Additionally, our pipeline utilizes biomedical ontologies such as RxNorm and MeSH to map entities to standardized concept identifiers. This ontology-based normalization ensures that references to the same biomedical concept are unified, regardless of linguistic or spelling differences. We evaluated our approach on a pilot corpus of 150 biomedical research articles, processing over 2,000 extracted triplets. The normalization pipeline reduced the number of unique entity variants by more than 50%, consolidating these into approximately 900 unique, semantically unified relationships. Manual review of a representative sample indicated entity normalization accuracy in the range of 90-95%. In conclusion, the integration of lexical, semantic, and ontology-based normalization offers a robust framework for reducing ambiguity and improving the interoperability of the resulting knowledge graphs. Moreover, this structured and unified representation of knowledge facilitates systematic reviews, meta-analyses, and data-driven decision-making in biomedical science, while enabling advanced querying, trend analysis, and the identification of novel associations between biomedical concepts
S. Krovvidi, Finn Vos, Laurent D. Hasson et al.· AHFE International· 0 citations
Biomedical knowledge discovery and clinical natural language processing increasingly rely on foundation models, but these approaches face persistent challenges, including incomplete curated knowledge, instability in language model outputs, and ambiguity in clinical interpretation. This dissertation argues that reliable biomedical AI requires integrating structured knowledge and foundation models through complementary control mechanisms rather than relying on any single method. To support this argument, the dissertation presents a unified framework across molecular and clinical domains. It develops an iterative prompt refinement approach with retrieval-augmented generation to improve the reliability of biomedical relation extraction, introduces graph transformer models to infer gene–gene relations and augment incomplete pathway knowledge, and proposes ontology-grounded retrieval and normalization pipelines to improve semantic alignment in multidisciplinary clinical notes. Together, these contributions define a layered reliability framework in which graph-based inference addresses knowledge incompleteness, retrieval-augmented prompt control mitigates instability, and ontology grounding reduces semantic ambiguity, providing a foundation for more reliable biomedical AI systems.
Knowledge Organization Systems like Ontologies and taxonomies are fundamental for structuring scientific knowledge, yet their manual curation presents a persistent bottleneck in knowledge management. While Large Language Models (LLMs) offer a scalable mechanism for automated ontology generation, their capacity to classify complex, domain-specific semantics requires systematic evaluation. In this paper, we assess the performance of five small, open-source LLMs (up to 9 billion parameters) in identifying semantic relationships between biomedical concepts. To support this evaluation, we introduce MeSH-Rel-4K, a dataset comprising 4K semantic relationships extracted from the Medical Subject Headings (MeSH). We analyse three adaptation strategies: standard prompting, Chain-of-Thought prompting, and fine-tuning. While parameter-constrained models traditionally struggle with the nuances of in-context logic, our results reveal that targeted fine-tuning increases the average F1-score by 34.1 percentage points. These results confirm that direct fine-tuning effectively exceeds the reasoning bottlenecks of smaller LLMs, providing an accurate, automated methodology for the construction and evolution of specialised biomedical ontologies.
Tanay Aggarwal, Angelo Salatino, Francesco Osborne et al.· 0 citations
Peered-reviewed literature provides reliable biomedical knowledge, which is essential for large language models (LLMs) in solving complex biomedical problems. However, how literature is retrieved and presented to LLMs can significantly influence their performance on domain-specific tasks.We present the Genomic Literature Knowledge Base (GLKB) and GLKB Agent to address these challenges. GLKB is a large-scale knowledge graph containing 14.6 million relationships among 3.2 million entities from 33 million PubMed abstracts and nine curated biomedical repositories. The current release includes articles published before March 2025. It supports diverse applications, including reinforcement learning, link prediction, and semantic embedding. The GLKB Agent is an agentic architecture that seamlessly connects LLMs to GLKB. It enables autonomous retrieval, reasoning, and deep research capabilities. Our evaluations demonstrate that the GLKB Agent dramatically improves LLM performance. Eight state-of-the-art LLMs achieve average accuracy gains of 27.5% on PubMedQA-HC, 24.8% on PubMedQA-Artificial, and 6.0% on BioASQ. Ablation tests confirm the agentic architecture is particularly effective for complex reasoning tasks. During datasource ablation tests, GLKB outperforms alternative data sources including PubMed, Wikipedia, and arXiv. Beyond question-answering, GLKB agent also demonstrates deep research capabilities through test-time reasoning. It generates comprehensive reports for literature reviews and hypothesis generation. The GLKB and GLKB Agent together provide a strong foundation for next-generation biomedical AI. Access is available at: https://glkb.org.
Yuanhao Huang, Zhaowei Han, Kevin Chang et al.· Proceedings of the 32nd ACM...· 0 citations
The rapid growth of biomedical knowledge has made the validation of automatically generated biological annotations a major bottleneck in biomedical curation. While computational methods can rapidly produce large numbers of candidate annotations, determining which are biologically valid still requires costly expert review. Prioritizing these candidates before manual curation has therefore become a fundamental challenge. Machine learning techniques can support this process by exploiting biomedical knowledge graphs (bioKGs), which capture biological entities and their functional associations. In this work, we propose a framework that leverages bioKGs to estimate the plausibility of candidate annotations and guide expert curation. Starting from knowledge graph embeddings, we train relation-specific binary classifiers using a community-based negative sampling strategy to obtain reliable confidence estimates. We then introduce a family of plausibility measures that combine classifier confidence, classifier reliability, and the semantic context provided by alternative relationships involving the same pair of biological entities. Unlike conventional confidence estimation, the proposed approach explicitly accounts for multiple biologically meaningful relations that may coexist between the same entities. Experimental results on five large bioKGs demonstrate that the proposed negative sampling strategy consistently improves classifier robustness, increasing balanced accuracy by an average of 5.8%. Moreover, the plausibility measures outperform classifier confidence alone, enabling more effective prioritization of candidate annotations for expert review. Overall, our results show that the use of bioKGs improves the efficiency of AI-assisted biomedical curation while preserving expert control over the final annotation assessment.
Emanuele Cavalleri, Miad Alavinezhad, Dario Malchiodi et al.· 0 citations