Automated Medical Information Retrieval via Agentic Filter-and-Refine Pipeline

Abstract

In clinical decision support systems, finding the suitable medical literature for complex patient scenarios is critical. Although dense vector-based models offer high accuracy, the global vector index requirement creates scalability problems, while word-based search engines remain insufficient due to vocabulary mismatch. In this study, an Autonomous Filter-and-Refine structure operating on standard text search infrastructures is proposed. The developed architecture overcomes the vocabulary mismatch by applying entity extraction and Hypothetical Document Embedding with a large language model, and then determines the most relevant articles with a context-aware ranker. In the experiments conducted on the PMC-Patients dataset; the proposed Autonomous Doctor Agent architecture exhibited equivalent results with vector-based models in top-ten level performance metrics; whereas in the top-three article band, it achieved superiority by leaving existing systems behind.