Hierarchical Multi-Layer Reranking Approach for Biomedical Retrieval-Augmented Generation

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) for domain-specific tasks such as medical question answering. However, recent reviews of RAG applications in the medical domain indicate that issues such as retrieval noise remain and require further investigation into mitigation strategies, with several prior works attempting to address this challenge. This study explores a multi-layer reranking approach that combines common RAG retrieval with topic and semantic tag similarity as "booster", rewarding documents with more relevant topics and tags. The main focus is on analyzing the contribution of multiple retrieval signals within a hierarchical reranking framework by using structured biomedical information such as PubMed—including MeSH-based topics—to refine document ranking. An ablation analysis comparing no reranking, full reranking, tag- and topic-only reranking was conducted at the retrieval level. Retrieval results across Recall@K shows high results, which means it is able to return relevant medical documents with the reranking method compared to the baseline configuration, with Recall@K reaching around 98.38% while baseline resulted in 97.92%. However, MRR result shows a slight decrease, indicating trade-offs in top-rank document ordering. Generation quality, evaluated using BLEU-4 and ROUGE, is comparatively lower. In contrast, domain-specific BERTScore (with a biomedical encoder) indicates high semantic similarity between generated and truth answers, averaging 96.99%. Overall, the results indicate that incorporating topic similarity and semantic tag signals can improve structured retrieval in domain-specific RAG systems.