Mapping Scientific Knowledge in Systematic Literature Reviews Through Embeddings and Bibliometric Techniques

Abstract

Systematic literature reviews (SLRs) face challenges from rapid publication growth and low-quality AI-generated content. Simple database queries often retrieve publications that are not thematically coherent, making meaningful clustering difficult. This study aims to develop and evaluate a hybrid method to automate cluster assessment in SLRs, combining statistical measures of semantic similarity (embeddings from nomic-embed-text-v1.5) with bibliometric metrics (shared references, keywords, and Jaccard indices). Three case studies (17, 301, and 3113 publications) from the field of management were analyzed using embedding-based pre-clustering filtration, k-means clustering, t-SNE visualization, and GPT-4 labeling, validated through independent expert assessment (mean rating 4.29/5). Our results show that statistical and bibliometric metrics complement each other: bibliometric metrics uncover intellectual lineage, while statistical metrics assess semantic cohesion. Removing the least relevant articles via pre-clustering filtering consistently enhanced cluster coherence in the analyzed cases and produced more coherent publication sets than the complex Boolean queries used as a baseline. Furthermore, dataset size influences validation: bibliometric metrics can be misleading for small collections due to sparse networks, whereas statistical metrics remain reliable. This hybrid method provides a reproducible, scalable, open-source approach for automated cluster assessment in SLRs and will be implemented in the EmbedSLR open-source software.