Performance Evaluation of Embedding-Based and Keyword-Based Retrieval in Text Description-Based Hotel Recommendation
TL;DR
It is suggested that embedding-based retrieval is more effective in capturing semantic information, including synonym usage and implicit contextual relationships, within the evaluated dataset, and compact embedding models such as MiniLM may provide an alternative approach to traditional keyword-based retrieval methods for small-scale recommendation systems.
Abstract
The massive volume of textual descriptions on hotel booking platforms makes it difficult for recommendation systems to accurately match user preferences. Traditional keyword-based retrieval methods, such as TF-IDF, often struggle to capture semantic relationships when relevant terms do not explicitly overlap. This study evaluates the performance of keyword-based (TF-IDF) and embedding-based (paraphrase-multilingual-MiniLM-L12-v2) retrieval approaches in a content-based hotel recommendation system using a small-scale dataset. The dataset consists of 30 unique Traveloka hotels in Yogyakarta collected from Kaggle, representing a resource-constrained experimental setting. Evaluation was conducted using a black-box approach with 10 dynamic synthetic queries and assessed through Precision@3 (P@3) and Mean Average Precision (MAP). The results indicate that MiniLM achieved higher retrieval effectiveness than TF-IDF, with a mean P@3 of 0.3667 and a mean MAP of 0.1378, compared with 0.3000 and 0.1333, respectively. These findings suggest that embedding-based retrieval is more effective in capturing semantic information, including synonym usage and implicit contextual relationships, within the evaluated dataset. Therefore, compact embedding models such as MiniLM may provide an alternative approach to traditional keyword-based retrieval methods for small-scale recommendation systems.