Evaluating Multilingual Sentence Embeddings for Indonesian Open-Domain Passage Retrieval

Abstract

Indonesian search systems rarely have labeled data for fine-tuning dense retrievers, yet it is unclear which ready-to-use multilingual encoders can beat a properly configured lexical baseline. This study compares LaBSE, paraphrase-multilingual-mpnet-base-v2, and multilingual-e5-large, used without fine-tuning by the authors, against BM25 on the Indonesian subset of MIRACL: 960 dev queries over a 50,000-passage Wikipedia corpus that contains all 8,286 judged passages. Passages were indexed with FAISS and systems were scored with nDCG@10, MRR, and Recall@100. Multilingual-e5-large was the only encoder to surpass BM25 with regex tokenization, reaching nDCG@10 of 0.713 versus 0.579 (+23.1%), whereas paraphrase-multilingual-mpnet (0.502) and LaBSE (0.227) fell below it. A naive whitespace-tokenized BM25 scored only 0.354, which would have made two of the three encoders look superior. The results suggest that retrieval-oriented training matters more than language coverage, although multilingual-e5-large was also fine-tuned on MIRACL training data and the evaluation covers only Wikipedia passages. A search prototype built on the best model shows practical use.