NGramLang: A Domain-Specific Language for Document Retrieval Using Character N-Grams and Okapi BM25 Ranking

Abstract

NGramLang is a small domain-specific language and interpreter designed to make the mechanics of document retrieval explicit and executable. Through ten declarative commands, a user loads text documents, aggregates them into a corpus, and ranks them against a free-text query. The language front end is implemented with JavaCC and follows a single-pass interpretation model in which grammar actions build a typed symbol table and enforce semantic constraints such as n-gram-size agreement. The retrieval back end represents documents, corpora and queries as vectors of overlapping character n-grams and scores documents using the Okapi BM25 weighting function with its standard parameters (k = 2.0, b = 0.75). The paper describes the grammar, the interpretation strategy, the vectorization procedure and the ranking model, and illustrates execution with a worked example: on four short documents with the query "test", the interpreter produces a BM25 ranking that places the most relevant document first and assigns a score of zero to a document sharing no query n-grams, confirming the pipeline works end to end.