Towards a New Grammar Checker for Bulgarian
TL;DR
The dataset BulMMLU:gr, modelled on the MMLU (Massive Multitask Language Understanding) benchmark, is presented, to test the ability of AI grammar-checking applications and different LLMs to detect grammatical errors in Bulgarian and thus provide a basis for analysing the need for a specialised LLM-based tool to detect such errors and suggest corrections.
Abstract
This paper presents the dataset Massive Multitask Language Understanding of Bulgarian Grammar (BulMMLU:gr), modelled on the MMLU (Massive Multitask Language Understanding) benchmark. Its aim is to test the ability of AI grammar-checking applications, as well as different LLMs, to detect grammatical errors in Bulgarian and thus provide a basis for analysing the need for a specialised LLM-based tool to detect such errors and suggest corrections. Experiments conducted with two grammar-checking applications and five LLMs reveal limitations that highlight the need for a new, dedicated Bulgarian grammar-checking system. BulMMLU:gr can be further extended and used both for evaluation and for fine-tuning LLMs for grammatical error detection and correction suggestions.