A cross-sectional evaluation of Large Language Model answers to dental questions.

Background

Patients increasingly turn to the Internet and, more recently, to Large Language Models (LLMs) to seek answers to dental questions. However, the safety of LLM-generated dental advice for patients has not been systematically assessed.

Methods

Thirty dental questions spanning six disciplines were developed and posed in Polish and English to six LLMs. Each question was answered five times per model per language, yielding 2,700 unique LLM-generated answers, each independently evaluated by two dentists. A binary text classifier combining Qwen3-Embedding-0.6B word embeddings with a Support Vector Machine (SVM) was trained on 1,680 labeled answers and evaluated on 420 held-out answers.

Results

Substantial inter-rater agreement on harmfulness was observed (Cohen's [Formula: see text]). Without a system prompt, 56% of outputs from gpt-4o, gpt-4o-mini, and llama-3.3-70b were classified as harmful (pooling both languages; a question was counted as harmful if at least one of its five generated answers was labeled harmful by at least one annotator). A safety-oriented system prompt reduced this rate by 26 percentage points to 30%. llama-3.3-70b showed markedly higher harmfulness in Polish than English ([Formula: see text], [Formula: see text]). The trained classifier showed a tendency to mark non-harmful answers as harmful, though performance was lower for Polish than English outputs.

Conclusions

Publicly available LLMs can pose potential safety risks to dental patients when used without guardrails. System prompts significantly mitigate harmful outputs. Language-specific model training and hallucination reduction strategies, such as retrieval-augmented generation, are recommended as future directions.