Linguistic Variety as a Framework for Evaluating LLM Behavior: A Comparative Study of Irish English and General American in Large Language Model Responses
Abstract
While large language models (LLMs) are increasingly used across social domains, current bias evaluations often rely on demographic proxies such as names, pronouns, and social categories. Linguistic variety itself receives less attention as an evaluative variable. This study therefore uses sociolinguistic variety to examine whether three LLMs respond differently to semantically equivalent prompts in General American English and Irish English. Thirty prompt pairs were administered twice to GPT-5, Claude, and Gemini, yielding 360 responses. Across the 180 paired comparisons, 92.8% differed in word count, although the direction varied by model: Claude produced longer Irish English responses on average, whereas GPT-5 and Gemini produced shorter responses. Claude explicitly referenced Irish English features in 55.0% of its Irish English responses, compared with 0.0% for GPT-5 and 1.7% for Gemini. No explicit correction, refusal, or researcher-observed tone shift occurred in either condition. These findings show variety-conditioned differences in response behavior, but they do not by themselves establish discriminatory biasThe study supports linguistic variety as an additional dimension for LLM differential-behavior evaluation and identifies model- and feature-specific patterns that warrant further evaluation with independent raters and additional varieties.