Human-Coded Evaluation of Machine Translation and Large Language Model Agents for Chinese City Publicity Texts

Abstract

This study evaluates how web-based machine translation (MT) systems and large language model (LLM) translation agents perform in the English translation of Chinese city publicity texts. City publicity translation is a high-stakes form of institutional intercultural communication: it must be factually accurate, culturally legible, pragmatically appropriate, accessible to international readers, and capable of representing a city image without exaggeration or distortion. A corpus of 90 official Chinese source segments from Qingdao, Xi'an, and Hangzhou was translated under four conditions: DeepL web MT, Google Translate web MT, a GPT-5.5 translation agent, and a DeepSeek V4 Pro translation agent, yielding 360 English translations. Two trained coders independently evaluated all translations on five 1-5 dimensions: accuracy, cultural adequacy, pragmatic appropriateness, audience accessibility, and city image representation. Formal coding showed acceptable to strong reliability: Cohen's kappa for primary issue coding was 0.777, and quadratic weighted kappa values for the five rating dimensions ranged from 0.786 to 0.847. The strongest composite score was observed for DeepL web MT (M=4.683), followed by GPT-5.5 (M=4.599), DeepSeek V4 Pro (M=4.553), and Google Translate (M=4.261). Paired permutation tests showed that DeepL, GPT-5.5, and DeepSeek V4 Pro all significantly outperformed Google Translate on composite quality, while the differences between DeepL and the two LLM agents were not statistically significant. The findings therefore do not support a simple claim that LLM agents uniformly surpass MT. Instead, they suggest that LLM agents can reach a strong MT baseline and may offer pragmatic and audience-oriented affordances, while their value depends on the benchmark system, the target discourse function, and the evaluation dimension.