Skip to content

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

Sep 2026 · The Proceedings of the Workshop on Open Reasoning Across Cultures Languages (ORACLE) · pp. 459-467 · 0 citations · 11 references
Computer Science

TL;DR

The results show that culturally specific kinship generation remains difficult even when the relationship is explicitly stated and motivate generation-based evaluation alongside multiple-choice testing.

Abstract

Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a recognition problem. We instead prompt five open weight LLMs to generate kinship terms in three non Western languages (Hindi, Tamil, and Korean) across two communicative tasks and pair this with a matched option-supported selection baseline. On identical relation language cells, GPT OSS120B selects the correct term in 90.67% of 75 valid cells but produces an accepted term in 36.00% of the corresponding attempts; Llama 3.370B shows the same pattern (77.92% versus 24.24%). Since the four-option condition displays the candidate terms and does not require script production, the difference is interpreted as an evaluation format gap rather than direct proof that lexical knowledge is intact. On explicitly specified L3 prompts, accuracy varies sharply, from GLM-5.1 at 72.29% to Llama-3.370B at 24.24%. The paternal-lineage advantage is language specific; it is large in Hindi but weak or reversed in Korean, while Tamil shared-term pairs provide a control for measurement variation. These results show that culturally specific kinship generation remains difficult even when the relationship is explicitly stated and motivate generation-based evaluation alongside multiple-choice testing.

Read PDF

Similar papers

Preprint Nov 2026

LLMs Cannot Be Neutral: Gender Sensitivity Across Candidates, Evaluators, and Models

Large language models (LLMs) are increasingly used to evaluate candidates for professional roles, raising concerns about their sensitivity to demographic identities. We examine how candidate and evaluator gender assignments affect decisions by LLM-as-a-Judge (LaJ) models and decision language models (DLMs). In a hypoth...

Danial Amin, Joni O. Salminen, Bernard J. Jansen · 0 citations

Investigating Multi-Cultural Value Selection Biases in LLMs

Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs under simultaneous identity cues, with a country persona in the system prompt and a second identity signaled in the user turn, either by namin...

S. Pawar, Arnav Arora, Dustin Wright et al. · 0 citations
#natural language process... Conference Open access Sep 2026

Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models

Claims about the grammatical competence of multilingual language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm, post-training, and language resource availability has not been systematically examined. We evaluate base and post-trained models from six families on Mul...

Zhan-Yu Chen, Jaap Jumelet · 0 citations
#artificial intelligence Review Nov 2026

How Well Can Large Language Models Impersonate Individual Authors? A Comparative Evaluation

This study investigates the authorship impersonation capacity of six widely used large language models (LLMs): Gemini 3 Pro, Claude Opus, GPT 5.2, GPT 5.2 Pro, DeepSeek 3.2, and Grok 4.1 Fast. The evaluation was conducted using short product review texts selected to ensure high within author consistency in individual w...

Harshit Gupta, Shunichi Ishihara · 0 citations
#natural language process... Preprint Sep 2026

The Concrete-Arbitrary Gap: Kinship Reasoning in LLMs Is Not Indifferent to Presentation

We test whether large language models solve formally matched kinship problems equally well when relations are expressed in familiar vocabulary or by explicitly defined nonce predicates. Across 500 paired graphs, concrete accuracy exceeds arbitrary accuracy by 35.6 percentage points in local Qwen3.8-27B, 26.6 in Gemma 4...

Thomas Pashby · 0 citations
#natural language process... Preprint Oct 2026

Do Language Models Understand Igbo Culture? Beyond Multiple Choice

The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use the Igbo language as a case study. Existing benchmarks test translated Western ideas, lacking depth. To address this, we introduce CAB-Igbo, a benchmark developed from ind...

Blessing Ngozi Uzor, Valdini Douglace Lemofouet, P. Anyanwu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.