Argument-based validation of the English–Japanese Phrasal Vocabulary Size Test: The role of familiarity in item difficulty

Abstract

This study validates the English–Japanese Phrasal Vocabulary Size Test (E–J PVST), a bilingual adaptation of the Phrasal Vocabulary Size Test (Martinez, 2011) assessing receptive knowledge of high-frequency English phrasal expressions among L1 Japanese adults. Using Chapelle et al.’s (2008) argument-based validation framework, evidence is synthesized for the domain definition, evaluation, generalization, and explanation inferences. The test was administered online to 226 adults spanning CEFR A1 to C1. Rasch analyses indicated high item reliability (.97) and person reliability (.88), and provided provisional support for a largely unidimensional structure. However, 12% of items showed underfit and 22% showed overfit, with distractor malfunction identified as a primary source of misfit. Scores increased monotonically across CEFR levels. Learner-reported familiarity was significantly correlated with item difficulty, whereas corpus frequency was not, with more familiar expressions tending to be easier. The study demonstrates that argument-based validation can identify specific strengths and weaknesses in low-stakes vocabulary tests, and highlights the role of subjective familiarity, more than corpus frequency, in predicting item difficulty for multiword expressions.