Exploring the use of mathematics observation tools across the contexts of the United States, Norway, and Finland: how can observation instruments shape our understanding of instructional quality when applied across contexts?

Abstract

Research on effective mathematics instruction informs teacher preparation, professional learning, student assessment, instructional materials, and classroom observation instruments. Observation protocols are designed to capture instructional quality and the extent to which research-based practices are embedded in classroom instruction. However, although such instruments are typically developed and validated within specific educational systems, they are increasingly used across national and cultural contexts, on the assumption that they will function similarly across settings. Relatively little research has examined how raters interpret and negotiate instructional quality when applying shared observation frameworks across contexts. This study investigates classroom observation as an interpretive process shaped by the interaction among protocol design, instructional context, and rater reasoning. Specifically, the study examines how researchers with different professional backgrounds used three validated observation instruments developed in the United States to analyze one mathematics lesson from the United States, Norway, and Finland. Using a mixed-methods design, the study combined observation scores with recorded collaborative scoring discussions to examine how raters justified and negotiated evaluative judgments. Through discourse-oriented qualitative analysis, the study identified recurring forms of interpretive variation related to operational ambiguity, contextual interpretation, construct overlap, and differing evidentiary thresholds across raters and instruments. The findings contribute to methodological discussions concerning interpretive validity, observer reasoning, and the transportability of classroom observation instruments across educational contexts. The study highlights the importance of examining not only observation scores, but also the interpretive processes through which judgments of instructional quality are produced.