Establishing quality indicators for higher-order thinking skills: development and validation using factor analysis

Abstract

Assessing higher-order thinking skills (HOTS) in mathematics education remains challenging because available instruments often measure critical thinking (CT), creative thinking (CrT), and problem-solving (PS) separately, limiting a coherent evaluation of their interrelated nature. This study developed and empirically validated an integrated set of quality indicators for HOTS assessment using a mixed-methods design. In the qualitative phase, focus group discussions (FGDs) with five experts in mathematics education and educational evaluation generated theory-driven indicators grounded in instructional practice. In the quantitative phase, 70 participants (university lecturers, secondary school teachers, and undergraduate students) were selected through stratified purposive sampling to evaluate the indicators. An exploratory factor analysis (EFA) was performed using principal axis factoring with varimax rotation, and factorability was confirmed using Kaiser-Meyer-Olkin (KMO) and Bartlett’s test of sphericity. The results showed satisfactory construct validity (KMO=0.75; Bartlett’s p<0.001) and acceptable factor loadings (≥0.50), supporting a stable multidimensional HOTS structure aligned with theoretical expectations. The validated indicators provide a practical measurement framework to support cognitively demanding mathematics instruction and serve as a foundation for subsequent confirmatory factor analysis (CFA) in larger, more diverse samples.