Human Value in Recommendations: Towards Improved Evaluation Measures

Abstract

When evaluating top-N recommendations, objectives beyond accuracy are increasingly considered, reflecting a broader shift toward user-centric evaluation. However, existing evaluation approaches present several limitations: (1) they fail to quantify how much users actually value beyond-accuracy objectives, (2) they lack strong user-centric validation, and (3) they overlook the full scope of user preferences, including intra-list composition and structure. These limitations highlight the need for evaluation approaches that better align with user preferences. Therefore, in the domain of movie recommendations, this proposed research investigates user preferences over multiple recommendation objectives in a controlled user study. By systematically varying the contribution of each objective and observing user preferences across these variations, the study infers both individual preferences and the relative importance of each objective. The overarching goal is to develop evaluation measures that better reflect how users actually perceive recommendation quality.