Methodology First: Rethinking Research Assessment in RecSys (FRAME)

Abstract

We propose a novel workshop aimed at piloting a new paradigm for recommender systems research execution and evaluation. The workshop will foster the identification and formalization of new recommendation tasks beyond the saturated top-N setting, with shared and rigorous evaluation methodologies. In parallel, the workshop will explore a new paradigm for research assessment and validation, where contributions are initially judged solely on their experimental and methodological rigor with respect to predefined, industry-relevant tasks, and only in a second stage evaluated for their originality and broader scientific impact. The workshop will combine (i) an invitation-only interactive discussion session aimed at jointly rethinking both the research agenda and the evaluation culture of the research community, and (ii) an open paper submission session, piloting new experimental and reviewing methodologies. The long-term goal is to establish an iterative initiative in which successive editions refine alternative assessment models, develop emerging recommendation tasks and evaluation protocols, and strengthen the connection between academic research and industry needs.