Teaching Recommenders to Listen: Bidirectional Natural-Language Control for Personalized News

Abstract

Personalized recommenders learn mostly from implicit signals such as clicks and a few coarse topic ratings. These signals are a narrow channel: a reader cannot easily ask for “more soccer World Cup coverage this month and less celebrity news,” and today’s systems give little indication of whether such a request was understood or acted on. This research asks how to let users steer a personalized news newsletter using natural language. Our aim is to support collection-level feedback in natural language—feedback about the balance and mix of the whole newsletter rather than a single article—and to make the exchange bidirectional, so the system both acts on a request and reports what it did. This extends natural-language critiquing, which has largely addressed individual recommendations, to an assembled collection, and foregrounds the natural-language interaction in a live newsletter setting. We pursue the goal through three planned studies on POPROX, a live platform that emails daily personalized newsletters (15 articles in five sections, from the Associated Press) to real subscribers and supports randomized experiments. Study 1 investigates how to elicit actionable collection-level instructions and develops a rubric for scoring them. Study 2 builds the channel—an LLM post-ranker over the deployed neural ranker (NRMS)—and evaluates it in a five-week randomized controlled trial (RCT). Study 3 adds a per-newsletter explanation of what the system understood, changed, and could not do, and tests whether it improves understanding and the quality of users’ revised instructions, while examining its effects on engagement. None of the studies has been run; this paper presents the plan, the designs, and a baseline-informed analysis.