Skip to content
Book Open access

POLO: Preference-Guided Multi-Turn Reinforcement Learning for Sample-Efficient Lead Optimization

Aug 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · pp. 12351-12360 · 0 citations · 45 references

TL;DR

POLO (Preference-guided Optimization for Lead Optimization), a multi-turn Reinforcement Learning framework for sample-efficient lead optimization, which extracts learning signals from each trajectory at two complementary levels: trajectory-level policy optimization reinforces successful multi-turn strategies, while turn-level preference learning ranks intermediate molecules by oracle-derived rewards and constructs within-trajectory pairwise comparisons to provide denser learning signals for effective modifications.

Abstract

Lead optimization in drug discovery requires iteratively refining molecular candidates while preserving structural similarity to the original compound. Since each evaluation is costly, sample efficiency, the ability to achieve strong performance with limited oracle calls, becomes critical. Existing methods, from genetic algorithms to LLM-based approaches, rarely leverage intermediate outcomes and oracle feedback within an optimization trajectory, which reduces sample efficiency and forces more oracle calls to reach high success rates. To address this, we present POLO (Preference-guided Optimization for Lead Optimization), a multi-turn Reinforcement Learning framework for sample-efficient lead optimization. At its core, POLO introduces Preference-Guided Policy Optimization (PGPO), which extracts learning signals from each trajectory at two complementary levels: trajectory-level policy optimization reinforces successful multi-turn strategies, while turn-level preference learning ranks intermediate molecules by oracle-derived rewards and constructs within-trajectory pairwise comparisons to provide denser learning signals for effective modifications. Crucially, these preference signals require no additional oracle calls, as they are derived from evaluations already collected during rollouts. Experiments demonstrate that POLO achieves 85% success rate on single-property tasks (40% improvement over best baselines) and 50% on multi-property tasks using only 500 oracle calls. Our code is available at https://github.com/REAL-Lab-NU/POLO.

Read PDF

Similar papers

#machine learning Preprint Aug 2026

Elite-Weighted Supervised Fine-tuning for Goal-Directed Molecular Optimization

This work introduces Elite-Weighted Supervised Fine-tuning (EW-SFT), which uses reward to guide elite selection of high-scoring molecules, and updates the model by its own pretraining loss on that set, and consistently outperforms the corresponding native optimizers.

Shiyun Wa, Yifei Wang, A. G. Green et al. · 0 citations
#artificial intelligence Preprint Oct 2026

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is diffi...

Hao-Ran Li, Z. Ge, Xiao-Ming Yuan et al. · 0 citations

MARCO: Multi-Round Agentic Reinforcement for Conditional Molecular Optimization

This work introduces MARCO, an evaluator-grounded reinforcement-learning framework that trains molecular editors on bounded proposal--feedback--revision trajectories and aggregates shaped turn rewards into an undiscounted trajectory return for group-relative policy optimization.

Shi-Cheng Fang, Yu-Xin Wang, Zhuo Yang et al. · 0 citations
Open access Aug 2026

A closed-loop reinforcement learning framework for rapid compound directed optimization

Generative artificial intelligence (AI) holds transformative potential for drug discovery, yet existing architectures typically operate in open loops without experimental feedback. Here we introduce rapid compound directed optimization (RCDO), a closed-loop reinforcement learning framework that accelerates the optimiza...

Han Wang, Dehua Lu, Weiping Lyu et al. · 0 citations
Open access Aug 2026

The impact of reward scalarization and weight scheduling on optimization dynamics in multi-objective molecular design

Inverse molecular design imposes multiple property constraints, requiring them to be combined into a scalar reward for reinforcement-learning (RL) fine-tuning; a step that can be highly sensitive to the reward formulation. Here, we systematically analyze how scalarization choices and stabilization mechanisms shape opti...

Lucas Leuschner, Oscar Palomino-Hernández · 0 citations
Open access Oct 2026

MoDeMol: A Decomposition-Based Evolutionary Framework for Multi-Objective Molecular Optimization

This work proposes a decomposition-based evolutionary framework built on a score-informed latent space (MoDeMol), with four core innovations that address representation alignment, structural redundancy, and search-resource allocation within a computational multi-target lead-optimization workflow under limited evaluatio...

Chen-Bing Guo, Cai Dai · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.