Curiosity-Guided Graph Neural Skill Discovery for Sparse-Reward Goal-Oriented Dialogue Systems

Abstract

Goal-oriented chatbots often operate in sparse-reward settings, where meaningful feedback (task success or user satisfaction) arrives only after many dialogue turns. This makes exploration and policy learning challenging. We propose a novel hierarchical reinforcement learning (HRL) framework that performs automated skill discovery using curiosity-driven exploration and graph-based temporal clustering. Dialogue trajectories are represented as a dynamic graph with utterance embeddings as nodes and transition probabilities as edges. A Graph Neural Network (GNN) combined with curiosity-weighted clustering identifies reusable dialogue skills (options) such as greeting, intent clarification, information gathering, and confirmation. These options are integrated into a hierarchical policy where a high-level policy selects skills and a low-level policy generates utterances. Evaluated on MultiWOZ 2.2 and a custom customer service dataset, the proposed framework achieves up to 22% higher task success rate and 18% shorter dialogues compared to strong HRL and flat RL baselines. The discovered skills demonstrate strong cross-domain transferability, making the framework practical for scalable, adaptive goal-oriented dialogue systems.