Yun-Ji Chu

Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

ReACT-TTS is proposed, a two-stage framework that uses a one-second pre-response listener facial sequence to plan the next utterance's emotion and prosody before speech realization, and connects the predicted response style to a Grad-TTS backbone for end-to-end speech realization.