Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

Abstract

Timbre transfer aims to modify the timbral identity of a musical recording while preserving its original melody and rhythm. While single-instrument timbre transfer has made substantial progress, existing approaches to multi-instrument settings commonly rely on separate-then-transfer pipelines that propagate source-separation artifacts and process the resulting stems independently. This paper proposes MixtureTT, to the best of our knowledge the first system for flexible per-stem timbre transfer directly from a polyphonic mixture. Given a mixture and a separate timbre reference for each target voice, MixtureTT jointly transfers all stems to the specified instruments. Its Joint Stem Diffusion Transformer first models cross-stem harmonic context and then refines each output under its own content and timbre conditions. All stems are generated within a single joint diffusion process rather than through separate per-stem diffusion runs. This design avoids propagating waveform-level separation errors through a cascade and produces more coherent multi-stem outputs. Despite operating under a harder input condition, MixtureTT outperforms practical separate-then-transfer pipelines on objective and subjective evaluations, while remaining competitive with single-instrument methods given ground-truth isolated stems. Results on both synthesized and real-audio SATB benchmarks further demonstrate its generalization. Controlled ablations show that jointly modeling cross-stem dependencies improves over an equivalent independent-stem variant, while the structure and timing of this coordination are also critical for effective mixture-level timbre transfer.