Codebook refinement transformer: post-retrieval correction for neural codec voice conversion

Abstract

We propose the Codebook Refinement Transformer (CRT), a non-autoregressive model that improves retrieval-based voice conversion on discrete neural audio codec tokens. CRT operates on the Mimi codec’s hierarchical residual vector quantized (RVQ) token space using a Conformer backbone with Feature-wise Linear Modulation (FiLM) speaker conditioning, trained with inference-matched FAISS corruption. We introduce confidence-selective decoding: CRT predictions override retrieved tokens only when softmax confidence exceeds a threshold. On a 100-pair zero-shot evaluation across VCTK speakers, CRT improves Mel Cepstral Distortion over the FAISS retrieval baseline (102.6 ± 15.1 vs. 103.3 ± 16.3) while achieving higher speaker embedding cosine similarity than kNN-VC (0.816 vs. 0.796) and reducing word error rate by 46% relative to the FAISS baseline (13.5% vs. 25.0%). CRT achieves real-time processing (RTF 0.90x on A100) with the neural network forward pass consuming only 8% of total latency. CRT’s token-to-token architecture integrates natively with codec-based speech systems, requiring no external vocoders or format conversion.