Accent-VITS: accent transfer for end-to-end TTS

Linhan Ma¹, Yongmao Zhang¹, Xinfa Zhu¹, Yi Lei¹, Ziqian Ning¹, Pengcheng Zhu², and Lei Xie^{1* 1} Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi'an, China ² Fuxi AI Lab, NetEase Inc., Hangzhou, China

1. Abstract

Accent transfer aims to transfer an accent from a source speaker to synthetic speech in the target speaker's voice. The main challenge is how to effectively disentangle speaker timbre and accent which are entangled in speech. This paper presents a VITS-based [1] end-to-end accent transfer model named Accent-VITS. Based on the main structure of VITS, Accent-VITS makes substantial improvements to enable effective and stable accent transfer. We leverage a hierarchical CVAE structure [2] to model accent pronunciation information and acoustic features, respectively, using bottleneck features and mel spectrums as constraints. Moreover, the text-to-wave mapping in VITS is decomposed into text-to-accent and accent-to-wave mappings in Accent-VITS. In this way, the disentanglement of accent and speaker timbre becomes be more stable and effective. Experiments on multi-accent and Mandarin datasets show that Accent-VITS achieves higher speaker similarity, accent similarity and speech naturalness as compared with a strong baseline Text2BN2Mel+Vocoder [3][4].

2. Dataset

Standard Mandarin Dataset -- DB1

Accent Mandarin Dataset with Multiple Accent Speakers

Dongbei (Northeast China):

Henan:

Shanghai:

Sichuan:

3. Demos--Accent Transfer

Accent: Dongbei (Northeast China)

Accent: Henan

Accent: Shanghai

Accent: Sichuan