引入预训练表示混合矢量量化和CTC的语音转换

doi:10.19678/j.issn.1000-3428.0067272

摘要/Abstract

摘要： 预训练模型通过自监督学习表示在非平行语料语音转换(VC)取得了重大突破。随着自监督预训练表示(SSPR)的广泛使用,预训练模型提取的特征中被证实包含更多的内容信息。提出一种基于SSPR同时结合矢量量化(VQ)和联结时序分类(CTC)的VC模型。将预训练模型提取的SSPR作为端到端模型的输入,用于提高单次语音转换质量。如何有效地解耦内容表示和说话人表示成为语音转换中的关键问题。使用SSPR作为初步的内容信息,采用VQ从语音中解耦内容和说话人表示。然而,仅使用VQ只能将内容信息离散化,很难将纯粹的内容表示从语音中分离出来,为了进一步消除内容信息中说话人的不变信息,提出CTC损失指导内容编码器。CTC不仅作为辅助网络加快模型收敛,同时其额外的文本监督可以与VQ联合优化,实现性能互补,学习纯内容表示。说话人表示采用风格嵌入学习,2种表示作为系统的输入进行语音转换。在开源的CMU数据集和VCTK语料库对所提的方法进行评估,实验结果表明,该方法在客观上的梅尔倒谱失真(MCD)达到8.896 dB,在主观上的语音自然度平均意见分数(MOS)和说话人相似度MOS分别为3.29和3.22,均优于基线模型,此方法在语音转换的质量和说话人相似度上能够获得最佳性能。

关键词: 预训练表示, 自监督学习, 矢量量化, 解耦, 联结时序分类

Abstract: Pre-trained models have achieved significant breakthroughs in nonparallel corpus Voice Conversion (VC) via Self-Supervised Pre-trained Representation (SSPR). Features extracted by pre-trained models contain a significant amount of content information owing to the widespread use of SSPR. This study proposes a VC model based on the combination of SSPR Vector Quantization (VQ) and Connectionist Temporal Classification (CTC). It uses the SSPR extracted from a pre-trained model as input to improve the quality of single VC. The effective decoupling of content and speaker representations has become a key issue in VC. Using SSPR as the initial content information, VQ is performed to decouple content and speaker representations from speech. However, performing only VQ discretizes only the content information, thus rendering it difficult to separate pure content representations from speech. To further eliminate speaker-invariant information from the content information, a CTC loss-guided content encoder is proposed. CTC not only serves as an auxiliary network to accelerate model convergence but also its additional text supervision can be jointly optimized with VQ to achieve complementary performance and learn pure-content representations. Speaker representations adopt style-embedding learning, and two representations are used as inputs for VC in the system. The proposed method is evaluated on the open-source CMU dataset and VCTK corpus. Experimental results show that the proposed method achieves an objective Mel-Cepstrum Distortion(MCD) of 8.896 dB, as well as subjective Mean Opinion Score (MOS) of speech naturalness and speaker similarity of 3.29 and 3.22, respectively, both of which are better than those of the baseline model. This method achieves the best performance in terms of VC quality and speaker similarity.

Key words: pre-trained representation, Self-Supervised Learning(SSL), Vector Quantization(VQ), decouple, Connectionist Temporal Classification(CTC)

中图分类号:

TP391

王琳, 黄浩. 引入预训练表示混合矢量量化和CTC的语音转换[J]. 计算机工程, 2024, 50(4): 313-320.

WANG Lin, HUANG Hao. Voice Conversion Combining Vector Quantization and CTC Introducing Pre-Trained Representation[J]. Computer Engineering, 2024, 50(4): 313-320.

https://www.ecice06.com/CN/Y2024/V50/I4/313

参考文献

[1] 张凯, 朱立新, 赵义正. 基于重训练高斯混合模型的语音转换方法[J]. 声学技术, 2010, 29(1):52-55. ZHANG K, ZHU L X, ZHAO Y Z. A voice conversion method based on retraining GMM[J]. Technical Acoustics, 2010, 29(1):52-55.(in Chinese)
[2] CHEN L H, LING Z H, LIU L J, et al. Voice conversion using deep neural networks with layer-wise generative training[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014, 22(12):1859-1872.
[3] SUN L F, KANG S Y, LI K, et al. Voice conversion using deep bidirectional long short-term memory based recurrent neural networks[C]//Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA:IEEE Press, 2015:4869-4873.
[4] SUN L F, LI K, WANG H, et al. Phonetic posteriorgrams for many-to-one voice conversion without parallel data training[C]//Proceedings of the IEEE International Conference on Multimedia and Expo. Washington D. C., USA:IEEE Press, 2016:1-6.
[5] 高俊峰, 陈俊国. 基于Style-CycleGAN-VC的非平行语料下的语音转换[J]. 计算机应用与软件, 2021, 38(9):133-139, 159. GAO J F, CHEN J G. Voice conversion with non-parallel corpus based on Style-CycleGAN-VC[J]. Computer Applications and Software, 2021, 38(9):133-139, 159.(in Chinese)
[6] KAMEOKA H, KANEKO T, TANAKA K, et al. ACVAE-VC:non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder[EB/OL].[2023-02-23]. https://arxiv.org/pdf/1808.05092.pdf.
[7] 谭智元. 基于自编码器的零样本语音转换系统研究[D]. 天津:天津大学, 2020. TAN Z Y. A study of zero-shot voice conversion system based on auto-encoder[D].Tianjin:Tianjin University, 2020. (in Chinese)
[8] SAITO Y, NAKAMURA T, IJIMA Y, et al. Non-parallel and many-to-many voice conversion using variational autoencoders integrating speech recognition and speaker verification[J]. Acoustical Science and Technology, 2021, 42(1):1-11.
[9] 李燕萍, 曹盼, 左宇涛, 等. 基于i向量和变分自编码相对生成对抗网络的语音转换[J]. 自动化学报, 2022, 48(7):1824-1833. LI Y P, CAO P, ZUO Y T, et al. Voice conversion based on i-vector with variational autoencoding relativistic standard generative adversarial network[J]. Acta Automatica Sinica, 2022, 48(7):1824-1833.(in Chinese)
[10] GRAVES A, FERNÁNDEZ S, GOMEZ F, et al.Connectionisttemporal classification:labelling unsegmentedsequence data with recurrent neural networks[EB/OL].[2023-02-23]. https://mediatum.ub.tum.de/doc/1292048/12285.pdf.
[11] WANG C Y, WU Y, QIAN Y, et al. UniSpeech:unified speech representation learning with labeled and unlabeled data[EB/OL].[2023-02-23]. https://arxiv.org/abs/2101.07597.
[12] VAN DEN OORD A, LI Y Z, VINYALS O. Representation learning with contrastive predictive coding[EB/OL].[2023-02-23]. https://arxiv.org/pdf/1807.03748.pdf.
[13] CHUNG Y A, GLASS J. Generative pre-training for speech with autoregressive predictive coding[C]//Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D. C., USA:IEEE Press, 2020:3497-3501.
[14] LIU A T, YANG S W, CHI P H, et al. Mockingjay:unsupervised speech representation learning with deep bidirectional transformer encoders[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA:IEEE Press, 2020:6419-6423.
[15] LIN Y Y, CHIEN C M, LIN J H, et al. FragmentVC:any-to-any voice conversion by end-to-end extracting and fusing fine-grained voice fragments with attention[C]//Proceedings of the International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA:IEEE Press, 2021:5939-5943.
[16] RAO K, SENIOR A, SAK H. Flat start training of CD-CTC-SMBR LSTM RNN acoustic models[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA:IEEE Press, 2016:5405-5409.
[17] ZHAO Z P, BAO Z T, ZHANG Z X, et al. Attention enhanced connectionist temporal classification for discrete speech emotion recognition[C]//Proceedings of the International Symposium on Computer Architecture. Phoenix, Arizona, USA:[s.n.], 2019:1-10.
[18] CHEN S Y, WANG C Y, CHEN Z Y, et al. WavLM:large-scale self-supervised pre-training for full stack speech processing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6):1505-1518.
[19] 邓宏贵, 郭晟伟, 李志坚. 基于哈夫曼编码的矢量量化图像压缩算法[J]. 计算机工程, 2010, 36(4):218-219, 222. DENG H G, GUO S W, LI Z J. VQ image compression algorithm based on Huffman coding[J]. Computer Engineering, 2010, 36(4):218-219, 222.(in Chinese)
[20] CHOU J C, YEH C C, LEE H Y. One-shot voice conversion by separating speaker and content representations with instance normalization[EB/OL].[2023-02-23]. https://arxiv.org/pdf/1904.05742.pdf.
[21] WANG Y X, STANTON D, ZHANG Y, et al. Style Tokens:unsupervised style modeling, control and transfer in end-to-end speech synthesis[EB/OL].[2023-02-23]. http://arxiv.org/abs/1803.09017.
[22] KUMAR K, KUMAR R, DE BOISSIERE T, et al. MelGAN:generative adversarial networks for conditional waveform synthesis[EB/OL].[2023-02-23]. http://arxiv.org/abs/1910.06711.
[23] VEAUX C, YAMAGISHI J, MACDONALD K. CSTR VCTK corpus:English multi-speaker corpus for CSTR voice cloning Toolkit[EB/OL].[2023-02-23]. https://datashare.ed.ac.uk/handle/10283/3443.
[24] KOMINEK J, BLACK A W. The CMU Arctic speech databases[C]//Proceedings of the 5th ISCA Speech Synthesis Workshop. Washington D. C., USA:IEEE Press, 2004:1-10.
[25] KANG X, HUANG H, HU Y, et al. Connectionist temporal classification loss for vector quantized variational autoencoder in zero-shot voice conversion[J]. Digital Signal Processing, 2021, 116(6):103110.
[26] HSU C C, HWANG H T, WU Y C, et al. Voice conversion from non-parallel corpora using variational auto-encoder[EB/OL].[2023-02-23]. https://arxiv.org/abs/1610.04019v1.

选择文件类型/文献管理软件名称

选择包含的内容