基于VITS的高性能歌声转换模型
(浙江中医药大学 医学技术与信息工程学院,浙江 杭州 310053)
摘要
歌声转换是将源歌唱者的声音转换为目标歌唱者的声音,但保留原有的内容和旋律。随着科技的发展,各种网络结构和模型相继提出,歌声转换的算法也变得多样化,但难免都会出现转换音频质量差、失真率高、音域缺失等问题。文章提出以高保真流为基础的多解耦特征约束的UVC(Ultra Singing Voice Conversion)模型,该模型以VIT模型为基础,通过结合ContentVec编码器和NSF-HIFI-GAN声码器,改进模型的输入和输出,极大地提高了转换音频的质量和流畅性,并具有较强的鲁棒性。
关键词
歌声转换;VITS;ContentVec编码器;NSF-HIFI-GAN声码器
参考文献
- [1] NISHIMURA M,HASHIMOTO K,OURA K,et al. Singing Voice Synthesis Based on Deep Neural Networks [C]//Interspeech.San Francisco:ISCA,2016:2478-2482.
- [2] KALCHBRENNER N,ELSEN E,SIMONYAN K,et al. Efficient Neural Audio Synthesis [J/OL].arXiv:1802.08435 [cs.SD].(2018-02-23).https://arxiv.org/abs/1802.08435.
- [3] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al. Generative Adversarial Nets [C]//28th International Conference on Neural Information Processing Systems.Montreal:NIPS,2014,2:2672-2680.
- [4] QIAN K Z,ZHANG Y,GAO H T,et al. ContentVec: An Improved Self-supervised Speech Representation by Disentangling Speakers [J/OL].arXiv:2204.09224 [cs.SD].(2022-04-20).https://arxiv.org/abs/2204.09224.
- [5] KONG J,KIM J,BAE J,et al. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis [J/OL].arXiv:2010.05646 [cs.SD].(2020-10-12).https://arxiv.org/abs/2010.05646.
- [6] KIM J,KONG J,SON J,et al. Conditional VARIATIONAL Autoencoder with ADVERSARIAL Learning for End-To-End Text-To-Speech [J/OL].arXiv:2106.06103 [cs.SD].(2021-06-11).https://arxiv.org/abs/2106.06103.
- [7] JIA Y,ZHANG Y,WEISS R J,et al. Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis [J/OL].arXiv:1806.04558 [cs.CL].(2018-06-12).https://arxiv.org/abs/1806.04558.
- [8] HSU W-N,BOLTE B,TSAI Y-H H,et al. HuBERT: Self-supervised Speech Representation Learning by Masked Prediction of Hidden Units [J]IEEE/ACM Transactions on Audio, Speech, and Language Processing,2021,29:3451-3460.
- [9] KINGMA D P,WELLING M. Auto-Encoding Variational Bayes[J/OL].arXiv:1312.6114 [stat.ML].(2013-12-20).https://arxiv.org/abs/1312.6114.
- [10] CHEN M J,TAN X,REN Y,et al. MultiSpeech: Multi-speaker Text to Speech with Transformer [J/OL].arXiv:2006.04664 [eess.AS].(2020-06-08).https://arxiv.org/abs/2006.04664.
- [11] BERNARD M. Phonemizer [EB/OL].[2024-06-26].https://github.com/bootphon/phonemizer.
- [12] VEAUX C,YAMAGISHI J,MACDONALD K. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit [EB/OL].(2017-04-04)[2024-07-11].https://datashare.ed.ac.uk/handle/10283/2651.
- [13] SONG H,CHEN S Y,CHEN Z,et al. Exploring WavLM on Speech Enhancement [C]//2022 IEEE Spoken Language Technology Workshop (SLT).Doha:IEEE,2023:451-457.
- [14] BIŃKOWSKI M,DONAHUE J,DIELEMAN S,et al. High Fidelity Speech Synthesis with Adversarial Networks [J/OL].arXiv:1909.11646 [cs.SD].(2019-09-25).https://arxiv.org/abs/1909.11646.
- [15] WANG X Y,BAI Y P. The Global Minmax K-means Algorithm [J].SpringerPlus,2016,5(1):1665.
- [16] WAN L,WANG Q,PAPIR A,et al. Generalized End-to-End Loss for Speaker Verification [C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).Calgary:IEEE,2018:4879-4883.
- [17] YANG G,YANG S,LIU K,et al. Multi-band Melgan: Faster Waveform Generation for High-quality Text-To-Speech[C]//2021 IEEE Spoken Language Technology Workshop (SLT).Shenzhen:IEEE,2021:492-498.
- [18] KINGMA D P,BA J. Adam: A Method for Stochastic Optimization [J/OL].arXiv:1412.6980 [cs.LG].(2014-12-22).https://arxiv.org/abs/1412.6980.
- [19] TRAORÉ C,PAUWELS E. Sequential Convergence of AdaGrad Algorithm for Smooth Convex Optimization [J].Operations Research Letters,2021,49(4):452-458.
- [20] LI W W,LIU L,ZHANG J X. AdaRW Training Optimization Algorithm for Deep Learning Model of Marine Target Detection Based on SAR [J].International Journal of Remote Sensing,2021,43(1):120-131.
作者简介
周柯汝(2004—),女,汉族,浙江绍兴人,本科在读,研究方向:计算机科学与技术;通信作者:金伟(1989—),男,汉族,浙江宁波人,讲师,博士,研究方向:人工智能。