High-performance Singing Voice Conversion Model Based on VITS
(School of Medical Technology and Information Engineering, Zhejiang Chinese Medical University, Hangzhou 310053, China)
Abstract
Singing voice conversion is the process of transforming the voice of the source singer into that of the target singer while retaining the original content and melody. With the development of technology, various network architectures and models have been put forward one after another, and the algorithms for singing voice conversion have also become diversified. However, problems such as poor quality of the converted audio, high distortion rates, and lack of vocal range are bound to occur. This paper proposes UVC (Ultra Singing Voice Conversion) model with multi-decoupled feature constraints based on high-fidelity flow. This model is built on the basis of the VIT model. By combining the ContentVec encoder and the NSF-HIFI-GAN vocoder, it improves the input and output of the model, greatly enhancing the quality and fluency of the converted audio and possessing strong robustness.
Keywords
singing voice conversion; VITS; ContentVec encoder; NSF-HIFI-GAN vocoder
References
- [1] NISHIMURA M,HASHIMOTO K,OURA K,et al. Singing Voice Synthesis Based on Deep Neural Networks [C]//Interspeech.San Francisco:ISCA,2016:2478-2482.
- [2] KALCHBRENNER N,ELSEN E,SIMONYAN K,et al. Efficient Neural Audio Synthesis [J/OL].arXiv:1802.08435 [cs.SD].(2018-02-23).https://arxiv.org/abs/1802.08435.
- [3] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al. Generative Adversarial Nets [C]//28th International Conference on Neural Information Processing Systems.Montreal:NIPS,2014,2:2672-2680.
- [4] QIAN K Z,ZHANG Y,GAO H T,et al. ContentVec: An Improved Self-supervised Speech Representation by Disentangling Speakers [J/OL].arXiv:2204.09224 [cs.SD].(2022-04-20).https://arxiv.org/abs/2204.09224.
- [5] KONG J,KIM J,BAE J,et al. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis [J/OL].arXiv:2010.05646 [cs.SD].(2020-10-12).https://arxiv.org/abs/2010.05646.
- [6] KIM J,KONG J,SON J,et al. Conditional VARIATIONAL Autoencoder with ADVERSARIAL Learning for End-To-End Text-To-Speech [J/OL].arXiv:2106.06103 [cs.SD].(2021-06-11).https://arxiv.org/abs/2106.06103.
- [7] JIA Y,ZHANG Y,WEISS R J,et al. Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis [J/OL].arXiv:1806.04558 [cs.CL].(2018-06-12).https://arxiv.org/abs/1806.04558.
- [8] HSU W-N,BOLTE B,TSAI Y-H H,et al. HuBERT: Self-supervised Speech Representation Learning by Masked Prediction of Hidden Units [J]IEEE/ACM Transactions on Audio, Speech, and Language Processing,2021,29:3451-3460.
- [9] KINGMA D P,WELLING M. Auto-Encoding Variational Bayes[J/OL].arXiv:1312.6114 [stat.ML].(2013-12-20).https://arxiv.org/abs/1312.6114.
- [10] CHEN M J,TAN X,REN Y,et al. MultiSpeech: Multi-speaker Text to Speech with Transformer [J/OL].arXiv:2006.04664 [eess.AS].(2020-06-08).https://arxiv.org/abs/2006.04664.
- [11] BERNARD M. Phonemizer [EB/OL].[2024-06-26].https://github.com/bootphon/phonemizer.
- [12] VEAUX C,YAMAGISHI J,MACDONALD K. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit [EB/OL].(2017-04-04)[2024-07-11].https://datashare.ed.ac.uk/handle/10283/2651.
- [13] SONG H,CHEN S Y,CHEN Z,et al. Exploring WavLM on Speech Enhancement [C]//2022 IEEE Spoken Language Technology Workshop (SLT).Doha:IEEE,2023:451-457.
- [14] BIŃKOWSKI M,DONAHUE J,DIELEMAN S,et al. High Fidelity Speech Synthesis with Adversarial Networks [J/OL].arXiv:1909.11646 [cs.SD].(2019-09-25).https://arxiv.org/abs/1909.11646.
- [15] WANG X Y,BAI Y P. The Global Minmax K-means Algorithm [J].SpringerPlus,2016,5(1):1665.
- [16] WAN L,WANG Q,PAPIR A,et al. Generalized End-to-End Loss for Speaker Verification [C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).Calgary:IEEE,2018:4879-4883.
- [17] YANG G,YANG S,LIU K,et al. Multi-band Melgan: Faster Waveform Generation for High-quality Text-To-Speech[C]//2021 IEEE Spoken Language Technology Workshop (SLT).Shenzhen:IEEE,2021:492-498.
- [18] KINGMA D P,BA J. Adam: A Method for Stochastic Optimization [J/OL].arXiv:1412.6980 [cs.LG].(2014-12-22).https://arxiv.org/abs/1412.6980.
- [19] TRAORÉ C,PAUWELS E. Sequential Convergence of AdaGrad Algorithm for Smooth Convex Optimization [J].Operations Research Letters,2021,49(4):452-458.
- [20] LI W W,LIU L,ZHANG J X. AdaRW Training Optimization Algorithm for Deep Learning Model of Marine Target Detection Based on SAR [J].International Journal of Remote Sensing,2021,43(1):120-131.
About the authors
Zhou Keru (2004-), female, Han ethnicity, native of Shaoxing, Zhejiang Province, undergraduate student, research direction: Computer Science and Technology; Corresponding author: Jin Wei (1989-), male, Han ethnicity, from Ningbo, Zhejiang, lecturer, PhD, research direction: artificial intelligence.