mirror of
https://github.com/storytold/vits-finetuning.git
synced 2026-10-13 23:53:25 +00:00
ZDisket
70 lines
4.3 KiB
Markdown
70 lines
4.3 KiB
Markdown
# VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
|
|
|
|
### Jaehyeon Kim, Jungil Kong, and Juhee Son
|
|
|
|
In our recent [paper](https://arxiv.org/abs/2106.06103), we propose VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech.
|
|
|
|
Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.
|
|
|
|
Visit our [demo](https://jaywalnut310.github.io/vits-demo/index.html) for audio samples.
|
|
|
|
We also provide the [pretrained models](https://drive.google.com/drive/folders/1ksarh-cJf3F5eKJjLVWY0X1j1qsQqiS2?usp=sharing).
|
|
|
|
** Update note: Thanks to [Rishikesh (ऋषिकेश)](https://github.com/jaywalnut310/vits/issues/1), our interactive TTS demo is now available on [Colab Notebook](https://colab.research.google.com/drive/1CO61pZizDj7en71NQG_aqqKdGaA_SaBf?usp=sharing).
|
|
|
|
<table style="width:100%">
|
|
<tr>
|
|
<th>VITS at training</th>
|
|
<th>VITS at inference</th>
|
|
</tr>
|
|
<tr>
|
|
<td><img src="resources/fig_1a.png" alt="VITS at training" height="400"></td>
|
|
<td><img src="resources/fig_1b.png" alt="VITS at inference" height="400"></td>
|
|
</tr>
|
|
</table>
|
|
|
|
|
|
## Pre-requisites
|
|
0. Python >= 3.6
|
|
0. Clone this repository
|
|
0. Install python requirements. Please refer [requirements.txt](requirements.txt)
|
|
1. You may need to install espeak first: `apt-get install espeak`
|
|
0. Download datasets
|
|
1. Download and extract the LJ Speech dataset, then rename or create a link to the dataset folder: `ln -s /path/to/LJSpeech-1.1/wavs DUMMY1`
|
|
1. For mult-speaker setting, download and extract the VCTK dataset, and downsample wav files to 22050 Hz. Then rename or create a link to the dataset folder: `ln -s /path/to/VCTK-Corpus/downsampled_wavs DUMMY2`
|
|
0. Build Monotonic Alignment Search and run preprocessing if you use your own datasets.
|
|
```sh
|
|
# Cython-version Monotonoic Alignment Search
|
|
cd monotonic_align
|
|
python setup.py build_ext --inplace
|
|
|
|
# Preprocessing (g2p) for your own datasets. Preprocessed phonemes for LJ Speech and VCTK have been already provided.
|
|
# python preprocess.py --text_index 1 --filelists filelists/ljs_audio_text_train_filelist.txt filelists/ljs_audio_text_val_filelist.txt filelists/ljs_audio_text_test_filelist.txt
|
|
# python preprocess.py --text_index 2 --filelists filelists/vctk_audio_sid_text_train_filelist.txt filelists/vctk_audio_sid_text_val_filelist.txt filelists/vctk_audio_sid_text_test_filelist.txt
|
|
```
|
|
|
|
|
|
## Training Exmaple
|
|
```sh
|
|
# LJ Speech
|
|
python train.py -c configs/ljs_base.json -m ljs_base
|
|
|
|
# VCTK
|
|
python train_ms.py -c configs/vctk_base.json -m vctk_base
|
|
```
|
|
## Exporting to TorchScript
|
|
Device can be either cpu or cuda. They must match at inference time, cpu-exported model on CPU and CUDA-exported model on GPU.
|
|
```sh
|
|
python export_ts.py --checkpoint vits.pth --device cpu --config ./configs/ljs_li44_tmbert_nmp_s1.json --out-path my_tsvits.pt --test-string "I like pizza"
|
|
```
|
|
|
|
## Inference with TorchScript
|
|
Device can be either cpu or cuda. Again, this must match the device you exported to TS with, cpu-exported model on CPU and CUDA-exported model on GPU.
|
|
```sh
|
|
python infer_ts.py --out-path res --device cpu --config ./configs/ljs_li44_tmbert_nmp_s1.json --checkpoint my_tsvits.pt --text "I like hamburgers"
|
|
```
|
|
Check `tsvitsfe.py` for the simple inference frontend class. The script above just calls it and saves to a wave file
|
|
|
|
## Inference Example
|
|
See [inference.ipynb](inference.ipynb)
|