Add Tacotron2-CAC

- Tacotron2 based on https://github.com/keonlee9420/Comprehensive-Tacotron2
- Removed DeepSpeaker and Tensorflow dependency
- Added ARPA function
- Replaced guided attention loss with Convolutional Attention Consistency

neo-tts/tacotron2-cac/.gitignore

@@ -0,0 +1,120 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+__pycache__
+.vscode
+.DS_Store
+
+# MFA
+montreal-forced-aligner/
+
+# data, checkpoint, and models
+raw_data/
+output/
+*.npy
+TextGrid/
+hifigan/*.pth.tar
+*.out
+deepspeaker/pretrained_models/*

neo-tts/tacotron2-cac/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 Keon Lee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

neo-tts/tacotron2-cac/README.md

@@ -0,0 +1,119 @@
+# Comprehensive Tacotron2 - PyTorch Implementation
+
+PyTorch Implementation of Google's [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://arxiv.org/abs/1712.05884). Unlike many previous implementations, this is kind of a **`Comprehensive Tacotron2`** where the model supports both single-, multi-speaker TTS and several techniques such as reduction factor to enforce the robustness of the decoder alignment. The model can learn alignment only in `5k`. We also implement Convolutional Attention Consistency, using the Mixer-TTS alignment method to replace guided attention loss.
+<p align="center">
+    <img src="img/model.png" width="80%">
+</p>
+
+The validation logs up to 70K of synthesized mel and alignment are shown below (LJSpeech_val_LJ038-0050 and VCTK_val_p323_008 from top to bottom).
+
+<p align="center">
+    <img src="./img/LJSpeech_val_LJ038-0050.gif" width="80%">
+</p>
+<p align="center">
+    <img src="./img/VCTK_val_p323_008.gif" width="80%">
+</p>
+
+# Quickstart
+
+## Dependencies
+You can install the Python dependencies with
+```
+pip3 install -r requirements.txt
+```
+
+## Inference
+
+You have to download the [pretrained models](https://drive.google.com/drive/folders/1fGhmSVnUzdS_8i_-7cX6pfaSIQ0oiYBM?usp=sharing) and put them in `output/ckpt/LJSpeech/` or `output/ckpt/VCTK/`.
+
+For a **single-speaker TTS**, run
+```
+python3 synthesize.py --text "YOUR_DESIRED_TEXT" --restore_step RESTORE_STEP --mode single --dataset LJSpeech
+```
+
+For a **multi-speaker TTS**, run
+```
+python3 synthesize.py --text "YOUR_DESIRED_TEXT" --speaker_id SPEAKER_ID --restore_step RESTORE_STEP --mode single --dataset VCTK
+```
+
+The generated utterances will be put in `output/result/`.
+
+
+## Batch Inference
+Batch inference is also supported, try
+
+```
+python3 synthesize.py --source preprocessed_data/LJSpeech/val.txt --restore_step RESTORE_STEP --mode batch --dataset LJSpeech
+```
+to synthesize all utterances in `preprocessed_data/LJSpeech/val.txt`. You can replace `LJSpeech` with `VCTK`. Note that only **1 batch size** is supported currently due to the autoregressive model architecture.
+
+# Training
+
+## Datasets
+
+The supported datasets are
+
+- [LJSpeech](https://keithito.com/LJ-Speech-Dataset/): a **single-speaker TTS** English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.
+- [VCTK](https://datashare.ed.ac.uk/handle/10283/3443): The CSTR VCTK Corpus includes speech data uttered by 110 English speakers (**multi-speaker TTS** ) with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.
+- Any of both **single-speaker TTS** dataset (e.g., [Blizzard Challenge 2013](https://www.synsig.org/index.php/Blizzard_Challenge_2013)) and **multi-speaker TTS** dataset (e.g., [LibriTTS](https://openslr.org/60/)) can be added following LJSpeech and VCTK, respectively.
+
+## Preprocessing
+
+- For a **multi-speaker TTS** with external speaker embedder, download [ResCNN Softmax+Triplet pretrained model](https://drive.google.com/file/d/1F9NvdrarWZNktdX9KlRYWWHDwRkip_aP) of [philipperemy's DeepSpeaker](https://github.com/philipperemy/deep-speaker) for the speaker embedding and locate it in `./deepspeaker/pretrained_models/`.
+
+- Run the preprocessing script
+  ```
+  python3 preprocess.py --dataset DATASET
+  ```
+
+## Training
+
+Train your model with
+```
+python3 train.py --dataset DATASET
+```
+
+# TensorBoard
+
+Use
+```
+tensorboard --logdir output/log
+```
+
+to serve TensorBoard on your localhost.
+<!-- The loss curves, synthesized mel-spectrograms, and audios are shown.
+
+![](./img/tensorboard_loss.png)
+![](./img/tensorboard_spec.png)
+![](./img/tensorboard_audio.png) -->
+
+# Implementation Issues
+
+- Support `n_frames_per_step>1` mode (which is not supported by [NVIDIA's tacotron2](https://github.com/NVIDIA/tacotron2)). This is the key factor to get the robustness of the decoder alignment as described in the paper. Also, it reduces the training & inference time by the factor time.
+- The current implementation provides pre-trained model of `n_frames_per_step==2`, but it should also work for any number greater than 2.
+- Add ZDisket's implementation of Convolutional Attention Consistency, helping model learn robust alignment very quickly.
+- Speaker embedding: training speaker embedder from scratch 
+
+- For the vocoder, the current implementation supports **HiFi-GAN** and **MelGAN**, which are much better than **WaveNet**.
+- Currently, `fp16_run` mode is not supported.
+
+# Citation
+
+```
+@misc{lee2021comprehensive-tacotron2,
+  author = {Lee, Keon},
+  title = {Comprehensive-Tacotron2},
+  year = {2021},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/keonlee9420/Comprehensive-Tacotron2}}
+}
+```
+
+# References
+- [NVIDIA's tacotron2](https://github.com/NVIDIA/tacotron2)
+- [keonlee9420's tacotron2_MMI](https://github.com/keonlee9420/tacotron2_MMI)
+- [keonlee9420's STYLER](https://github.com/keonlee9420/STYLER)
+- [philipperemy's DeepSpeaker](https://github.com/philipperemy/deep-speaker)
+- [espnet's diagonal guided attention loss](https://github.com/espnet/espnet/blob/e962a3c609ad535cd7fb9649f9f9e9e0a2a27291/espnet/nets/pytorch_backend/e2e_tts_tacotron2.py#L25)
+- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://arxiv.org/abs/1910.10838)

neo-tts/tacotron2-cac/audio/__init__.py

@@ -0,0 +1,3 @@
+import audio.tools
+import audio.stft
+import audio.audio_processing

neo-tts/tacotron2-cac/audio/audio_processing.py

@@ -0,0 +1,100 @@
+import torch
+import numpy as np
+import librosa.util as librosa_util
+from scipy.signal import get_window
+
+
+def window_sumsquare(
+    window,
+    n_frames,
+    hop_length,
+    win_length,
+    n_fft,
+    dtype=np.float32,
+    norm=None,
+):
+    """
+    # from librosa 0.6
+    Compute the sum-square envelope of a window function at a given hop length.
+
+    This is used to estimate modulation effects induced by windowing
+    observations in short-time fourier transforms.
+
+    Parameters
+    ----------
+    window : string, tuple, number, callable, or list-like
+        Window specification, as in `get_window`
+
+    n_frames : int > 0
+        The number of analysis frames
+
+    hop_length : int > 0
+        The number of samples to advance between frames
+
+    win_length : [optional]
+        The length of the window function.  By default, this matches `n_fft`.
+
+    n_fft : int > 0
+        The length of each analysis frame.
+
+    dtype : np.dtype
+        The data type of the output
+
+    Returns
+    -------
+    wss : np.ndarray, shape=`(n_fft + hop_length * (n_frames - 1))`
+        The sum-squared envelope of the window function
+    """
+    if win_length is None:
+        win_length = n_fft
+
+    n = n_fft + hop_length * (n_frames - 1)
+    x = np.zeros(n, dtype=dtype)
+
+    # Compute the squared window at the desired length
+    win_sq = get_window(window, win_length, fftbins=True)
+    win_sq = librosa_util.normalize(win_sq, norm=norm) ** 2
+    win_sq = librosa_util.pad_center(win_sq, n_fft)
+
+    # Fill the envelope
+    for i in range(n_frames):
+        sample = i * hop_length
+        x[sample : min(n, sample + n_fft)] += win_sq[: max(0, min(n_fft, n - sample))]
+    return x
+
+
+def griffin_lim(magnitudes, stft_fn, n_iters=30):
+    """
+    PARAMS
+    ------
+    magnitudes: spectrogram magnitudes
+    stft_fn: STFT class with transform (STFT) and inverse (ISTFT) methods
+    """
+
+    angles = np.angle(np.exp(2j * np.pi * np.random.rand(*magnitudes.size())))
+    angles = angles.astype(np.float32)
+    angles = torch.autograd.Variable(torch.from_numpy(angles))
+    signal = stft_fn.inverse(magnitudes, angles).squeeze(1)
+
+    for i in range(n_iters):
+        _, angles = stft_fn.transform(signal)
+        signal = stft_fn.inverse(magnitudes, angles).squeeze(1)
+    return signal
+
+
+def dynamic_range_compression(x, C=1, clip_val=1e-5):
+    """
+    PARAMS
+    ------
+    C: compression factor
+    """
+    return torch.log(torch.clamp(x, min=clip_val) * C)
+
+
+def dynamic_range_decompression(x, C=1):
+    """
+    PARAMS
+    ------
+    C: compression factor used to compress
+    """
+    return torch.exp(x) / C

neo-tts/tacotron2-cac/audio/stft.py

@@ -0,0 +1,156 @@
+import torch
+import torch.nn.functional as F
+from torch.autograd import Variable
+import numpy as np
+from scipy.signal import get_window
+from librosa.util import pad_center, tiny
+from librosa.filters import mel as librosa_mel_fn
+
+from audio.audio_processing import (
+    dynamic_range_compression,
+    dynamic_range_decompression,
+    window_sumsquare,
+)
+
+
+class STFT(torch.nn.Module):
+    """adapted from Prem Seetharaman's https://github.com/pseeth/pytorch-stft"""
+    def __init__(self, filter_length=800, hop_length=200, win_length=800,
+                 window='hann'):
+        super(STFT, self).__init__()
+        self.filter_length = filter_length
+        self.hop_length = hop_length
+        self.win_length = win_length
+        self.window = window
+        self.forward_transform = None
+        scale = self.filter_length / self.hop_length
+        fourier_basis = np.fft.fft(np.eye(self.filter_length))
+
+        cutoff = int((self.filter_length / 2 + 1))
+        fourier_basis = np.vstack([np.real(fourier_basis[:cutoff, :]),
+                                   np.imag(fourier_basis[:cutoff, :])])
+
+        forward_basis = torch.FloatTensor(fourier_basis[:, None, :])
+        inverse_basis = torch.FloatTensor(
+            np.linalg.pinv(scale * fourier_basis).T[:, None, :])
+
+        if window is not None:
+            assert(filter_length >= win_length)
+            # get window and zero center pad it to filter_length
+            fft_window = get_window(window, win_length, fftbins=True)
+            fft_window = pad_center(fft_window, filter_length)
+            fft_window = torch.from_numpy(fft_window).float()
+
+            # window the bases
+            forward_basis *= fft_window
+            inverse_basis *= fft_window
+
+        self.register_buffer('forward_basis', forward_basis.float())
+        self.register_buffer('inverse_basis', inverse_basis.float())
+
+    def transform(self, input_data):
+        num_batches = input_data.size(0)
+        num_samples = input_data.size(1)
+
+        self.num_samples = num_samples
+
+        # similar to librosa, reflect-pad the input
+        input_data = input_data.view(num_batches, 1, num_samples)
+        input_data = F.pad(
+            input_data.unsqueeze(1),
+            (int(self.filter_length / 2), int(self.filter_length / 2), 0, 0),
+            mode='reflect')
+        input_data = input_data.squeeze(1)
+
+        forward_transform = F.conv1d(
+            input_data,
+            Variable(self.forward_basis, requires_grad=False),
+            stride=self.hop_length,
+            padding=0)
+
+        cutoff = int((self.filter_length / 2) + 1)
+        real_part = forward_transform[:, :cutoff, :]
+        imag_part = forward_transform[:, cutoff:, :]
+
+        magnitude = torch.sqrt(real_part**2 + imag_part**2)
+        phase = torch.autograd.Variable(
+            torch.atan2(imag_part.data, real_part.data))
+
+        return magnitude, phase
+
+    def inverse(self, magnitude, phase):
+        recombine_magnitude_phase = torch.cat(
+            [magnitude*torch.cos(phase), magnitude*torch.sin(phase)], dim=1)
+
+        inverse_transform = F.conv_transpose1d(
+            recombine_magnitude_phase,
+            Variable(self.inverse_basis, requires_grad=False),
+            stride=self.hop_length,
+            padding=0)
+
+        if self.window is not None:
+            window_sum = window_sumsquare(
+                self.window, magnitude.size(-1), hop_length=self.hop_length,
+                win_length=self.win_length, n_fft=self.filter_length,
+                dtype=np.float32)
+            # remove modulation effects
+            approx_nonzero_indices = torch.from_numpy(
+                np.where(window_sum > tiny(window_sum))[0])
+            window_sum = torch.autograd.Variable(
+                torch.from_numpy(window_sum), requires_grad=False)
+            window_sum = window_sum.cuda() if magnitude.is_cuda else window_sum
+            inverse_transform[:, :, approx_nonzero_indices] /= window_sum[approx_nonzero_indices]
+
+            # scale by hop ratio
+            inverse_transform *= float(self.filter_length) / self.hop_length
+
+        inverse_transform = inverse_transform[:, :, int(self.filter_length/2):]
+        inverse_transform = inverse_transform[:, :, :-int(self.filter_length/2):]
+
+        return inverse_transform
+
+    def forward(self, input_data):
+        self.magnitude, self.phase = self.transform(input_data)
+        reconstruction = self.inverse(self.magnitude, self.phase)
+        return reconstruction
+
+
+class TacotronSTFT(torch.nn.Module):
+    def __init__(self, filter_length=1024, hop_length=256, win_length=1024,
+                 n_mel_channels=80, sampling_rate=22050, mel_fmin=0.0,
+                 mel_fmax=8000.0):
+        super(TacotronSTFT, self).__init__()
+        self.n_mel_channels = n_mel_channels
+        self.sampling_rate = sampling_rate
+        self.stft_fn = STFT(filter_length, hop_length, win_length)
+        mel_basis = librosa_mel_fn(
+            sampling_rate, filter_length, n_mel_channels, mel_fmin, mel_fmax)
+        mel_basis = torch.from_numpy(mel_basis).float()
+        self.register_buffer('mel_basis', mel_basis)
+
+    def spectral_normalize(self, magnitudes):
+        output = dynamic_range_compression(magnitudes)
+        return output
+
+    def spectral_de_normalize(self, magnitudes):
+        output = dynamic_range_decompression(magnitudes)
+        return output
+
+    def mel_spectrogram(self, y):
+        """Computes mel-spectrograms from a batch of waves
+        PARAMS
+        ------
+        y: Variable(torch.FloatTensor) with shape (B, T) in range [-1, 1]
+
+        RETURNS
+        -------
+        mel_output: torch.FloatTensor of shape (B, n_mel_channels, T)
+        """
+        assert(torch.min(y.data) >= -1)
+        assert(torch.max(y.data) <= 1)
+
+        magnitudes, phases = self.stft_fn.transform(y)
+        magnitudes = magnitudes.data
+        mel_output = torch.matmul(self.mel_basis, magnitudes)
+        mel_output = self.spectral_normalize(mel_output)
+        return mel_output

neo-tts/tacotron2-cac/audio/tools.py

@@ -0,0 +1,41 @@
+import torch
+import numpy as np
+from scipy.io.wavfile import write
+
+from audio.audio_processing import griffin_lim
+
+
+def get_mel_from_wav(audio, _stft):
+    audio = torch.clamp(torch.FloatTensor(audio).unsqueeze(0), -1, 1)
+    audio = torch.autograd.Variable(audio, requires_grad=False)
+    melspec = _stft.mel_spectrogram(audio)
+    melspec = torch.squeeze(melspec, 0).numpy().astype(np.float32)
+
+    return melspec
+
+
+def inv_mel_spec(mel, out_filename, _stft, griffin_iters=60):
+    mel = torch.stack([mel])
+    mel_decompress = _stft.spectral_de_normalize(mel)
+    mel_decompress = mel_decompress.transpose(1, 2).data.cpu()
+    spec_from_mel_scaling = 1000
+    spec_from_mel = torch.mm(mel_decompress[0], _stft.mel_basis)
+    spec_from_mel = spec_from_mel.transpose(0, 1).unsqueeze(0)
+    spec_from_mel = spec_from_mel * spec_from_mel_scaling
+
+    audio = griffin_lim(
+        torch.autograd.Variable(spec_from_mel[:, :, :-1]), _stft._stft_fn, griffin_iters
+    )
+
+    audio = audio.squeeze()
+    audio = audio.cpu().numpy()
+    audio_path = out_filename
+    write(audio_path, _stft.sampling_rate, audio)
+
+
+def mel_normalize(mel, m_min, m_max):
+    return 2 * (mel - m_min)/(m_max-m_min) - 1
+
+
+def mel_denormalize(mel_norm, m_min, m_max):
+    return (m_max - m_min) * ((mel_norm)+1)/2 + m_min

neo-tts/tacotron2-cac/config/LJSpeech/model.yaml

@@ -0,0 +1,39 @@
+encoder:
+  encoder_kernel_size: 5
+  encoder_n_convolutions: 3
+  encoder_embedding_dim: 512
+  symbols_embedding_dim: 512
+decoder:
+  n_frames_per_step: 1 # any of reduction factor is supported
+  decoder_rnn_dim: 1024
+  prenet_dim: 256
+  max_decoder_steps: 1000
+  gate_threshold: 0.5
+  p_attention_dropout: 0.1
+  p_decoder_dropout: 0.1
+attention:
+  attention_rnn_dim: 1024
+  attention_dim: 128
+location_layer:
+  attention_location_n_filters: 32
+  attention_location_kernel_size: 31
+postnet:
+  postnet_embedding_dim: 512
+  postnet_kernel_size: 5
+  postnet_n_convolutions: 5
+
+# gst:
+#   use_gst: False
+#   conv_filters: [32, 32, 64, 64, 128, 128]
+#   gru_hidden: 128
+#   token_size: 128
+#   n_style_token: 10
+#   attn_head: 4
+
+multi_speaker: False
+
+max_seq_len: 1000
+
+vocoder:
+  model: "HiFi-GAN" # support 'HiFi-GAN', 'MelGAN'
+  speaker: "LJSpeech" # support  'LJSpeech', 'universal'

neo-tts/tacotron2-cac/config/LJSpeech/preprocess.yaml

@@ -0,0 +1,28 @@
+dataset: "LJSpeech"
+
+path:
+  corpus_path: "./LJSpeech-1.1"
+  preprocessed_path: "./preprocessed_data/LJSpeech"
+
+preprocessing:
+  sanity_check: False # if Ture, then preprocess a single data and quit
+  val_size: 512
+  text:
+    text_cleaners: ["english_cleaners"]
+    language: "en"
+  audio:
+    sampling_rate: 22050
+    max_wav_value: 32768.0
+    skip_len: 14848
+    trim_top_db: 23
+    silence_audio_size: 5
+    pre_emphasis: False
+  stft:
+    filter_length: 1024
+    hop_length: 256
+    win_length: 1024
+  mel:
+    n_mel_channels: 80
+    mel_fmin: 0
+    mel_fmax: 8000 # please set to 8000 for HiFi-GAN vocoder, set to null for MelGAN vocoder
+    normalize: False

neo-tts/tacotron2-cac/config/LJSpeech/train.yaml

@@ -0,0 +1,23 @@
+path:
+  ckpt_path: "./output/ckpt/LJSpeech"
+  log_path: "./output/log/LJSpeech"
+  result_path: "./output/result/LJSpeech"
+optimizer:
+  mask_padding: True # set model's padded outputs to padded values
+  guided_attn: True # apply diagonal guided attention loss
+  guided_sigma: 0.4
+  guided_lambda: 1.0
+  fp16_run: False
+  batch_size: 64
+  betas: [0.9, 0.999]
+  eps: 0.00000001
+  weight_decay: 0.000001
+  grad_clip_thresh: 1.0
+  grad_acc_step: 1
+  init_lr: 0.001
+step:
+  total_step: 900000
+  log_step: 100
+  synth_step: 500
+  val_step: 1000
+  save_step: 5000

neo-tts/tacotron2-cac/config/SingleSpk/model.yaml

@@ -0,0 +1,39 @@
+encoder:
+  encoder_kernel_size: 5
+  encoder_n_convolutions: 3
+  encoder_embedding_dim: 512
+  symbols_embedding_dim: 512
+decoder:
+  n_frames_per_step: 1 # any of reduction factor is supported
+  decoder_rnn_dim: 1024
+  prenet_dim: 256
+  max_decoder_steps: 1000
+  gate_threshold: 0.5
+  p_attention_dropout: 0.1
+  p_decoder_dropout: 0.1
+attention:
+  attention_rnn_dim: 1024
+  attention_dim: 128
+location_layer:
+  attention_location_n_filters: 32
+  attention_location_kernel_size: 31
+postnet:
+  postnet_embedding_dim: 512
+  postnet_kernel_size: 5
+  postnet_n_convolutions: 5
+
+# gst:
+#   use_gst: False
+#   conv_filters: [32, 32, 64, 64, 128, 128]
+#   gru_hidden: 128
+#   token_size: 128
+#   n_style_token: 10
+#   attn_head: 4
+
+multi_speaker: False
+
+max_seq_len: 1000
+
+vocoder:
+  model: "HiFi-GAN" # support 'HiFi-GAN', 'MelGAN'
+  speaker: "LJSpeech" # support  'LJSpeech', 'universal'

neo-tts/tacotron2-cac/config/SingleSpk/preprocess.yaml

@@ -0,0 +1,28 @@
+dataset: "LJSpeech"
+
+path:
+  corpus_path: "./datasets"
+  preprocessed_path: "./preprocessed_data/SingleSpk"
+
+preprocessing:
+  sanity_check: False # if Ture, then preprocess a single data and quit
+  val_size: 512
+  text:
+    text_cleaners: ["vbasic_cleaners"]
+    language: "en"
+  audio:
+    sampling_rate: 44100
+    max_wav_value: 32768.0
+    skip_len: 14848
+    trim_top_db: 23
+    silence_audio_size: 5
+    pre_emphasis: False
+  stft:
+    filter_length: 2048
+    hop_length: 512
+    win_length: 2048
+  mel:
+    n_mel_channels: 80
+    mel_fmin: 20
+    mel_fmax: 11025 # please set to 8000 for HiFi-GAN vocoder, set to null for MelGAN vocoder
+    normalize: False

neo-tts/tacotron2-cac/config/SingleSpk/train.yaml

@@ -0,0 +1,23 @@
+path:
+  ckpt_path: "./output/ckpt/SingleSpk"
+  log_path: "./output/log/SingleSpk"
+  result_path: "./output/result/SingleSpk"
+optimizer:
+  mask_padding: True # set model's padded outputs to padded values
+  guided_attn: True # apply diagonal guided attention loss
+  guided_sigma: 0.4
+  guided_lambda: 1.0
+  fp16_run: False
+  batch_size: 64
+  betas: [0.9, 0.999]
+  eps: 0.00000001
+  weight_decay: 0.000001
+  grad_clip_thresh: 1.0
+  grad_acc_step: 1
+  init_lr: 0.001
+step:
+  total_step: 900000
+  log_step: 100
+  synth_step: 500
+  val_step: 1000
+  save_step: 5000

neo-tts/tacotron2-cac/config/SingleSpk22/model.yaml

@@ -0,0 +1,39 @@
+encoder:
+  encoder_kernel_size: 5
+  encoder_n_convolutions: 3
+  encoder_embedding_dim: 512
+  symbols_embedding_dim: 512
+decoder:
+  n_frames_per_step: 1 # any of reduction factor is supported
+  decoder_rnn_dim: 1024
+  prenet_dim: 256
+  max_decoder_steps: 1000
+  gate_threshold: 0.5
+  p_attention_dropout: 0.1
+  p_decoder_dropout: 0.1
+attention:
+  attention_rnn_dim: 1024
+  attention_dim: 128
+location_layer:
+  attention_location_n_filters: 32
+  attention_location_kernel_size: 31
+postnet:
+  postnet_embedding_dim: 512
+  postnet_kernel_size: 5
+  postnet_n_convolutions: 5
+
+# gst:
+#   use_gst: False
+#   conv_filters: [32, 32, 64, 64, 128, 128]
+#   gru_hidden: 128
+#   token_size: 128
+#   n_style_token: 10
+#   attn_head: 4
+
+multi_speaker: False
+
+max_seq_len: 1000
+
+vocoder:
+  model: "HiFi-GAN" # support 'HiFi-GAN', 'MelGAN'
+  speaker: "universal" # support  'LJSpeech', 'universal'

neo-tts/tacotron2-cac/config/SingleSpk22/preprocess.yaml

@@ -0,0 +1,28 @@
+dataset: "LJSpeech"
+
+path:
+  corpus_path: "./datasets"
+  preprocessed_path: "./preprocessed_data/SingleSpk"
+
+preprocessing:
+  sanity_check: False # if Ture, then preprocess a single data and quit
+  val_size: 512
+  text:
+    text_cleaners: ["vbasic_cleaners"]
+    language: "en"
+  audio:
+    sampling_rate: 22050
+    max_wav_value: 32768.0
+    skip_len: 14848
+    trim_top_db: 23
+    silence_audio_size: 5
+    pre_emphasis: False
+  stft:
+    filter_length: 1024
+    hop_length: 256
+    win_length: 1024
+  mel:
+    n_mel_channels: 80
+    mel_fmin: 0
+    mel_fmax: 8000 # please set to 8000 for HiFi-GAN vocoder, set to null for MelGAN vocoder
+    normalize: False

neo-tts/tacotron2-cac/config/SingleSpk22/train.yaml

@@ -0,0 +1,23 @@
+path:
+  ckpt_path: "./output/ckpt/SingleSpk"
+  log_path: "./output/log/SingleSpk"
+  result_path: "./output/result/SingleSpk"
+optimizer:
+  mask_padding: True # set model's padded outputs to padded values
+  guided_attn: True # apply diagonal guided attention loss
+  guided_sigma: 0.4
+  guided_lambda: 1.0
+  fp16_run: False
+  batch_size: 48
+  betas: [0.9, 0.999]
+  eps: 0.00000001
+  weight_decay: 0.000001
+  grad_clip_thresh: 1.0
+  grad_acc_step: 1
+  init_lr: 0.001
+step:
+  total_step: 900000
+  log_step: 100
+  synth_step: 500
+  val_step: 1000
+  save_step: 5000

neo-tts/tacotron2-cac/config/VCTK/model.yaml

@@ -0,0 +1,40 @@
+encoder:
+  encoder_kernel_size: 5
+  encoder_n_convolutions: 3
+  speaker_embeddint_dim: 512
+  encoder_embedding_dim: 512
+  symbols_embedding_dim: 512
+decoder:
+  n_frames_per_step: 2 # any of reduction factor is supported
+  decoder_rnn_dim: 1024
+  prenet_dim: 256
+  max_decoder_steps: 1000
+  gate_threshold: 0.5
+  p_attention_dropout: 0.1
+  p_decoder_dropout: 0.1
+attention:
+  attention_rnn_dim: 1024
+  attention_dim: 128
+location_layer:
+  attention_location_n_filters: 32
+  attention_location_kernel_size: 31
+postnet:
+  postnet_embedding_dim: 512
+  postnet_kernel_size: 5
+  postnet_n_convolutions: 5
+
+# gst:
+#   use_gst: False
+#   conv_filters: [32, 32, 64, 64, 128, 128]
+#   gru_hidden: 128
+#   token_size: 128
+#   n_style_token: 10
+#   attn_head: 4
+
+multi_speaker: True
+
+max_seq_len: 1400 # max sequence length of VCTK is 1304
+
+vocoder:
+  model: "HiFi-GAN" # support 'HiFi-GAN', 'MelGAN'
+  speaker: "universal" # support  'LJSpeech', 'universal'

neo-tts/tacotron2-cac/config/VCTK/preprocess.yaml

@@ -0,0 +1,32 @@
+dataset: "VCTK"
+
+path:
+  corpus_path: "/mnt/nfs2/speech-datasets/en/VCTK-Corpus-0.92"
+  wav_tag: "mic1"
+  wav_dir: "wav48_silence_trimmed"
+  txt_dir: "txt"
+  preprocessed_path: "./preprocessed_data/VCTK"
+
+preprocessing:
+  sanity_check: False # if Ture, then preprocess a single data and quit
+  speaker_embedder: "DeepSpeaker" # support 'none', 'DeepSpeaker'
+  val_size: 512
+  text:
+    text_cleaners: ["english_cleaners"]
+    language: "en"
+  audio:
+    sampling_rate: 22050
+    max_wav_value: 32768.0
+    skip_len: 14848
+    trim_top_db: 23
+    silence_audio_size: 5
+    pre_emphasis: False
+  stft:
+    filter_length: 1024
+    hop_length: 256
+    win_length: 1024
+  mel:
+    n_mel_channels: 80
+    mel_fmin: 0
+    mel_fmax: 8000 # please set to 8000 for HiFi-GAN vocoder, set to null for MelGAN vocoder
+    normalize: False

neo-tts/tacotron2-cac/config/VCTK/train.yaml

@@ -0,0 +1,23 @@
+path:
+  ckpt_path: "./output/ckpt/VCTK"
+  log_path: "./output/log/VCTK"
+  result_path: "./output/result/VCTK"
+optimizer:
+  mask_padding: True # set model's padded outputs to padded values
+  guided_attn: True # apply diagonal guided attention loss
+  guided_sigma: 0.4
+  guided_lambda: 1.0
+  fp16_run: False
+  batch_size: 64
+  betas: [0.9, 0.999]
+  eps: 0.00000001
+  weight_decay: 0.000001
+  grad_clip_thresh: 1.0
+  grad_acc_step: 1
+  init_lr: 0.001
+step:
+  total_step: 900000
+  log_step: 100
+  synth_step: 500
+  val_step: 1000
+  save_step: 5000

neo-tts/tacotron2-cac/dataset.py

@@ -0,0 +1,209 @@
+import json
+import math
+import os
+
+import numpy as np
+from torch.utils.data import Dataset
+
+from utils.tools import pad_1D, pad_2D
+
+
+class Dataset(Dataset):
+    def __init__(
+        self, filename, preprocess_config, model_config, train_config, sort=False, drop_last=False
+    ):
+        self.dataset_name = preprocess_config["dataset"]
+        self.preprocessed_path = preprocess_config["path"]["preprocessed_path"]
+        self.batch_size = train_config["optimizer"]["batch_size"]
+        self.n_frames_per_step = model_config["decoder"]["n_frames_per_step"]
+        self.load_spker_embed = model_config["multi_speaker"] \
+            and preprocess_config["preprocessing"]["speaker_embedder"] != 'none'
+
+        self.basename, self.speaker, self.raw_text = self.process_meta(
+            filename
+        )
+        with open(os.path.join(self.preprocessed_path, "speakers.json")) as f:
+            self.speaker_map = json.load(f)
+        self.sort = sort
+        self.drop_last = drop_last
+
+    def __len__(self):
+        return len(self.raw_text)
+
+    def __getitem__(self, idx):
+        basename = self.basename[idx]
+        speaker = self.speaker[idx]
+        speaker_id = self.speaker_map[speaker]
+        raw_text = self.raw_text[idx]
+        phone_path = os.path.join(
+            self.preprocessed_path,
+            "text",
+            "{}-text-{}.npy".format(speaker, basename),
+        )
+        text = np.load(phone_path)
+        mel_path = os.path.join(
+            self.preprocessed_path,
+            "mel",
+            "{}-mel-{}.npy".format(speaker, basename),
+        )
+        mel = np.load(mel_path)
+        spker_embed = np.load(os.path.join(
+            self.preprocessed_path,
+            "spker_embed",
+            "{}-spker_embed.npy".format(speaker),
+        )) if self.load_spker_embed else None
+
+        sample = {
+            "id": basename,
+            "speaker": speaker_id,
+            "text": text,
+            "raw_text": raw_text,
+            "mel": mel,
+            "spker_embed": spker_embed,
+        }
+
+        return sample
+
+    def process_meta(self, filename):
+        with open(
+            os.path.join(self.preprocessed_path, filename), "r", encoding="utf-8"
+        ) as f:
+            name = []
+            speaker = []
+            text = []
+            for line in f.readlines():
+                n, s, t = line.strip("\n").split("|")
+                name.append(n)
+                speaker.append(s)
+                text.append(t)
+            return name, speaker, text
+
+    def get_gates_from_mel_lens(self, mel_lens, max_target_len):
+        gates = np.zeros([mel_lens.shape[0], max_target_len])
+        for i, mel_len in enumerate(mel_lens):
+            gates[i, mel_len-1:] = 1
+        return gates
+
+    def reprocess(self, data, idxs):
+        ids = [data[idx]["id"] for idx in idxs]
+        speakers = [data[idx]["speaker"] for idx in idxs]
+        texts = [data[idx]["text"] for idx in idxs]
+        raw_texts = [data[idx]["raw_text"] for idx in idxs]
+        mels = [data[idx]["mel"] for idx in idxs]
+        spker_embeds = np.concatenate(np.array([data[idx]["spker_embed"] for idx in idxs]), axis=0) \
+            if self.load_spker_embed else None
+
+        text_lens = np.array([text.shape[0] for text in texts])
+        mel_lens = np.array([mel.shape[0] for mel in mels])
+
+        max_target_len = max(mel_lens)
+        r_len_pad = max_target_len % self.n_frames_per_step
+        if r_len_pad != 0:
+            max_target_len += self.n_frames_per_step - r_len_pad
+            assert max_target_len % self.n_frames_per_step == 0
+
+        speakers = np.array(speakers)
+        texts = pad_1D(texts)
+        mels = pad_2D(mels, max_target_len)
+        gates = self.get_gates_from_mel_lens(mel_lens, max_target_len)
+
+        return (
+            ids,
+            raw_texts,
+            speakers,
+            texts,
+            text_lens,
+            max(text_lens),
+            mels,
+            mel_lens,
+            max_target_len,
+            r_len_pad,
+            gates,
+            spker_embeds,
+        )
+
+    def collate_fn(self, data):
+        data_size = len(data)
+
+        if self.sort:
+            len_arr = np.array([d["text"].shape[0] for d in data])
+            idx_arr = np.argsort(-len_arr)
+        else:
+            idx_arr = np.arange(data_size)
+
+        tail = idx_arr[len(idx_arr) - (len(idx_arr) % self.batch_size) :]
+        idx_arr = idx_arr[: len(idx_arr) - (len(idx_arr) % self.batch_size)]
+        idx_arr = idx_arr.reshape((-1, self.batch_size)).tolist()
+        if not self.drop_last and len(tail) > 0:
+            idx_arr += [tail.tolist()]
+
+        output = list()
+        for idx in idx_arr:
+            output.append(self.reprocess(data, idx))
+
+        return output
+
+
+class TextDataset(Dataset):
+    def __init__(self, filepath, preprocess_config, model_config):
+
+        self.preprocessed_path = preprocess_config["path"]["preprocessed_path"]
+        self.load_spker_embed = model_config["multi_speaker"] \
+            and preprocess_config["preprocessing"]["speaker_embedder"] != 'none'
+
+        self.basename, self.speaker, self.raw_text = self.process_meta(
+            filepath
+        )
+        with open(
+            os.path.join(
+                preprocess_config["path"]["preprocessed_path"], "speakers.json"
+            )
+        ) as f:
+            self.speaker_map = json.load(f)
+
+    def __len__(self):
+        return len(self.raw_text)
+
+    def __getitem__(self, idx):
+        basename = self.basename[idx]
+        speaker = self.speaker[idx]
+        speaker_id = self.speaker_map[speaker]
+        raw_text = self.raw_text[idx]
+        phone_path = os.path.join(
+            self.preprocessed_path,
+            "text",
+            "{}-text-{}.npy".format(speaker, basename),
+        )
+        text = np.load(phone_path)
+        spker_embed = np.load(os.path.join(
+            self.preprocessed_path,
+            "spker_embed",
+            "{}-spker_embed.npy".format(speaker),
+        )) if self.load_spker_embed else None
+
+        return (basename, speaker_id, text, raw_text, spker_embed)
+
+    def process_meta(self, filename):
+        with open(filename, "r", encoding="utf-8") as f:
+            name = []
+            speaker = []
+            text = []
+            for line in f.readlines():
+                n, s, t = line.strip("\n").split("|")
+                name.append(n)
+                speaker.append(s)
+                text.append(t)
+            return name, speaker, text
+
+    def collate_fn(self, data):
+        ids = [d[0] for d in data]
+        speakers = np.array([d[1] for d in data])
+        texts = [d[2] for d in data]
+        raw_texts = [d[3] for d in data]
+        text_lens = np.array([text.shape[0] for text in texts])
+        spker_embeds = np.concatenate(np.array([d[4] for d in data]), axis=0) \
+            if self.load_spker_embed else None
+
+        texts = pad_1D(texts)
+
+        return ids, raw_texts, speakers, texts, text_lens, max(text_lens), spker_embeds