storytold/storyteller-ml
2
0
Fork 0
mirror of https://github.com/storytold/storyteller-ml.git synced 2026-10-13 23:53:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Introduce code from Rust monorepo. This is a mess

Browse Source
This commit is contained in:
Brandon Thomas
2022-02-28 05:43:54 -05:00
parent 921fb49f9a
commit ba326bfcfd
1058 changed files with 192192 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
future-todo/ControllableTalkNet/Dockerfile
+29
View File
@@ -0,0 +1,29 @@
FROM nvidia/cuda:11.0.3-base-ubuntu20.04
ENV TZ=Etc/GMT
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update && apt-get install -y git ffmpeg python3.8 python3-pip curl
RUN git clone -q https://github.com/SortAnon/ControllableTalkNet /talknet
RUN git clone -q https://github.com/SortAnon/hifi-gan /talknet/hifi-gan
RUN python3.8 -m pip --no-cache-dir install -r "/talknet/requirements.txt" -f https://download.pytorch.org/whl/torch_stable.html
RUN python3.8 -m pip --no-cache-dir install git+https://github.com/SortAnon/NeMo.git
RUN python3.8 -m pip uninstall -y pesq
RUN python3.8 -m pip install pesq==0.0.2
RUN touch /talknet/is_docker
WORKDIR /talknet
EXPOSE 8050
RUN printf "#!/bin/bash \
\necho Updating TalkNet... \
\ngit -C /talknet reset --hard origin/main -q \
\ngit -C /talknet pull origin main -q \
\necho Updating HiFi-GAN... \
\ngit -C /talknet reset --hard origin/main -q \
\ngit -C /talknet pull origin main -q \
\necho Updating Python dependencies... \
\npython3.8 -m pip --quiet --no-cache-dir install -r /talknet/requirements.txt -f https://download.pytorch.org/whl/torch_stable.html \
\npython3.8 -m pip --quiet --no-cache-dir install git+https://github.com/SortAnon/NeMo.git \
\necho Launching TalkNet... \
\npython3.8 talknet_offline.py\n" > /talknet/docker_launch.sh
RUN chmod +x /talknet/docker_launch.sh
CMD ["./docker_launch.sh"]
future-todo/ControllableTalkNet/LICENSE
+121
View File
@@ -0,0 +1,121 @@
Creative Commons Legal Code
CC0 1.0 Universal
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
HEREUNDER.
Statement of Purpose
The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator
and subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").
Certain owners wish to permanently relinquish those rights to a Work for
the purpose of contributing to a commons of creative, cultural and
scientific works ("Commons") that the public can reliably and without fear
of later claims of infringement build upon, modify, incorporate in other
works, reuse and redistribute as freely as possible in any form whatsoever
and for any purposes, including without limitation commercial purposes.
These owners may contribute to the Commons to promote the ideal of a free
culture and the further production of creative, cultural and scientific
works, or to gain reputation or greater distribution for their Work in
part through the use and efforts of others.
For these and/or other purposes and motivations, and without any
expectation of additional consideration or compensation, the person
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
is an owner of Copyright and Related Rights in the Work, voluntarily
elects to apply CC0 to the Work and publicly distribute the Work under its
terms, with knowledge of his or her Copyright and Related Rights in the
Work and the meaning and intended legal effect of CC0 on those rights.
1. Copyright and Related Rights. A Work made available under CC0 may be
protected by copyright and related or neighboring rights ("Copyright and
Related Rights"). Copyright and Related Rights include, but are not
limited to, the following:
i. the right to reproduce, adapt, distribute, perform, display,
communicate, and translate a Work;
ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or
likeness depicted in a Work;
iv. rights protecting against unfair competition in regards to a Work,
subject to the limitations in paragraph 4(a), below;
v. rights protecting the extraction, dissemination, use and reuse of data
in a Work;
vi. database rights (such as those arising under Directive 96/9/EC of the
European Parliament and of the Council of 11 March 1996 on the legal
protection of databases, and under any national implementation
thereof, including any amended or successor version of such
directive); and
vii. other similar, equivalent or corresponding rights throughout the
world based on applicable law or treaty, and any national
implementations thereof.
2. Waiver. To the greatest extent permitted by, but not in contravention
of, applicable law, Affirmer hereby overtly, fully, permanently,
irrevocably and unconditionally waives, abandons, and surrenders all of
Affirmer's Copyright and Related Rights and associated claims and causes
of action, whether now known or unknown (including existing as well as
future claims and causes of action), in the Work (i) in all territories
worldwide, (ii) for the maximum duration provided by applicable law or
treaty (including future time extensions), (iii) in any current or future
medium and for any number of copies, and (iv) for any purpose whatsoever,
including without limitation commercial, advertising or promotional
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
member of the public at large and to the detriment of Affirmer's heirs and
successors, fully intending that such Waiver shall not be subject to
revocation, rescission, cancellation, termination, or any other legal or
equitable action to disrupt the quiet enjoyment of the Work by the public
as contemplated by Affirmer's express Statement of Purpose.
3. Public License Fallback. Should any part of the Waiver for any reason
be judged legally invalid or ineffective under applicable law, then the
Waiver shall be preserved to the maximum extent permitted taking into
account Affirmer's express Statement of Purpose. In addition, to the
extent the Waiver is so judged Affirmer hereby grants to each affected
person a royalty-free, non transferable, non sublicensable, non exclusive,
irrevocable and unconditional license to exercise Affirmer's Copyright and
Related Rights in the Work (i) in all territories worldwide, (ii) for the
maximum duration provided by applicable law or treaty (including future
time extensions), (iii) in any current or future medium and for any number
of copies, and (iv) for any purpose whatsoever, including without
limitation commercial, advertising or promotional purposes (the
"License"). The License shall be deemed effective as of the date CC0 was
applied by Affirmer to the Work. Should any part of the License for any
reason be judged legally invalid or ineffective under applicable law, such
partial invalidity or ineffectiveness shall not invalidate the remainder
of the License, and in such case Affirmer hereby affirms that he or she
will not (i) exercise any of his or her remaining Copyright and Related
Rights in the Work or (ii) assert any associated claims and causes of
action with respect to the Work, in either case contrary to Affirmer's
express Statement of Purpose.
4. Limitations and Disclaimers.
a. No trademark or patent rights held by Affirmer are waived, abandoned,
surrendered, licensed or otherwise affected by this document.
b. Affirmer offers the Work as-is and makes no representations or
warranties of any kind concerning the Work, express, implied,
statutory or otherwise, including without limitation warranties of
title, merchantability, fitness for a particular purpose, non
infringement, or the absence of latent or other defects, accuracy, or
the present or absence of errors, whether or not discoverable, all to
the greatest extent permissible under applicable law.
c. Affirmer disclaims responsibility for clearing rights of other persons
that may apply to the Work or any use thereof, including without
limitation any person's Copyright and Related Rights in the Work.
Further, Affirmer disclaims responsibility for obtaining any necessary
consents, permissions or other rights required for any use of the
Work.
d. Affirmer understands and acknowledges that Creative Commons is not a
party to this document and has no duty or obligation with respect to
this CC0 or use of the Work.
future-todo/ControllableTalkNet/README.md
+26
View File
@@ -0,0 +1,26 @@
# Controllable TalkNet
Controllable TalkNet is a web application that lets you synthesize speech,
which mimics the pitch and pacing of an existing audio clip. It's based on [NVIDIA's implementation](https://github.com/NVIDIA/NeMo)
of TalkNet 2, with some changes to support singing synthesis and higher audio quality.
## Requirements
* A Google account to run Colab, or...
* An NVIDIA GPU with 4+ GB of VRAM
* 10 GB of free space
## How to run
### Google Colab
* [Go to the Colab notebook](https://colab.research.google.com/drive/1aj6Jk8cpRw7SsN3JSYCv57CrR6s0gYPB) and follow the instructions.
### TalkNet Offline (Windows)
* [Download the setup script](https://github.com/SortAnon/ControllableTalkNet/releases/latest/download/TalkNetOffline.zip)
and extract it to a folder.
* Run setup.bat. The initial setup will take about 20 minutes.
* When it's done, run talknet.bat to start TalkNet on http://127.0.0.1:8050/. To download updates, run update.bat.
### Docker (Linux)
* Install Docker and NVIDIA Container Toolkit.
* [Download the Dockerfile.](https://raw.githubusercontent.com/SortAnon/ControllableTalkNet/main/Dockerfile)
Open a terminal, and navigate to the directory where you saved it.
* Run ```docker build -t talknet-offline .``` to build the image. Add ```sudo``` if you're not using rootless Docker.
* Run ```docker run -it --gpus all -p 8050:8050 talknet-offline``` to start TalkNet on http://127.0.0.1:8050/.
future-todo/ControllableTalkNet/TalkNet_Training_Offline.ipynb
+925
View File
@@ -0,0 +1,925 @@
{
"cells": [
{
"cell_type": "markdown",
"source": [
"# TalkNet Training (offline)\n",
"\n",
"To train a 22KHz TalkNet, run the cells below and follow the instructions.\n",
"\n",
"The notebook will automatically resume training any models from the last saved checkpoint. If you're resuming from a new session, always re-run step 1 first."
],
"metadata": {
"id": "Gss5Ox_RNiba"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 1:** Configure training data paths. Create the following and change the paths below:\n",
"* A dataset of .wav files, packaged as a .zip or .tar file\n",
"* Training and validation filelists, in LJSpeech format with relative paths (note: ARPABET transcripts are not supported)\n",
"* An output path for checkpoints"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"### CHANGE THESE ###\n",
"\n",
"dataset = \"/absolute/path_to_dataset.zip\"\n",
"train_filelist = \"/absolute/train_filelist.txt\"\n",
"val_filelist = \"/absolute/val_filelist.txt\"\n",
"output_dir = \"/absolute/name_of_character\"\n",
"\n",
"### ------------ ###\n",
"\n",
"import os\n",
"import gdown\n",
"import platform\n",
"\n",
"assert os.path.exists(dataset), \"Cannot find dataset\"\n",
"assert os.path.exists(train_filelist), \"Cannot find training filelist\"\n",
"assert os.path.exists(val_filelist), \"Cannot find validation filelist\"\n",
"if not os.path.exists(output_dir):\n",
" os.makedirs(output_dir)\n",
"\n",
"if os.getcwd() != output_dir:\n",
" cwd = os.getcwd()\n",
"else:\n",
" os.chdir(cwd)\n",
" \n",
"# Download pre-trained models\n",
"if not os.path.exists(\"talknet_spect.nemo\"):\n",
" print(\"Downloading pre-trained models...\")\n",
" zip_path = \"TalkNet_Pretrained.zip\"\n",
" if not os.path.exists(zip_path) or os.stat(zip_path).st_size < 100:\n",
" d = 'https://drive.google.com/uc?id='\n",
" gdown.download(d+\"19wSym9mNEnmzLS9XdPlfNAW9_u-mP1hR\", zip_path, quiet=False)\n",
" print(platform.system())\n",
" if \"Windows\" in platform.system():\n",
" !tar -xf {zip_path}\n",
" else:\n",
" !unzip -qo {zip_path}\n",
" os.remove(zip_path)\n",
"\n",
"print(\"OK\")"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "nfSawDUD5tqv"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 2:** Dataset processing, part 1.\n",
"\n",
"If this step fails, try the following:\n",
"* Make sure your filelists are correct. They should have relative \n",
"paths that match the contents of the archive."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"import os\n",
"import shutil\n",
"import sys\n",
"import json\n",
"import nemo\n",
"import torch\n",
"import torchaudio\n",
"import numpy as np\n",
"#from pysptk import sptk\n",
"from pathlib import Path\n",
"from tqdm.notebook import tqdm\n",
"import ffmpeg\n",
"\n",
"def fix_transcripts(inpath):\n",
" found_arpabet = False\n",
" found_grapheme = False\n",
" with open(inpath, \"r\", encoding=\"utf8\") as f:\n",
" lines = f.readlines()\n",
" with open(inpath, \"w\", encoding=\"utf8\") as f:\n",
" for l in lines:\n",
" if l.strip() == \"\":\n",
" continue\n",
" if \"{\" in l:\n",
" if not found_arpabet:\n",
" print(\"Warning: Skipping ARPABET lines (not supported).\")\n",
" found_arpabet = True\n",
" else:\n",
" f.write(l)\n",
" found_grapheme = True\n",
" assert found_grapheme, \"No non-ARPABET lines found in \" + inpath\n",
"\n",
"def generate_json(inpath, outpath):\n",
" output = \"\"\n",
" sample_rate = 22050\n",
" with open(inpath, \"r\", encoding=\"utf8\") as f:\n",
" for l in f.readlines():\n",
" lpath = l.split(\"|\")[0].strip()\n",
" if lpath[:5] != \"wavs/\":\n",
" lpath = \"wavs/\" + lpath\n",
" lpath = os.path.join(output_dir, \"_training_data\", lpath)\n",
" size = os.stat(lpath).st_size\n",
" x = {\n",
" \"audio_filepath\": lpath,\n",
" \"duration\": size / (sample_rate * 2),\n",
" \"text\": l.split(\"|\")[1].strip(),\n",
" }\n",
" output += json.dumps(x) + \"\\n\"\n",
" with open(outpath, \"w\", encoding=\"utf8\") as w:\n",
" w.write(output)\n",
"\n",
"def convert_to_22k(inpath):\n",
" if inpath.strip()[-4:].lower() != \".wav\":\n",
" print(\"Warning: \" + inpath.strip() + \" is not a .wav file!\")\n",
" return\n",
" ffmpeg.input(inpath).output(\n",
" inpath + \"_22k.wav\",\n",
" ar=\"22050\",\n",
" ac=\"1\",\n",
" acodec=\"pcm_s16le\",\n",
" map_metadata=\"-1\",\n",
" fflags=\"+bitexact\",\n",
" ).overwrite_output().run(quiet=True)\n",
" os.remove(inpath)\n",
" os.rename(inpath + \"_22k.wav\", inpath)\n",
"\n",
"# Extract dataset\n",
"os.chdir(output_dir)\n",
"\n",
"if os.path.exists(\"_training_data\"):\n",
" shutil.rmtree(\"_training_data\")\n",
"os.mkdir(\"_training_data\")\n",
"os.chdir(\"_training_data\")\n",
"os.mkdir(\"wavs\")\n",
"os.chdir(\"wavs\")\n",
"if dataset[-4:] == \".zip\":\n",
" if \"Windows\" in platform.system():\n",
" !tar -xf \"{dataset}\"\n",
" else:\n",
" !unzip -qo \"{dataset}\"\n",
"elif dataset[-4:] == \".tar\":\n",
" !tar -xf \"{dataset}\"\n",
"else:\n",
" raise Exception(\"Unknown extension for dataset\")\n",
"if os.path.exists(os.path.join(output_dir, \"_training_data\", \"wavs\", \"wavs\")):\n",
" shutil.move(\n",
" os.path.join(output_dir, \"_training_data\", \"wavs\", \"wavs\"), \n",
" os.path.join(output_dir, \"_training_data\", \"tempwavs\")\n",
" )\n",
" shutil.rmtree(os.path.join(output_dir, \"_training_data\", \"wavs\"))\n",
" shutil.move(\n",
" os.path.join(output_dir, \"_training_data\", \"tempwavs\"), \n",
" os.path.join(output_dir, \"_training_data\", \"wavs\")\n",
" )\n",
"# Filelist for preprocessing\n",
"os.chdir(output_dir)\n",
"shutil.copy(train_filelist, \"trainfiles.txt\")\n",
"shutil.copy(val_filelist, \"valfiles.txt\")\n",
"fix_transcripts(\"trainfiles.txt\")\n",
"fix_transcripts(\"valfiles.txt\")\n",
"seen_files = []\n",
"with open(\"trainfiles.txt\") as f:\n",
" t = f.read().split(\"\\n\")\n",
"with open(\"valfiles.txt\") as f:\n",
" v = f.read().split(\"\\n\")\n",
" all_filelist = t[:] + v[:]\n",
"with open(\"allfiles.txt\", \"w\") as f:\n",
" for x in all_filelist:\n",
" if x.strip() == \"\":\n",
" continue\n",
" if x.split(\"|\")[0] not in seen_files:\n",
" seen_files.append(x.split(\"|\")[0])\n",
" f.write(x.strip() + \"\\n\")\n",
"\n",
"# Ensure audio is 22k\n",
"print(\"Converting audio...\")\n",
"for r, _, f in os.walk(os.path.join(\"_training_data\", \"wavs\")):\n",
" for name in tqdm(f):\n",
" convert_to_22k(os.path.join(r, name))\n",
"\n",
"# Convert to JSON\n",
"generate_json(\"trainfiles.txt\", \"trainfiles.json\")\n",
"generate_json(\"valfiles.txt\", \"valfiles.json\")\n",
"generate_json(\"allfiles.txt\", \"allfiles.json\")\n",
"\n",
"print(\"OK\")"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "bxFr3Fdi_kOC"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 3:** Dataset processing, part 2. This takes a while, but\n",
"you only have to run this once per dataset (results are saved to Drive).\n",
"\n",
"If this step fails, try the following:\n",
"* Make sure your dataset only contains WAV files."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Extract phoneme duration\n",
"\n",
"import json\n",
"from nemo.collections.asr.models import EncDecCTCModel\n",
"asr_model = EncDecCTCModel.from_pretrained(model_name=\"asr_talknet_aligner\").cpu().eval()\n",
"\n",
"def forward_extractor(tokens, log_probs, blank):\n",
" \"\"\"Computes states f and p.\"\"\"\n",
" n, m = len(tokens), log_probs.shape[0]\n",
" # `f[s, t]` -- max sum of log probs for `s` first codes\n",
" # with `t` first timesteps with ending in `tokens[s]`.\n",
" f = np.empty((n + 1, m + 1), dtype=float)\n",
" f.fill(-(10 ** 9))\n",
" p = np.empty((n + 1, m + 1), dtype=int)\n",
" f[0, 0] = 0.0 # Start\n",
" for s in range(1, n + 1):\n",
" c = tokens[s - 1]\n",
" for t in range((s + 1) // 2, m + 1):\n",
" f[s, t] = log_probs[t - 1, c]\n",
" # Option #1: prev char is equal to current one.\n",
" if s == 1 or c == blank or c == tokens[s - 3]:\n",
" options = f[s : (s - 2 if s > 1 else None) : -1, t - 1]\n",
" else: # Is not equal to current one.\n",
" options = f[s : (s - 3 if s > 2 else None) : -1, t - 1]\n",
" f[s, t] += np.max(options)\n",
" p[s, t] = np.argmax(options)\n",
" return f, p\n",
"\n",
"\n",
"def backward_extractor(f, p):\n",
" \"\"\"Computes durs from f and p.\"\"\"\n",
" n, m = f.shape\n",
" n -= 1\n",
" m -= 1\n",
" durs = np.zeros(n, dtype=int)\n",
" if f[-1, -1] >= f[-2, -1]:\n",
" s, t = n, m\n",
" else:\n",
" s, t = n - 1, m\n",
" while s > 0:\n",
" durs[s - 1] += 1\n",
" s -= p[s, t]\n",
" t -= 1\n",
" assert durs.shape[0] == n\n",
" assert np.sum(durs) == m\n",
" assert np.all(durs[1::2] > 0)\n",
" return durs\n",
"\n",
"def preprocess_tokens(tokens, blank):\n",
" new_tokens = [blank]\n",
" for c in tokens:\n",
" new_tokens.extend([c, blank])\n",
" tokens = new_tokens\n",
" return tokens\n",
"\n",
"data_config = {\n",
" 'manifest_filepath': \"allfiles.json\",\n",
" 'sample_rate': 22050,\n",
" 'labels': asr_model.decoder.vocabulary,\n",
" 'batch_size': 1,\n",
"}\n",
"\n",
"parser = nemo.collections.asr.data.audio_to_text.AudioToCharWithDursF0Dataset.make_vocab(\n",
" notation='phonemes', punct=True, spaces=True, stresses=False, add_blank_at=\"last\"\n",
")\n",
"\n",
"dataset = nemo.collections.asr.data.audio_to_text._AudioTextDataset(\n",
" manifest_filepath=data_config['manifest_filepath'], sample_rate=data_config['sample_rate'], parser=parser,\n",
")\n",
"\n",
"dl = torch.utils.data.DataLoader(\n",
" dataset=dataset, batch_size=data_config['batch_size'], collate_fn=dataset.collate_fn, shuffle=False,\n",
")\n",
"\n",
"blank_id = asr_model.decoder.num_classes_with_blank - 1\n",
"\n",
"if os.path.exists(os.path.join(output_dir, \"durations.pt\")):\n",
" print(\"durations.pt already exists; skipping\")\n",
"else:\n",
" dur_data = {}\n",
" for sample_idx, test_sample in tqdm(enumerate(dl), total=len(dl)):\n",
" log_probs, _, greedy_predictions = asr_model(\n",
" input_signal=test_sample[0], input_signal_length=test_sample[1]\n",
" )\n",
"\n",
" log_probs = log_probs[0].cpu().detach().numpy()\n",
" seq_ids = test_sample[2][0].cpu().detach().numpy()\n",
"\n",
" target_tokens = preprocess_tokens(seq_ids, blank_id)\n",
"\n",
" f, p = forward_extractor(target_tokens, log_probs, blank_id)\n",
" durs = backward_extractor(f, p)\n",
"\n",
" dur_key = Path(dl.dataset.collection[sample_idx].audio_file).stem\n",
" dur_data[dur_key] = {\n",
" 'blanks': torch.tensor(durs[::2], dtype=torch.long).cpu().detach(), \n",
" 'tokens': torch.tensor(durs[1::2], dtype=torch.long).cpu().detach()\n",
" }\n",
"\n",
" del test_sample\n",
"\n",
" torch.save(dur_data, os.path.join(output_dir, \"durations.pt\"))\n",
"\n",
"#Extract F0 (pitch)\n",
"import crepe\n",
"from scipy.io import wavfile\n",
"\n",
"def crepe_f0(audio_file, hop_length=256):\n",
" sr, audio = wavfile.read(audio_file)\n",
" audio_x = np.arange(0, len(audio)) / 22050.0\n",
" time, frequency, confidence, activation = crepe.predict(audio, sr, viterbi=True)\n",
"\n",
" x = np.arange(0, len(audio), hop_length) / 22050.0\n",
" freq_interp = np.interp(x, time, frequency)\n",
" conf_interp = np.interp(x, time, confidence)\n",
" audio_interp = np.interp(x, audio_x, np.absolute(audio)) / 32768.0\n",
" weights = [0.5, 0.25, 0.25]\n",
" audio_smooth = np.convolve(audio_interp, np.array(weights)[::-1], \"same\")\n",
"\n",
" conf_threshold = 0.25\n",
" audio_threshold = 0.0005\n",
" for i in range(len(freq_interp)):\n",
" if conf_interp[i] < conf_threshold:\n",
" freq_interp[i] = 0.0\n",
" if audio_smooth[i] < audio_threshold:\n",
" freq_interp[i] = 0.0\n",
"\n",
" # Hack to make f0 and mel lengths equal\n",
" if len(audio) % hop_length == 0:\n",
" freq_interp = np.pad(freq_interp, pad_width=[0, 1])\n",
" return torch.from_numpy(freq_interp.astype(np.float32))\n",
"\n",
"if os.path.exists(os.path.join(output_dir, \"f0s.pt\")):\n",
" print(\"f0s.pt already exists; skipping\")\n",
"else:\n",
" f0_data = {}\n",
" with open(\"allfiles.json\") as f:\n",
" for i, l in enumerate(f.readlines()):\n",
" print(str(i))\n",
" audio_path = json.loads(l)[\"audio_filepath\"]\n",
" f0_data[Path(audio_path).stem] = crepe_f0(audio_path)\n",
"\n",
" # calculate f0 stats (mean & std) only for train set\n",
" with open(\"trainfiles.json\") as f:\n",
" train_ids = {Path(json.loads(l)[\"audio_filepath\"]).stem for l in f}\n",
" all_f0 = torch.cat([f0[f0 >= 1e-5] for f0_id, f0 in f0_data.items() if f0_id in train_ids])\n",
"\n",
" F0_MEAN, F0_STD = all_f0.mean().item(), all_f0.std().item() \n",
" print(\"F0_MEAN: \" + str(F0_MEAN) + \", F0_STD: \" + str(F0_STD))\n",
" torch.save(f0_data, os.path.join(output_dir, \"f0s.pt\"))\n",
" with open(os.path.join(output_dir, \"f0_info.json\"), \"w\") as f:\n",
" f.write(json.dumps({\"FO_MEAN\": F0_MEAN, \"F0_STD\": F0_STD}))\n",
"\n",
"print(\"OK\")"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "sos9vsxPkIN7"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 4:** Train duration predictor.\n",
"\n",
"If CUDA runs out of memory, try the following:\n",
"* Click on Kernel -> Restart, re-run step 1, and try again.\n",
"* If that doesn't help, reduce the batch size (default 64)."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"batch_size = 64\n",
"\n",
"import os\n",
"from hydra.experimental import compose, initialize\n",
"from hydra.core.global_hydra import GlobalHydra\n",
"from omegaconf import OmegaConf\n",
"import pytorch_lightning as pl\n",
"from nemo.collections.common.callbacks import LogEpochTimeCallback\n",
"from nemo.collections.tts.models import TalkNetDursModel\n",
"from nemo.core.config import hydra_runner\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"epochs = 20\n",
"learning_rate = 1e-3\n",
"min_learning_rate = 3e-6\n",
"load_checkpoints = True\n",
"\n",
"def train(cfg):\n",
" cfg.sample_rate = 22050\n",
" cfg.train_dataset = os.path.join(output_dir, \"trainfiles.json\")\n",
" cfg.validation_datasets = os.path.join(output_dir, \"valfiles.json\")\n",
" cfg.durs_file = os.path.join(output_dir, \"durations.pt\")\n",
" cfg.f0_file = os.path.join(output_dir, \"f0s.pt\")\n",
" cfg.trainer.accelerator = \"dp\"\n",
" cfg.trainer.max_epochs = epochs\n",
" cfg.trainer.check_val_every_n_epoch = 5\n",
" cfg.model.train_ds.dataloader_params.batch_size = batch_size\n",
" cfg.model.validation_ds.dataloader_params.batch_size = batch_size\n",
" cfg.model.optim.lr = learning_rate\n",
" cfg.model.optim.sched.min_lr = min_learning_rate\n",
" cfg.exp_manager.exp_dir = output_dir\n",
"\n",
" # Find checkpoints\n",
" ckpt_path = \"\"\n",
" if load_checkpoints:\n",
" path0 = os.path.join(output_dir, \"TalkNetDurs\")\n",
" if os.path.exists(path0):\n",
" path1 = sorted(os.listdir(path0))\n",
" for i in range(len(path1)):\n",
" path2 = os.path.join(path0, path1[-(1+i)], \"checkpoints\")\n",
" if os.path.exists(path2):\n",
" match = [x for x in os.listdir(path2) if \"last.ckpt\" in x]\n",
" if len(match) > 0:\n",
" ckpt_path = os.path.join(path2, match[0])\n",
" print(\"Resuming training from \" + match[0])\n",
" break\n",
" \n",
" if ckpt_path != \"\":\n",
" trainer = pl.Trainer(**cfg.trainer, resume_from_checkpoint = ckpt_path)\n",
" model = TalkNetDursModel(cfg=cfg.model, trainer=trainer)\n",
" else:\n",
" warmstart_path = os.path.join(cwd, \"talknet_durs.nemo\")\n",
" trainer = pl.Trainer(**cfg.trainer)\n",
" model = TalkNetDursModel.restore_from(warmstart_path, override_config_path=cfg)\n",
" model.set_trainer(trainer)\n",
" model.setup_training_data(cfg.model.train_ds)\n",
" model.setup_validation_data(cfg.model.validation_ds)\n",
" model.setup_optimization(cfg.model.optim)\n",
" print(\"Warm-starting from \" + warmstart_path)\n",
" exp_manager(trainer, cfg.get('exp_manager', None))\n",
" trainer.callbacks.extend([pl.callbacks.LearningRateMonitor(), LogEpochTimeCallback()]) # noqa\n",
" trainer.fit(model)\n",
"\n",
"os.chdir(cwd)\n",
"GlobalHydra().clear()\n",
"initialize(config_path=\"conf\")\n",
"cfg = compose(config_name=\"talknet-durs\")\n",
"train(cfg)\n"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "nM7-bMpKO7U2"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 5:** Train pitch predictor.\n",
"\n",
"If CUDA runs out of memory, try the following:\n",
"* Click on Kernel -> Restart, re-run step 1, and try again.\n",
"* If that doesn't help, reduce the batch size (default 64)."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"batch_size = 64\n",
"epochs = 50\n",
"\n",
"import json\n",
"\n",
"with open(os.path.join(output_dir, \"f0_info.json\"), \"r\") as f:\n",
" f0_info = json.load(f)\n",
" f0_mean = f0_info[\"FO_MEAN\"]\n",
" f0_std = f0_info[\"F0_STD\"]\n",
"\n",
"learning_rate = 1e-3\n",
"min_learning_rate = 3e-6\n",
"load_checkpoints = True\n",
"\n",
"import os\n",
"from hydra.experimental import compose, initialize\n",
"from hydra.core.global_hydra import GlobalHydra\n",
"from omegaconf import OmegaConf\n",
"import pytorch_lightning as pl\n",
"from nemo.collections.common.callbacks import LogEpochTimeCallback\n",
"from nemo.collections.tts.models import TalkNetPitchModel\n",
"from nemo.core.config import hydra_runner\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"def train(cfg):\n",
" cfg.sample_rate = 22050\n",
" cfg.train_dataset = os.path.join(output_dir, \"trainfiles.json\")\n",
" cfg.validation_datasets = os.path.join(output_dir, \"valfiles.json\")\n",
" cfg.durs_file = os.path.join(output_dir, \"durations.pt\")\n",
" cfg.f0_file = os.path.join(output_dir, \"f0s.pt\")\n",
" cfg.trainer.accelerator = \"dp\"\n",
" cfg.trainer.max_epochs = epochs\n",
" cfg.trainer.check_val_every_n_epoch = 5\n",
" cfg.model.f0_mean=f0_mean\n",
" cfg.model.f0_std=f0_std\n",
" cfg.model.train_ds.dataloader_params.batch_size = batch_size\n",
" cfg.model.validation_ds.dataloader_params.batch_size = batch_size\n",
" cfg.model.optim.lr = learning_rate\n",
" cfg.model.optim.sched.min_lr = min_learning_rate\n",
" cfg.exp_manager.exp_dir = output_dir\n",
"\n",
" # Find checkpoints\n",
" ckpt_path = \"\"\n",
" if load_checkpoints:\n",
" path0 = os.path.join(output_dir, \"TalkNetPitch\")\n",
" if os.path.exists(path0):\n",
" path1 = sorted(os.listdir(path0))\n",
" for i in range(len(path1)):\n",
" path2 = os.path.join(path0, path1[-(1+i)], \"checkpoints\")\n",
" if os.path.exists(path2):\n",
" match = [x for x in os.listdir(path2) if \"last.ckpt\" in x]\n",
" if len(match) > 0:\n",
" ckpt_path = os.path.join(path2, match[0])\n",
" print(\"Resuming training from \" + match[0])\n",
" break\n",
" \n",
" if ckpt_path != \"\":\n",
" trainer = pl.Trainer(**cfg.trainer, resume_from_checkpoint = ckpt_path)\n",
" model = TalkNetPitchModel(cfg=cfg.model, trainer=trainer)\n",
" else:\n",
" warmstart_path = os.path.join(cwd, \"talknet_pitch.nemo\")\n",
" trainer = pl.Trainer(**cfg.trainer)\n",
" model = TalkNetPitchModel.restore_from(warmstart_path, override_config_path=cfg)\n",
" model.set_trainer(trainer)\n",
" model.setup_training_data(cfg.model.train_ds)\n",
" model.setup_validation_data(cfg.model.validation_ds)\n",
" model.setup_optimization(cfg.model.optim)\n",
" print(\"Warm-starting from \" + warmstart_path)\n",
" exp_manager(trainer, cfg.get('exp_manager', None))\n",
" trainer.callbacks.extend([pl.callbacks.LearningRateMonitor(), LogEpochTimeCallback()]) # noqa\n",
" trainer.fit(model)\n",
"\n",
"os.chdir(cwd)\n",
"GlobalHydra().clear()\n",
"initialize(config_path=\"conf\")\n",
"cfg = compose(config_name=\"talknet-pitch\")\n",
"train(cfg)"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "JLfm00NuJfon"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 6:** Train spectrogram generator. 200+ epochs are recommended. \n",
"\n",
"This is the slowest of the three models to train, and the hardest to\n",
"get good results from. If your character sounds noisy or robotic,\n",
"try improving the dataset, or adjusting the epochs and learning rate.\n",
"\n",
"If CUDA runs out of memory, try the following:\n",
"* Click on Kernel -> Restart, re-run step 1, and try again.\n",
"* If that doesn't help, reduce the batch size (default 32)."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"epochs = 200\n",
"batch_size = 32\n",
"\n",
"# Advanced settings. You can probably leave these at their defaults (1e-3, 3e-6, empty, checked).\n",
"learning_rate = 1e-3\n",
"min_learning_rate = 3e-6\n",
"pretrained_path = \"\"\n",
"load_checkpoints = True\n",
"\n",
"import os\n",
"from hydra.experimental import compose, initialize\n",
"from hydra.core.global_hydra import GlobalHydra\n",
"from omegaconf import OmegaConf\n",
"import pytorch_lightning as pl\n",
"from nemo.collections.common.callbacks import LogEpochTimeCallback\n",
"from nemo.collections.tts.models import TalkNetSpectModel\n",
"from nemo.core.config import hydra_runner\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"def train(cfg):\n",
" cfg.sample_rate = 22050\n",
" cfg.train_dataset = os.path.join(output_dir, \"trainfiles.json\")\n",
" cfg.validation_datasets = os.path.join(output_dir, \"valfiles.json\")\n",
" cfg.durs_file = os.path.join(output_dir, \"durations.pt\")\n",
" cfg.f0_file = os.path.join(output_dir, \"f0s.pt\")\n",
" cfg.trainer.accelerator = \"dp\"\n",
" cfg.trainer.max_epochs = epochs\n",
" cfg.trainer.check_val_every_n_epoch = 5\n",
" cfg.model.train_ds.dataloader_params.batch_size = batch_size\n",
" cfg.model.validation_ds.dataloader_params.batch_size = batch_size\n",
" cfg.model.optim.lr = learning_rate\n",
" cfg.model.optim.sched.min_lr = min_learning_rate\n",
" cfg.exp_manager.exp_dir = output_dir\n",
"\n",
" # Find checkpoints\n",
" ckpt_path = \"\"\n",
" if load_checkpoints:\n",
" path0 = os.path.join(output_dir, \"TalkNetSpect\")\n",
" if os.path.exists(path0):\n",
" path1 = sorted(os.listdir(path0))\n",
" for i in range(len(path1)):\n",
" path2 = os.path.join(path0, path1[-(1+i)], \"checkpoints\")\n",
" if os.path.exists(path2):\n",
" match = [x for x in os.listdir(path2) if \"last.ckpt\" in x]\n",
" if len(match) > 0:\n",
" ckpt_path = os.path.join(path2, match[0])\n",
" print(\"Resuming training from \" + match[0])\n",
" break\n",
" \n",
" if ckpt_path != \"\":\n",
" trainer = pl.Trainer(**cfg.trainer, resume_from_checkpoint = ckpt_path)\n",
" model = TalkNetSpectModel(cfg=cfg.model, trainer=trainer)\n",
" else:\n",
" if pretrained_path != \"\":\n",
" warmstart_path = pretrained_path\n",
" else:\n",
" warmstart_path = os.path.join(cwd, \"talknet_spect.nemo\")\n",
" trainer = pl.Trainer(**cfg.trainer)\n",
" model = TalkNetSpectModel.restore_from(warmstart_path, override_config_path=cfg)\n",
" model.set_trainer(trainer)\n",
" model.setup_training_data(cfg.model.train_ds)\n",
" model.setup_validation_data(cfg.model.validation_ds)\n",
" model.setup_optimization(cfg.model.optim)\n",
" print(\"Warm-starting from \" + warmstart_path)\n",
" exp_manager(trainer, cfg.get('exp_manager', None))\n",
" trainer.callbacks.extend([pl.callbacks.LearningRateMonitor(), LogEpochTimeCallback()]) # noqa\n",
" trainer.fit(model)\n",
"\n",
"os.chdir(cwd)\n",
"GlobalHydra().clear()\n",
"initialize(config_path=\"conf\")\n",
"cfg = compose(config_name=\"talknet-spect\")\n",
"train(cfg)"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "N9hh4WPHbCcn"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 7:** Generate GTA spectrograms. This will help HiFi-GAN learn what your TalkNet model sounds like.\n",
"\n",
"If this step fails, make sure you've finished training the spectrogram generator."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"import sys\n",
"import os\n",
"import torch\n",
"import numpy as np\n",
"from tqdm import tqdm\n",
"from nemo.collections.tts.models import TalkNetSpectModel\n",
"import shutil\n",
"\n",
"def fix_paths(inpath):\n",
" output = \"\"\n",
" with open(inpath, \"r\", encoding=\"utf8\") as f:\n",
" for l in f.readlines():\n",
" if l[:5].lower() != \"wavs/\":\n",
" output += \"wavs/\" + l\n",
" else:\n",
" output += l\n",
" with open(inpath, \"w\", encoding=\"utf8\") as w:\n",
" w.write(output)\n",
"\n",
"shutil.copyfile(train_filelist, os.path.join(cwd, \"hifi-gan\", \"training.txt\"))\n",
"shutil.copyfile(val_filelist, os.path.join(cwd, \"hifi-gan\", \"validation.txt\"))\n",
"fix_paths(os.path.join(cwd, \"hifi-gan\", \"training.txt\"))\n",
"fix_paths(os.path.join(cwd, \"hifi-gan\", \"validation.txt\"))\n",
"fix_paths(os.path.join(output_dir, \"allfiles.txt\"))\n",
"\n",
"os.chdir(cwd)\n",
"outdir = os.path.join(output_dir, \"_training_data\", \"wavs\")\n",
"if not os.path.exists(outdir):\n",
" os.mkdir(outdir)\n",
"\n",
"model_path = \"\"\n",
"path0 = os.path.join(output_dir, \"TalkNetSpect\")\n",
"if os.path.exists(path0):\n",
" path1 = sorted(os.listdir(path0))\n",
" for i in range(len(path1)):\n",
" path2 = os.path.join(path0, path1[-(1+i)], \"checkpoints\")\n",
" if os.path.exists(path2):\n",
" match = [x for x in os.listdir(path2) if \"TalkNetSpect.nemo\" in x]\n",
" if len(match) > 0:\n",
" model_path = os.path.join(path2, match[0])\n",
" break\n",
"assert model_path != \"\", \"TalkNetSpect.nemo not found\"\n",
"\n",
"dur_path = os.path.join(output_dir, \"durations.pt\")\n",
"f0_path = os.path.join(output_dir, \"f0s.pt\")\n",
"\n",
"model = TalkNetSpectModel.restore_from(model_path)\n",
"model.eval()\n",
"with open(os.path.join(output_dir, \"allfiles.txt\"), \"r\", encoding=\"utf-8\") as f:\n",
" dataset = f.readlines()\n",
"durs = torch.load(dur_path)\n",
"f0s = torch.load(f0_path)\n",
"\n",
"for x in tqdm(dataset):\n",
" x_name = os.path.splitext(os.path.basename(x.split(\"|\")[0].strip()))[0]\n",
" x_tokens = model.parse(text=x.split(\"|\")[1].strip())\n",
" x_durs = (\n",
" torch.stack(\n",
" (\n",
" durs[x_name][\"blanks\"],\n",
" torch.cat((durs[x_name][\"tokens\"], torch.zeros(1).int())),\n",
" ),\n",
" dim=1,\n",
" )\n",
" .view(-1)[:-1]\n",
" .view(1, -1)\n",
" .to(\"cuda:0\")\n",
" )\n",
" x_f0s = f0s[x_name].view(1, -1).to(\"cuda:0\")\n",
" x_spect = model.force_spectrogram(tokens=x_tokens, durs=x_durs, f0=x_f0s)\n",
" rel_path = os.path.splitext(x.split(\"|\")[0].strip())[0][5:]\n",
" abs_dir = os.path.join(outdir, os.path.dirname(rel_path))\n",
" if abs_dir != \"\" and not os.path.exists(abs_dir):\n",
" os.makedirs(abs_dir, exist_ok=True)\n",
" np.save(os.path.join(outdir, rel_path + \".npy\"), x_spect.detach().cpu().numpy())\n"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "Ajfyfz2p9Ior"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 8:** Train HiFi-GAN. 2,000+ steps are recommended.\n",
"Stop this cell to finish training the model.\n",
"\n",
"* Click on Kernel -> Restart, re-run step 1, and try again.\n",
"If this step still fails to start, make sure step 7 finished successfully.\n",
"\n",
"Note: If the training process starts at step 2500000, delete the HiFiGAN folder and try again."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"import gdown\n",
"d = 'https://drive.google.com/uc?id='\n",
"\n",
"os.chdir(os.path.join(cwd, \"hifi-gan\"))\n",
"\n",
"if not os.path.exists(os.path.join(output_dir, \"HiFiGAN\")):\n",
" os.makedirs(os.path.join(output_dir, \"HiFiGAN\"))\n",
"if not os.path.exists(os.path.join(output_dir, \"HiFiGAN\", \"do_00000000\")):\n",
" print(\"Downloading universal model...\")\n",
" gdown.download(d+\"1qpgI41wNXFcH-iKq1Y42JlBC9j0je8PW\", os.path.join(output_dir, \"HiFiGAN\", \"g_00000000\"), quiet=False)\n",
" gdown.download(d+\"1O63eHZR9t1haCdRHQcEgMfMNxiOciSru\", os.path.join(output_dir, \"HiFiGAN\", \"do_00000000\"), quiet=False)\n",
" start_from_universal = \"--warm_start True \"\n",
"else:\n",
" start_from_universal = \"\"\n",
"\n",
"hifi_train = os.path.join(cwd, \"hifi-gan\", \"training.txt\")\n",
"hifi_val = os.path.join(cwd, \"hifi-gan\", \"validation.txt\")\n",
"hifi_wavs = os.path.join(output_dir, \"_training_data\")\n",
"\n",
"!python train.py --fine_tuning True --config config_v1b.json \\\n",
"{start_from_universal} \\\n",
"--checkpoint_interval 250 --checkpoint_path \"{os.path.join(output_dir, 'HiFiGAN')}\" \\\n",
"--input_training_file \"{hifi_train}\" \\\n",
"--input_validation_file \"{hifi_val}\" \\\n",
"--input_wavs_dir \"{hifi_wavs}\"\n"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "yVBjGhRB9hUJ"
}
},
{
"cell_type": "markdown",
"source": [
"**Step 9:** Package the models. They'll be saved to the output directory as [character_name]_TalkNet.zip.\n",
"\n",
"When done, upload it to Google Drive, with permissions set to \"Anyone with the link\". \n",
"You can then use it with TalkNet by selecting \"Custom model\" as your character. \n",
"\n",
"This cell will also delete the training checkpoints and logs.\n",
"That should free up roughly 2 GB of space.\n",
"If you wish to keep them, set delete_checkpoints to False."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"character_name = \"Character\"\n",
"delete_checkpoints = True\n",
"\n",
"import shutil\n",
"from zipfile import ZipFile\n",
"\n",
"def find_talknet(model_dir):\n",
" ckpt_path = \"\"\n",
" path0 = os.path.join(output_dir, model_dir)\n",
" if os.path.exists(path0):\n",
" path1 = sorted(os.listdir(path0))\n",
" for i in range(len(path1)):\n",
" path2 = os.path.join(path0, path1[-(1+i)], \"checkpoints\")\n",
" if os.path.exists(path2):\n",
" match = [x for x in os.listdir(path2) if \".nemo\" in x]\n",
" if len(match) > 0:\n",
" ckpt_path = os.path.join(path2, match[0])\n",
" break\n",
" assert ckpt_path != \"\", \"Couldn't find \" + model_dir\n",
" return ckpt_path\n",
"\n",
"durs_path = find_talknet(\"TalkNetDurs\")\n",
"pitch_path = find_talknet(\"TalkNetPitch\")\n",
"spect_path = find_talknet(\"TalkNetSpect\")\n",
"assert os.path.exists(os.path.join(output_dir, \"HiFiGAN\", \"g_00000000\")), \"Couldn't find HiFi-GAN\"\n",
"\n",
"zip = ZipFile(os.path.join(output_dir, character_name + \"_TalkNet.zip\"), 'w')\n",
"zip.write(durs_path, \"TalkNetDurs.nemo\")\n",
"zip.write(pitch_path, \"TalkNetPitch.nemo\")\n",
"zip.write(spect_path, \"TalkNetSpect.nemo\")\n",
"zip.write(os.path.join(output_dir, \"HiFiGAN\", \"g_00000000\"), \"hifiganmodel\")\n",
"zip.write(os.path.join(output_dir, \"HiFiGAN\", \"config.json\"), \"config.json\")\n",
"zip.write(os.path.join(output_dir, \"f0_info.json\"), \"f0_info.json\")\n",
"zip.close()\n",
"print(\"Archived model to \" + os.path.join(output_dir, character_name + \"_TalkNet.zip\"))\n",
"\n",
"if delete_checkpoints:\n",
" shutil.rmtree((os.path.join(output_dir, \"TalkNetDurs\")))\n",
" shutil.rmtree((os.path.join(output_dir, \"TalkNetPitch\")))\n",
" shutil.rmtree((os.path.join(output_dir, \"TalkNetSpect\")))\n",
" shutil.rmtree((os.path.join(output_dir, \"HiFiGAN\")))\n"
],
"outputs": [],
"metadata": {
"cellView": "form",
"id": "5OtwfrTT-blU"
}
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "TalkNet_Training_Offline.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
future-todo/ControllableTalkNet/__init__.py
View File
future-todo/ControllableTalkNet/assets/dash_customfont.css
+420
View File
@@ -0,0 +1,420 @@
@font-face {
font-family: 'EquestriaWebfont';
src: url('Equestria.woff') format('woff');
}
/* Table of contents
––––––––––––––––––––––––––––––––––––––––––––––––––
- Plotly.js
- Grid
- Base Styles
- Typography
- Links
- Buttons
- Forms
- Lists
- Code
- Tables
- Spacing
- Utilities
- Clearing
- Media Queries
*/
/* PLotly.js
–––––––––––––––––––––––––––––––––––––––––––––––––– */
/* plotly.js's modebar's z-index is 1001 by default
* https://github.com/plotly/plotly.js/blob/7e4d8ab164258f6bd48be56589dacd9bdd7fded2/src/css/_modebar.scss#L5
* In case a dropdown is above the graph, the dropdown's options
* will be rendered below the modebar
* Increase the select option's z-index
*/
/* This was actually not quite right -
dropdowns were overlapping each other (edited October 26)
.Select {
z-index: 1002;
}*/
/* Grid
–––––––––––––––––––––––––––––––––––––––––––––––––– */
.container {
position: relative;
width: 100%;
max-width: 960px;
margin: 0 auto;
padding: 0 20px;
box-sizing: border-box; }
.column,
.columns {
width: 100%;
float: left;
box-sizing: border-box; }
/* For devices larger than 400px */
@media (min-width: 400px) {
.container {
width: 85%;
padding: 0; }
}
/* For devices larger than 550px */
@media (min-width: 550px) {
.container {
width: 80%; }
.column,
.columns {
margin-left: 4%; }
.column:first-child,
.columns:first-child {
margin-left: 0; }
.one.column,
.one.columns { width: 4.66666666667%; }
.two.columns { width: 13.3333333333%; }
.three.columns { width: 22%; }
.four.columns { width: 30.6666666667%; }
.five.columns { width: 39.3333333333%; }
.six.columns { width: 48%; }
.seven.columns { width: 56.6666666667%; }
.eight.columns { width: 65.3333333333%; }
.nine.columns { width: 74.0%; }
.ten.columns { width: 82.6666666667%; }
.eleven.columns { width: 91.3333333333%; }
.twelve.columns { width: 100%; margin-left: 0; }
.one-third.column { width: 30.6666666667%; }
.two-thirds.column { width: 65.3333333333%; }
.one-half.column { width: 48%; }
/* Offsets */
.offset-by-one.column,
.offset-by-one.columns { margin-left: 8.66666666667%; }
.offset-by-two.column,
.offset-by-two.columns { margin-left: 17.3333333333%; }
.offset-by-three.column,
.offset-by-three.columns { margin-left: 26%; }
.offset-by-four.column,
.offset-by-four.columns { margin-left: 34.6666666667%; }
.offset-by-five.column,
.offset-by-five.columns { margin-left: 43.3333333333%; }
.offset-by-six.column,
.offset-by-six.columns { margin-left: 52%; }
.offset-by-seven.column,
.offset-by-seven.columns { margin-left: 60.6666666667%; }
.offset-by-eight.column,
.offset-by-eight.columns { margin-left: 69.3333333333%; }
.offset-by-nine.column,
.offset-by-nine.columns { margin-left: 78.0%; }
.offset-by-ten.column,
.offset-by-ten.columns { margin-left: 86.6666666667%; }
.offset-by-eleven.column,
.offset-by-eleven.columns { margin-left: 95.3333333333%; }
.offset-by-one-third.column,
.offset-by-one-third.columns { margin-left: 34.6666666667%; }
.offset-by-two-thirds.column,
.offset-by-two-thirds.columns { margin-left: 69.3333333333%; }
.offset-by-one-half.column,
.offset-by-one-half.columns { margin-left: 52%; }
}
/* Base Styles
–––––––––––––––––––––––––––––––––––––––––––––––––– */
/* NOTE
html is set to 62.5% so that all the REM measurements throughout Skeleton
are based on 10px sizing. So basically 1.5rem = 15px :) */
html {
font-size: 62.5%; }
body {
font-size: 1.5em; /* currently ems cause chrome bug misinterpreting rems on body element */
line-height: 1.6;
font-weight: 400;
font-family: "Open Sans", "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
color: rgb(50, 50, 50); }
/* Typography
–––––––––––––––––––––––––––––––––––––––––––––––––– */
h1, h2, h3, h4, h5, h6 {
margin-top: 0;
margin-bottom: 0;
font-weight: 300; }
h1 { font-size: 4.5rem; line-height: 1.2; letter-spacing: -.1rem; margin-bottom: 2rem; }
h2 { font-size: 3.6rem; line-height: 1.25; letter-spacing: -.1rem; margin-bottom: 1.8rem; margin-top: 1.8rem;}
h3 { font-size: 3.0rem; line-height: 1.3; letter-spacing: -.1rem; margin-bottom: 1.5rem; margin-top: 1.5rem;}
h4 { font-size: 2.6rem; line-height: 1.35; letter-spacing: -.08rem; margin-bottom: 1.2rem; margin-top: 1.2rem;}
h5 { font-size: 2.2rem; line-height: 1.5; letter-spacing: -.05rem; margin-bottom: 0.6rem; margin-top: 0.6rem;}
h6 { font-size: 2.0rem; line-height: 1.6; letter-spacing: 0; margin-bottom: 0.75rem; margin-top: 0.75rem;}
p {
margin-top: 0; }
/* Blockquotes
–––––––––––––––––––––––––––––––––––––––––––––––––– */
blockquote {
border-left: 4px lightgrey solid;
padding-left: 1rem;
margin-top: 2rem;
margin-bottom: 2rem;
margin-left: 0rem;
}
/* Links
–––––––––––––––––––––––––––––––––––––––––––––––––– */
a {
color: #1EAEDB;
text-decoration: underline;
cursor: pointer;}
a:hover {
color: #0FA0CE; }
/* Buttons
–––––––––––––––––––––––––––––––––––––––––––––––––– */
.button,
button,
input[type="submit"],
input[type="reset"],
input[type="button"] {
display: inline-block;
height: 38px;
padding: 0 30px;
color: #555;
text-align: center;
font-size: 11px;
font-weight: 600;
line-height: 38px;
letter-spacing: .1rem;
text-transform: uppercase;
text-decoration: none;
white-space: nowrap;
background-color: transparent;
border-radius: 4px;
border: 1px solid #bbb;
cursor: pointer;
box-sizing: border-box; }
.button:hover,
button:hover,
input[type="submit"]:hover,
input[type="reset"]:hover,
input[type="button"]:hover,
.button:focus,
button:focus,
input[type="submit"]:focus,
input[type="reset"]:focus,
input[type="button"]:focus {
color: #333;
border-color: #888;
outline: 0; }
.button.button-primary,
button.button-primary,
input[type="submit"].button-primary,
input[type="reset"].button-primary,
input[type="button"].button-primary {
color: #FFF;
background-color: #33C3F0;
border-color: #33C3F0; }
.button.button-primary:hover,
button.button-primary:hover,
input[type="submit"].button-primary:hover,
input[type="reset"].button-primary:hover,
input[type="button"].button-primary:hover,
.button.button-primary:focus,
button.button-primary:focus,
input[type="submit"].button-primary:focus,
input[type="reset"].button-primary:focus,
input[type="button"].button-primary:focus {
color: #FFF;
background-color: #1EAEDB;
border-color: #1EAEDB; }
/* Forms
–––––––––––––––––––––––––––––––––––––––––––––––––– */
input[type="email"],
input[type="number"],
input[type="search"],
input[type="text"],
input[type="tel"],
input[type="url"],
input[type="password"],
textarea,
select {
height: 38px;
padding: 6px 10px; /* The 6px vertically centers text on FF, ignored by Webkit */
background-color: #fff;
border: 1px solid #D1D1D1;
border-radius: 4px;
box-shadow: none;
box-sizing: border-box;
font-family: inherit;
font-size: inherit; /*https://stackoverflow.com/questions/6080413/why-doesnt-input-inherit-the-font-from-body*/}
/* Removes awkward default styles on some inputs for iOS */
input[type="email"],
input[type="number"],
input[type="search"],
input[type="text"],
input[type="tel"],
input[type="url"],
input[type="password"],
textarea {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none; }
textarea {
min-height: 65px;
padding-top: 6px;
padding-bottom: 6px; }
input[type="email"]:focus,
input[type="number"]:focus,
input[type="search"]:focus,
input[type="text"]:focus,
input[type="tel"]:focus,
input[type="url"]:focus,
input[type="password"]:focus,
textarea:focus,
select:focus {
border: 1px solid #33C3F0;
outline: 0; }
label,
legend {
display: block;
margin-bottom: 0px; }
fieldset {
padding: 0;
border-width: 0; }
input[type="checkbox"],
input[type="radio"] {
display: inline; }
label > .label-body {
display: inline-block;
margin-left: .5rem;
font-weight: normal; }
/* Lists
–––––––––––––––––––––––––––––––––––––––––––––––––– */
ul {
list-style: circle inside; }
ol {
list-style: decimal inside; }
ol, ul {
padding-left: 0;
margin-top: 0; }
ul ul,
ul ol,
ol ol,
ol ul {
margin: 1.5rem 0 1.5rem 3rem;
font-size: 90%; }
li {
margin-bottom: 1rem; }
/* Tables
–––––––––––––––––––––––––––––––––––––––––––––––––– */
table {
border-collapse: collapse;
}
th:not(.CalendarDay),
td:not(.CalendarDay) {
padding: 12px 15px;
text-align: left;
border-bottom: 1px solid #E1E1E1; }
th:first-child:not(.CalendarDay),
td:first-child:not(.CalendarDay) {
padding-left: 0; }
th:last-child:not(.CalendarDay),
td:last-child:not(.CalendarDay) {
padding-right: 0; }
/* Spacing
–––––––––––––––––––––––––––––––––––––––––––––––––– */
button,
.button {
margin-bottom: 0rem; }
input,
textarea,
select,
fieldset {
margin-bottom: 0rem; }
pre,
dl,
figure,
table,
form {
margin-bottom: 0rem; }
p,
ul,
ol {
margin-bottom: 0.75rem; }
/* Utilities
–––––––––––––––––––––––––––––––––––––––––––––––––– */
.u-full-width {
width: 100%;
box-sizing: border-box; }
.u-max-full-width {
max-width: 100%;
box-sizing: border-box; }
.u-pull-right {
float: right; }
.u-pull-left {
float: left; }
/* Misc
–––––––––––––––––––––––––––––––––––––––––––––––––– */
hr {
margin-top: 3rem;
margin-bottom: 3.5rem;
border-width: 0;
border-top: 1px solid #E1E1E1; }
/* Clearing
–––––––––––––––––––––––––––––––––––––––––––––––––– */
/* Self Clearing Goodness */
.container:after,
.row:after,
.u-cf {
content: "";
display: table;
clear: both; }
/* Media Queries
–––––––––––––––––––––––––––––––––––––––––––––––––– */
/*
Note: The best way to structure the use of media queries is to create the queries
near the relevant code. For example, if you wanted to change the styles for buttons
on small devices, paste the mobile query code up in the buttons section and style it
there.
*/
/* Larger than mobile */
@media (min-width: 400px) {}
/* Larger than phablet (also point when grid becomes active) */
@media (min-width: 550px) {}
/* Larger than tablet */
@media (min-width: 750px) {}
/* Larger than desktop */
@media (min-width: 1000px) {}
/* Larger than Desktop HD */
@media (min-width: 1200px) {}
future-todo/ControllableTalkNet/controllable_talknet.py
+1097
View File
File diff suppressed because it is too large Load Diff
future-todo/ControllableTalkNet/diffsvc_gui.py
+368
View File
@@ -0,0 +1,368 @@
import sys
import os
import io
import base64
import dash
from jupyter_dash import JupyterDash
import dash_core_components as dcc
import dash_html_components as html
from dash.exceptions import PreventUpdate
import gdown
import traceback
from scipy.io import wavfile
import numpy as np
import torch
sys.path.append("DiffSVC_inference_only")
from end2end import load_e2e_diffsvc, endtoend_from_path, write_to_file
app = JupyterDash(__name__)
UPLOAD_DIRECTORY = "/content"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
app.layout = html.Div(
children=[
html.H1(
children="DiffSVC",
style={
"font-family": "EquestriaWebfont",
"color": "#280e5f",
"font-size": "4em",
"text-align": "center",
"margin-top": "0em",
"margin-bottom": "0em",
},
),
html.Label("Character selection", htmlFor="speaker-dropdown"),
dcc.Dropdown(
id="speaker-dropdown",
options=[
{
"label": "Twilight Sparkle",
"value": "Twilight",
},
{
"label": "Discord",
"value": "Discord",
},
{
"label": "Pinkie Pie",
"value": "Pinkie",
},
{"label": "Nancy", "value": "Nancy"},
],
value=None,
style={
"max-width": "90vw",
"width": "20em",
"margin-bottom": "0.7em",
},
),
html.Label(
"Upload reference audio to " + UPLOAD_DIRECTORY,
htmlFor="reference-dropdown",
),
dcc.Store(id="current-f0s"),
dcc.Store(id="current-filename"),
dcc.Loading(
id="audio-loading",
children=[
html.Div(
[
html.Button(
"Update file list",
id="update-button",
style={
"margin-right": "10px",
},
),
dcc.Dropdown(
id="reference-dropdown",
options=[],
value=None,
style={
"max-width": "80vw",
"width": "30em",
},
),
],
style={
"width": "100%",
"display": "flex",
"align-items": "center",
"justify-content": "center",
"flex-direction": "row",
"margin-left": "50px",
"vertical-align": "middle",
},
),
html.Div(
id="audio-loading-output",
style={
"font-style": "italic",
"margin-bottom": "0.7em",
"text-align": "center",
},
),
],
type="default",
),
html.Div(
[
dcc.Checklist(
id="svc-options",
options=[
{"label": "Singing mode", "value": "no_pc"},
],
value=[],
)
],
style={
"width": "100%",
"display": "flex",
"align-items": "center",
"justify-content": "center",
"flex-direction": "row",
"margin-left": "50px",
"margin-bottom": "0.7em",
},
),
html.Div(
[
html.Label("Steps", htmlFor="linsteps-input"),
dcc.Input(
id="linsteps-input",
type="number",
value="1000",
style={"width": "7em", "margin-left": "10px"},
min=1,
max=1000,
step=1,
),
html.Label("Step strength", htmlFor="linend-input"),
dcc.Input(
id="linend-input",
type="number",
value="0.06",
style={"width": "7em", "margin-left": "10px"},
min=0.01,
max=0.3,
step=0.01,
),
],
style={
"display": "grid",
"grid-template-columns": "auto 100px",
"margin-bottom": "0.7em",
},
),
dcc.Loading(
html.Div(
[
html.Button(
"Generate",
id="gen-button",
),
html.Audio(
id="audio-out",
controls=True,
style={
"display": "none",
},
),
html.Div(
id="generated-info",
style={
"font-style": "italic",
},
),
],
style={
"width": "100%",
"display": "flex",
"align-items": "center",
"justify-content": "center",
"flex-direction": "column",
},
)
),
html.Footer(
children="""
Presented by the Pony Preservation Project. Models by Cookie.
""",
style={"margin-top": "2em", "font-size": "0.7em"},
),
],
style={
"width": "100%",
"display": "flex",
"align-items": "center",
"justify-content": "center",
"flex-direction": "column",
"background-color": "#FFF",
},
)
playback_style = {
"margin-top": "0.3em",
"margin-bottom": "0.3em",
"display": "block",
"width": "600px",
"max-width": "90vw",
}
playback_hide = {
"display": "none",
}
@app.callback(
dash.dependencies.Output("reference-dropdown", "options"),
[
dash.dependencies.Input("update-button", "n_clicks"),
],
)
def update_filelist(n_clicks):
filelist = []
supported_formats = [".wav", ".ogg", ".mp3", "flac"]
for x in os.listdir(UPLOAD_DIRECTORY):
if x[-4:].lower() in supported_formats:
filelist.append({"label": x, "value": x})
return filelist
@app.callback(
[
dash.dependencies.Output("audio-loading-output", "children"),
dash.dependencies.Output("current-filename", "data"),
],
[
dash.dependencies.Input("reference-dropdown", "value"),
],
)
def select_file(dropdown_value):
if dropdown_value is not None:
return [
"Selected " + dropdown_value,
dropdown_value,
]
else:
return ["No audio selected", None]
def download_model(drive_id, outname):
d = "https://drive.google.com/uc?id="
model_dir = os.path.join(UPLOAD_DIRECTORY, "models")
out_fullpath = os.path.join(model_dir, outname)
out_dir = os.path.dirname(out_fullpath)
if not os.path.exists(model_dir):
os.mkdir(model_dir)
if not os.path.exists(out_fullpath):
os.makedirs(out_dir, exist_ok=True)
gdown.download(
d + drive_id,
out_fullpath,
quiet=False,
)
return out_fullpath
svc_model, svc_name, last_end, last_step = None, None, None, None
@app.callback(
[
dash.dependencies.Output("audio-out", "src"),
dash.dependencies.Output("generated-info", "children"),
dash.dependencies.Output("audio-out", "style"),
],
[dash.dependencies.Input("gen-button", "n_clicks")],
[
dash.dependencies.State("speaker-dropdown", "value"),
dash.dependencies.State("svc-options", "value"),
dash.dependencies.State("current-filename", "data"),
dash.dependencies.State("linend-input", "value"),
dash.dependencies.State("linsteps-input", "value"),
],
)
def generate_audio(n_clicks, speaker, svc_options, wav_name, lin_end, lin_n_steps):
global svc_model, svc_name, last_end, last_step
if n_clicks is None:
raise PreventUpdate
if speaker is None:
return [None, "No character selected", playback_hide]
if wav_name is None:
return [None, "No reference audio selected", playback_hide]
try:
diffsvc_id = "1Uh17L1JtynFDgX9X6jKLPtD_t4IcTEp9"
asr_id = "1qt0pGhCbH0TltFFwSP2Oy3NClthcpr7N"
hifigan_id = "1QQT0HjMhGgDuPhyesZYbvck2lEY7GHAB"
hifigan_config_id = "1mi_O54zi6nW2eU6VuAO_Y4gv_Df-H9Yv"
diffsvc_path = download_model(
diffsvc_id,
os.path.join(diffsvc_id, "diffsvc_model"),
)
asr_path = download_model(
asr_id,
os.path.join(asr_id, "asr_model"),
)
hifigan_path = download_model(
hifigan_id,
os.path.join(hifigan_id, "hifigan_model"),
)
download_model(
hifigan_config_id,
os.path.join(hifigan_id, "config.json"),
)
if svc_name != diffsvc_id:
svc_model = load_e2e_diffsvc(
diffsvc_path=diffsvc_path,
dilated_asr_path=asr_path,
hifigan_path=hifigan_path,
device=DEVICE,
)
svc_name = diffsvc_id
if last_end != lin_end or last_step != lin_n_steps:
svc_model[0].generator.diffusion.set_noise_schedule(
1e-4, float(lin_end), int(lin_n_steps), device=DEVICE
)
last_end = lin_end
last_step = lin_n_steps
pred_audio = endtoend_from_path(
*svc_model,
os.path.join(UPLOAD_DIRECTORY, wav_name),
speaker,
"no_pc" not in svc_options,
"no_pc" not in svc_options,
"no_pc" not in svc_options,
t_max_step=int(lin_n_steps),
)
pred_audio *= 32768.0
buffer = io.BytesIO()
wavfile.write(
buffer,
svc_model[4].sampling_rate,
pred_audio.squeeze().cpu().numpy().astype(np.int16),
)
b64 = base64.b64encode(buffer.getvalue())
sound = "data:audio/x-wav;base64," + b64.decode("ascii")
return [sound, "Conversion complete", playback_style]
except Exception:
return [None, str(traceback.format_exc()), playback_hide]
if __name__ == "__main__":
app.run_server(
mode="external",
debug=True,
dev_tools_ui=True,
dev_tools_hot_reload=True,
threaded=True,
)
future-todo/ControllableTalkNet/model_lists/sortanon_models.json
+165
View File
@@ -0,0 +1,165 @@
[
{"source": "MLP",
"characters": [
{
"name": "Applejack",
"drive_id": "1kpEjZ3YqMN3chKSXODOqayEm581rxj4r",
"is_singing": false
},
{
"name": "Celestia",
"drive_id": "1whXXcnXu9XPcI60xIkTEofpaDYDOw5yB",
"is_singing": false
},
{
"name": "Chrysalis",
"drive_id": "1bb5jKAcQcEQbx1feVwT1UmAEgocINh-E",
"is_singing": false
},
{
"name": "Discord",
"drive_id": "1Cg9Oc_K9UDe5WgVDAcaCSbbBoo-Npj1E",
"is_singing": false
},
{
"name": "Fluttershy",
"drive_id": "1KgVnjrnxZTXgjnI56ilkq5G4UJCbbwZZ",
"is_singing": false
},
{
"name": "Luna",
"drive_id": "1_ztAbe5YArCMwyyQ_G9lUiz74ym5xJKC",
"is_singing": false
},
{
"name": "Maud Pie",
"drive_id": "132G6oD0HHPPn4t1H6IkYv18_F0UVLWgi",
"is_singing": false
},
{
"name": "Pinkie Pie",
"drive_id": "1CdYZ2r52mtgJsFs88U0ZViMSnzpQ_HRp",
"is_singing": false
},
{
"name": "Rainbow Dash",
"drive_id": "1k3EMXxLC0fLvfxzGbeP6B6plgu9hqCSx",
"is_singing": false
},
{
"name": "Rarity",
"drive_id": "1QWBvQSso4guc1LRUD40WRJ8DY2CfqHGK",
"is_singing": false
},
{
"name": "Spike",
"drive_id": "1TKFdmFLttjjzByj2fZW8J70ZHjR-RTwc",
"is_singing": false
},
{
"name": "Starlight Glimmer",
"drive_id": "1M1AMBq_xjwGTNzRUCXtSLIDJHbcSs3zR",
"is_singing": false
},
{
"name": "Sunset Shimmer",
"drive_id": "1x1aJt06lBvzUWRlxJ9CEKcFHxQxZPpST",
"is_singing": false
},
{
"name": "Trixie",
"drive_id": "1a3CYt0-oTTSFjxtZvAVMpClTmQteYua5",
"is_singing": false
},
{
"name": "Twilight Sparkle",
"drive_id": "1QnOliOAmerMUNuo2wXoH-YoainoSjZen",
"is_singing": false
},
{
"name": "Twilight Sparkle (whispering)",
"drive_id": "14_TUQVirITdyBh9etfNV8KFFhi_PUs30",
"is_singing": false
},
{
"name": "Zecora",
"drive_id": "1gL0hqqB7952Q1S185moQd_DRCFfIa3_g",
"is_singing": false
},
{
"name": "Twilight Sparkle (singing)",
"drive_id": "10CENYWV5ugTXZbnsldN6OKR7wkDEe7V7",
"is_singing": true
},
{
"name": "Pinkie Pie (singing)",
"drive_id": "1lo-8wI8iB3GArrzIDlaNTA1If8WX0OEf",
"is_singing": true
},
{
"name": "Applejack (singing)",
"drive_id": "1kegd0J2G4nogaX4IOB2O_vd8msaqMx8M",
"is_singing": true
},
{
"name": "Fluttershy (singing)",
"drive_id": "1BBdTHis91MwnHTt7tD_xtZ-nQ9SgvqD6",
"is_singing": true
},
{
"name": "Rarity (singing)",
"drive_id": "1EI7npfQeQYuP6b_Gy8iwA5zAg9Yj5MJA",
"is_singing": true
},
{
"name": "Rainbow Dash (singing)",
"drive_id": "1q4RgCEVJCSaR_1Dh0rxs6JVRJt30hQRK",
"is_singing": true
},
{
"name": "Apple Bloom",
"drive_id": "1Wm-7gqws0B3j8mtSQa77Tdk3RZwHEwT5",
"is_singing": false
},
{
"name": "Scootaloo",
"drive_id": "1YkV1VtP1w5XOx3jYYarrCKSzXCB_FLCy",
"is_singing": false
},
{
"name": "Sweetie Belle",
"drive_id": "1jLX0Py6j8uY93Fjf2l0HOZQYXiShfWUO",
"is_singing": false
},
{
"name": "Cozy Glow",
"drive_id": "1cUQQ_4KXTXbmH3MqwwyZu77SpMr20HAL",
"is_singing": false
},
{
"name": "Big McIntosh",
"drive_id": "1UzL5vo_ykmRdbi-5JOlsQzNKkoV4tJBP",
"is_singing": false
},
{
"name": "Cadance",
"drive_id": "1w98tMJPfGhWvl797gImMfErPXDz_hA-6",
"is_singing": false
},
{
"name": "Granny Smith",
"drive_id": "1FJuxLB6MhcHLtmEnOIxhN-yxJ85kmacu",
"is_singing": false
},
{
"name": "Shining Armor",
"drive_id": "1PUUS71w2ik0uuycNB30nXFze8C7O8OzY",
"is_singing": false
},
{
"name": "Tirek",
"drive_id": "1rcPDqgDeCIHGDdvfOo-fxfA1XeM4g3CB",
"is_singing": false
}
]}
]
future-todo/ControllableTalkNet/mycroft_talknet.py
+114
View File
@@ -0,0 +1,114 @@
import os
from flask import Flask, request, send_file
import sys
import torch
import numpy as np
from scipy.io import wavfile
import io
from nemo.collections.tts.models import TalkNetSpectModel
from nemo.collections.tts.models import TalkNetPitchModel
from nemo.collections.tts.models import TalkNetDursModel
import json
sys.path.append("hifi-gan")
from env import AttrDict
from meldataset import MAX_WAV_VALUE
from models import Generator
from denoiser import Denoiser
app = Flask(__name__)
RUN_PATH = os.path.dirname(os.path.realpath(__file__))
DEVICE = "cuda:0"
def load_hifigan(model_name, conf_name):
# Load HiFi-GAN
conf = os.path.join("hifi-gan", conf_name + ".json")
with open(conf) as f:
json_config = json.loads(f.read())
h = AttrDict(json_config)
torch.manual_seed(h.seed)
hifigan = Generator(h).to(torch.device(DEVICE))
state_dict_g = torch.load(model_name, map_location=torch.device(DEVICE))
hifigan.load_state_dict(state_dict_g["generator"])
hifigan.eval()
hifigan.remove_weight_norm()
denoiser = Denoiser(hifigan, mode="normal")
return hifigan, h, denoiser
def generate_json(input, outpath):
output = ""
sample_rate = 22050
lpath = input.split("|")[0].strip()
size = os.stat(lpath).st_size
x = {
"audio_filepath": lpath,
"duration": size / (sample_rate * 2),
"text": input.split("|")[1].strip(),
}
output += json.dumps(x) + "\n"
with open(outpath, "w", encoding="utf8") as w:
w.write(output)
def load_talknet(talknet_path):
with torch.no_grad():
tnmodel = TalkNetSpectModel.restore_from(talknet_path)
durs_path = os.path.join(os.path.dirname(talknet_path), "TalkNetDurs.nemo")
tndurs = TalkNetDursModel.restore_from(durs_path)
tnmodel.add_module("_durs_model", tndurs)
pitch_path = os.path.join(os.path.dirname(talknet_path), "TalkNetPitch.nemo")
tnpitch = TalkNetPitchModel.restore_from(pitch_path)
tnmodel.add_module("_pitch_model", tnpitch)
tnmodel.eval()
return tnmodel
def generate_audio(transcript, tnmodel, hifigan, denoiser):
with torch.no_grad():
tokens = tnmodel.parse(text=transcript.strip())
spect = tnmodel.generate_spectrogram(tokens=tokens)
y_g_hat = hifigan(spect.float())
audio = y_g_hat.squeeze()
audio = audio * MAX_WAV_VALUE
audio_denoised = denoiser(audio.view(1, -1), strength=35)[:, 0]
audio_np = audio_denoised.detach().cpu().numpy().reshape(-1).astype(np.int16)
buffer = io.BytesIO()
wavfile.write(buffer, 22050, audio_np)
return buffer
@app.route("/", methods=["GET"])
def get_check():
return "TalkNet server online"
@app.route("/api/tts", methods=["GET"])
def get_tts():
if "text" not in request.args:
return ""
transcript = request.args.get("text")
return send_file(
generate_audio(transcript, tnmodel, hifigan, denoiser),
attachment_filename="audio.wav",
mimetype="audio/x-wav",
)
if __name__ == "__main__":
hifigan, h, denoiser = load_hifigan(
os.path.join(
RUN_PATH, "models", "1QnOliOAmerMUNuo2wXoH-YoainoSjZen", "hifiganmodel"
),
"config_v1",
)
tnmodel = load_talknet(
os.path.join(
RUN_PATH, "models", "1QnOliOAmerMUNuo2wXoH-YoainoSjZen", "TalkNetSpect.nemo"
)
)
app.run(debug=False)
future-todo/ControllableTalkNet/requirements.txt
+28
View File
@@ -0,0 +1,28 @@
tensorflow==2.4.1
tensorboard==2.4.1
dash==1.21.0
dash-bootstrap-components==0.13.0
jupyter-dash==0.4.0
psola==0.0.1
wget==3.2
unidecode==1.2.0
pysptk==0.1.18
frozendict==2.0.3
torch==1.8.1+cu111
torchvision==0.9.1+cu111
torchaudio==0.8.1
torchtext==0.9.1
torch_stft==0.1.4
kaldiio==2.17.2
pydub==0.25.1
pyannote.audio==1.1.2
g2p_en==2.1.0
pesq==0.0.2
pystoi==0.3.3
crepe==0.0.12
resampy==0.2.2
ffmpeg-python==0.2.0
tqdm
gdown==3.13.0
editdistance==0.5.3
ipywidgets==7.6.3
future-todo/ControllableTalkNet/talknet_offline.py
+17
View File
@@ -0,0 +1,17 @@
if __name__ == "__main__":
import os
from controllable_talknet import *
if os.path.exists("/talknet/is_docker"):
app.run_server(
host="0.0.0.0",
mode="external",
debug=False,
threaded=True,
)
else:
app.run_server(
mode="external",
debug=False,
threaded=True,
)
future-todo/ControllableTalkNet/talknet_singer.py
+225
View File
@@ -0,0 +1,225 @@
from collections import OrderedDict
from typing import List
import torch
from hydra.utils import instantiate
from omegaconf import DictConfig
from pytorch_lightning import Trainer
from torch import nn
from torch.nn import functional as F
import numpy as np
from nemo.collections.asr.data.audio_to_text import AudioToCharWithDursF0Dataset
from nemo.collections.tts.helpers.helpers import get_mask_from_lengths
from nemo.collections.tts.models.base import SpectrogramGenerator
from nemo.collections.tts.modules.talknet import (
GaussianEmbedding,
MaskedInstanceNorm1d,
StyleResidual,
)
from nemo.core import Exportable
from nemo.core.classes import ModelPT, PretrainedModelInfo, typecheck
from nemo.core.neural_types import MelSpectrogramType, NeuralType
class TalkNetSingerModel(SpectrogramGenerator, Exportable):
"""TalkNet's mel spectrogram prediction pipeline."""
@property
def output_types(self):
return OrderedDict(
{"mel-spectrogram": NeuralType(("B", "D", "T"), MelSpectrogramType())}
)
def __init__(self, cfg: DictConfig, trainer: "Trainer" = None):
super().__init__(cfg=cfg, trainer=trainer)
typecheck.set_typecheck_enabled(enabled=False)
cfg = self._cfg
self.vocab = AudioToCharWithDursF0Dataset.make_vocab(
**cfg.train_ds.dataset.vocab
)
self.blanking = cfg.train_ds.dataset.blanking
self.preprocessor = instantiate(cfg.preprocessor)
self.embed = GaussianEmbedding(self.vocab, cfg.d_char)
self.norm_f0 = MaskedInstanceNorm1d(1)
self.res_f0 = StyleResidual(cfg.d_char, 1, kernel_size=3)
self.model = instantiate(cfg.model)
d_out = cfg.model.jasper[-1].filters
self.proj = nn.Conv1d(d_out, cfg.n_mels, kernel_size=1)
def forward(self, text, text_len, durs, f0):
x, x_len = self.embed(text, durs).transpose(1, 2), durs.sum(-1)
f0_sin = np.nan_to_num(np.sin(9.06472 * np.log(0.0363636 * f0.cpu())))
f0_cos = np.nan_to_num(np.cos(9.06472 * np.log(0.0363636 * f0.cpu())))
f0_note = torch.stack(
(torch.from_numpy(f0_sin), torch.from_numpy(f0_cos)), 1
).to(self.device)
f0, f0_mask = f0.clone(), f0 > 0.0
f0 = self.norm_f0(f0.unsqueeze(1), f0_mask)
f0[~f0_mask.unsqueeze(1)] = 0.0
x = self.res_f0(x, f0)
x = torch.cat((x, f0_note), 1)
y, _ = self.model(x, x_len)
mel = self.proj(y)
return mel
@staticmethod
def _metrics(true_mel, true_mel_len, pred_mel):
loss = F.mse_loss(pred_mel, true_mel, reduction="none").mean(dim=-2)
mask = get_mask_from_lengths(true_mel_len)
loss *= mask.float()
loss = loss.sum() / mask.sum()
return loss
def training_step(self, batch, batch_idx):
audio, audio_len, text, text_len, durs, f0, f0_mask = batch
mel, mel_len = self.preprocessor(audio, audio_len)
pred_mel = self(text=text, text_len=text_len, durs=durs, f0=f0)
loss = self._metrics(true_mel=mel, true_mel_len=mel_len, pred_mel=pred_mel)
train_log = {"train_loss": loss}
return {"loss": loss, "progress_bar": train_log, "log": train_log}
def validation_step(self, batch, batch_idx):
audio, audio_len, text, text_len, durs, f0, f0_mask = batch
mel, mel_len = self.preprocessor(audio, audio_len)
pred_mel = self(text=text, text_len=text_len, durs=durs, f0=f0)
loss = self._metrics(true_mel=mel, true_mel_len=mel_len, pred_mel=pred_mel)
val_log = {"val_loss": loss}
self.log_dict(
val_log, prog_bar=False, on_epoch=True, logger=True, sync_dist=True
)
@staticmethod
def _loader(cfg):
dataset = instantiate(cfg.dataset)
return torch.utils.data.DataLoader( # noqa
dataset=dataset,
collate_fn=dataset.collate_fn,
**cfg.dataloader_params,
)
def setup_training_data(self, cfg):
self._train_dl = self._loader(cfg)
def setup_validation_data(self, cfg):
self._validation_dl = self._loader(cfg)
def setup_test_data(self, cfg):
"""Omitted."""
pass
def parse(self, text: str, **kwargs) -> torch.Tensor:
return torch.tensor(self.vocab.encode(text)).long().unsqueeze(0).to(self.device)
def generate_spectrogram(self, tokens: torch.Tensor, **kwargs) -> torch.Tensor:
assert hasattr(self, "_durs_model") and hasattr(self, "_pitch_model")
if self.blanking:
tokens = [
AudioToCharWithDursF0Dataset.interleave(
x=torch.empty(len(t) + 1, dtype=torch.long, device=t.device).fill_(
self.vocab.blank
),
y=t,
)
for t in tokens
]
tokens = AudioToCharWithDursF0Dataset.merge(
tokens, value=self.vocab.pad, dtype=torch.long
)
text_len = torch.tensor(tokens.shape[-1], dtype=torch.long).unsqueeze(0)
durs = self._durs_model(tokens, text_len)
durs = durs.exp() - 1
durs[durs < 0.0] = 0.0
durs = durs.round().long()
# Pitch
f0_sil, f0_body = self._pitch_model(tokens, text_len, durs)
sil_mask = f0_sil.sigmoid() > 0.5
f0 = f0_body * self._pitch_model.f0_std + self._pitch_model.f0_mean
f0 = (~sil_mask * f0).float()
# Spect
mel = self(tokens, text_len, durs, f0)
return mel
def forward_for_export(self, tokens: torch.Tensor, text_len: torch.Tensor):
durs = self._durs_model(tokens, text_len)
durs = durs.exp() - 1
durs[durs < 0.0] = 0.0
durs = durs.round().long()
# Pitch
f0_sil, f0_body = self._pitch_model(tokens, text_len, durs)
sil_mask = f0_sil.sigmoid() > 0.5
f0 = f0_body * self._pitch_model.f0_std + self._pitch_model.f0_mean
f0 = (~sil_mask * f0).float()
# Spect
x, x_len = self.embed(tokens, durs).transpose(1, 2), durs.sum(-1)
f0_sin = np.nan_to_num(np.sin(9.06472 * np.log(0.0363636 * f0.cpu())))
f0_cos = np.nan_to_num(np.cos(9.06472 * np.log(0.0363636 * f0.cpu())))
f0_note = torch.stack(
(torch.from_numpy(f0_sin), torch.from_numpy(f0_cos)), 1
).to(self.device)
f0, f0_mask = f0.clone(), f0 > 0.0
f0 = self.norm_f0(f0.unsqueeze(1), f0_mask)
f0[~f0_mask.unsqueeze(1)] = 0.0
x = self.res_f0(x, f0)
x = torch.cat((x, f0_note), 1)
y, _ = self.model(x, x_len)
mel = self.proj(y)
return mel
def force_spectrogram(
self, tokens: torch.Tensor, durs: torch.Tensor, f0: torch.Tensor, **kwargs
) -> torch.Tensor:
if self.blanking:
tokens = [
AudioToCharWithDursF0Dataset.interleave(
x=torch.empty(len(t) + 1, dtype=torch.long, device=t.device).fill_(
self.vocab.blank
),
y=t,
)
for t in tokens
]
tokens = AudioToCharWithDursF0Dataset.merge(
tokens, value=self.vocab.pad, dtype=torch.long
)
text_len = torch.tensor(tokens.shape[-1], dtype=torch.long).unsqueeze(0)
durs_len = torch.tensor(durs.shape[-1], dtype=torch.long).unsqueeze(0)
assert text_len == durs_len
# Spect
mel = self(tokens, text_len, durs, f0)
return mel
@classmethod
def list_available_models(cls) -> "List[PretrainedModelInfo]":
"""
This method returns a list of pre-trained model which can be instantiated directly from NVIDIA's NGC cloud.
Returns:
List of available pre-trained models.
"""
list_of_models = []
model = PretrainedModelInfo(
pretrained_model_name="tts_en_talknet",
location=(
"https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_talknet/versions/1.0.0rc1/files"
"/talknet_spect.nemo"
),
description=(
"This model is trained on LJSpeech sampled at 22050Hz, and can be used to generate female "
"English voices with an American accent."
),
class_=cls, # noqa
aliases=["TalkNet-22050Hz"],
)
list_of_models.append(model)
return list_of_models
future-todo/README.md
View File
future-todo/vocodes_common_talknet.py
+484
View File
@@ -0,0 +1,484 @@
import sys
sys.path.append('waveglow/')
import shutil
import json
import numpy as np
import torch
#from tacotron2.hparams import create_hparams
#from tacotron2.model import Tacotron2
#from tacotron2.layers import TacotronSTFT
#from tacotron2.audio_processing import griffin_lim
#from tacotron2.text import text_to_sequence
#from waveglow.denoiser import Denoiser
from scipy.io.wavfile import write as write_wav
# Hifi-Gan
#from hifigan.env import AttrDict
#from hifigan.models import Generator
#from hifigan.denoiser import Denoiser as HifiDenoiser
#from hifigan.meldataset import mel_spectrogram
import resampy
import scipy
# For metadata
import subprocess
import magic
import os
# For talknet
from ControllableTalkNet.controllable_talknet import arpa_parse
from ControllableTalkNet.controllable_talknet import to_arpa
from nemo.collections.tts.models import TalkNetSpectModel
from nemo.collections.tts.models import TalkNetDursModel
from nemo.collections.tts.models import TalkNetPitchModel
from ControllableTalkNet.controllable_talknet import load_hifigan
from hifigan.meldataset import MAX_WAV_VALUE
from hifigan.meldataset import mel_spectrogram
def print_gpu_info():
print('========================================')
print('Python interpreter', sys.executable)
print('PyTorch version', torch.__version__)
print('CUDA Available?', torch.cuda.is_available())
print('CUDA Device count', torch.cuda.device_count())
#try:
# from tensorflow.python.client import device_lib
# print('local devices', str(device_lib.list_local_devices()).replace("\n", "\n "))
#except ImportError:
# print('no tensorflow - cannot list devices')
# pass
print('========================================', flush=True)
def load_tacotron_model(checkpoint_path):
hparams = create_hparams()
hparams.sampling_rate = 22050 # Don't change this
hparams.max_decoder_steps = 1000 # How long the audio will be before it cuts off (1000 is about 11 seconds)
hparams.gate_threshold = 0.1 # Model must be 90% sure the clip is over before ending generation (the higher this number is, the more likely that the AI will keep generating until it reaches the Max Decoder Steps)
model = Tacotron2(hparams)
print('Loading synthesizer at path: {}'.format(checkpoint_path))
model.load_state_dict(torch.load(checkpoint_path)['state_dict'])
_ = model.cuda().eval().half()
# NB(bt, 2021-05-31): Trying to get everything on the same device
device = "cuda:0"
model = model.to(device)
return model
def load_waveglow_model(checkpoint_path):
waveglow = torch.load(checkpoint_path)['model']
waveglow.cuda().eval().half()
for k in waveglow.convinv:
k.float()
denoiser = Denoiser(waveglow)
return waveglow
def load_hifigan_model(checkpoint_path, super_resolution = False):
conf_name = 'config_32k' if super_resolution else 'config_v1'
conf = os.path.join("hifigan", conf_name + ".json")
with open(conf) as f:
json_config = json.loads(f.read())
h = AttrDict(json_config)
torch.manual_seed(h.seed)
hifigan = Generator(h).to(torch.device("cuda"))
state_dict_g = torch.load(checkpoint_path, map_location=torch.device("cuda"))
hifigan.load_state_dict(state_dict_g["generator"])
hifigan.eval()
hifigan.remove_weight_norm()
denoiser = HifiDenoiser(hifigan, mode="normal")
return hifigan, h, denoiser
def preprocess_text(text, raw_input=True):
# TODO: Handle ARPAbet
pre_sequence = ""
for line in text.split("\n"):
if len(line) < 1:
continue
if raw_input:
if line[-1] != ";":
line = line + ";"
else:
#line = ARPA(line)
pass
pre_sequence += line
with torch.no_grad():
# save VRAM by not including gradients
sequence = np.array(text_to_sequence(pre_sequence, ['english_cleaners']))[None, :]
sequence = torch.autograd.Variable(torch.from_numpy(sequence)).cuda().long()
return sequence
class NumpyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.ndarray):
return obj.tolist()
return json.JSONEncoder.default(self, obj)
def save_spectorgram_json_file(mel_outputs_postnet, output_spectrogram_filename):
# squeeze(0) -> 3D to 2D (by removing the singular 1-length "wrapper" dimension)
# transpose() -> originally 80x{N}, we turn to {N}x80
mel_for_json = mel_outputs_postnet.cpu().numpy().squeeze(0).transpose()
mel_for_scaling = mel_for_json.copy()
max_value = np.amax(mel_for_scaling)
min_value = np.amin(mel_for_scaling)
mel_range = max_value - min_value
mel_for_scaling -= min_value
mel_for_scaling *= (255.0 / mel_range)
mel_for_scaling = mel_for_scaling.astype('int32')
json_data = {
'mel': mel_for_json,
'mel_scaled': mel_for_scaling,
}
with open(output_spectrogram_filename, 'w') as outfile:
json.dump(json_data, outfile, cls=NumpyEncoder)
def save_wav_audio_file(audio, output_audio_filename, sampling_rate=22050):
output_audio = audio[0].data.cpu().numpy().astype(np.float32)
write_wav(output_audio_filename, sampling_rate, output_audio)
def generate_metadata_file(output_audio_filename, output_metadata_filename):
# ==== METADATA (1) ====
command = [
"ffprobe",
"-loglevel", "quiet",
"-print_format", "json",
"-show_format",
"-show_streams",
output_audio_filename
]
pipe = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, err = pipe.communicate()
ffmpeg_metadata = json.loads(out)
print(ffmpeg_metadata)
# ==== METADATA (2) ====
mime_type = magic.from_file(output_audio_filename, mime=True)
file_size_bytes = os.path.getsize(output_audio_filename)
duration_millis = int(float(ffmpeg_metadata['format']['duration']) * 1000)
metadata = {
'duration_millis': duration_millis,
'mimetype': mime_type,
'file_size_bytes': file_size_bytes,
}
with open(output_metadata_filename, 'w') as json_file:
json.dump(metadata, json_file)
class TacotronWaveglowPipeline:
def __init__(self):
# Cache of "path" => loaded model
#self.tacotron_model_cache = {}
#self.waveglow = None
#self.waveglow_filename = None
#self.hifigan = None
#self.hifigan_denoiser = None
#self.hifigan_h = None
#self.hifigan_filename = None
#self.hifigan_super_resolution = None
#self.hifigan_super_resolution_denoiser = None
#self.hifigan_super_resolution_h = None
#self.hifigan_super_resolution_filename = None
pass
def maybe_load_hifigan(self, checkpoint_path):
if self.hifigan_filename == checkpoint_path:
print('Hifi-gan model already loaded into memory: {}'.format(checkpoint_path))
return
print('Dumping previous Hifi-gan model from memory: {}'.format(self.hifigan_filename))
print('Loading Hifi-gan model into memory: {}'.format(checkpoint_path), flush=True)
hifigan, h, denoiser = load_hifigan_model(checkpoint_path, False)
self.hifigan = hifigan
self.hifigan_denoiser = denoiser
self.hifigan_h = h
self.hifigan_filename = checkpoint_path
def maybe_load_hifigan_super_resolution(self, checkpoint_path):
if self.hifigan_super_resolution_filename == checkpoint_path:
print('Hifi-gan (SuperRes) model already loaded into memory: {}'.format(checkpoint_path))
return
print('Dumping previous Hifi-gan (SuperRes) model from memory: {}'.format(self.hifigan_super_resolution_filename))
print('Loading Hifi-gan (SuperRes) model into memory: {}'.format(checkpoint_path), flush=True)
hifigan, h, denoiser = load_hifigan_model(checkpoint_path, True)
self.hifigan_super_resolution = hifigan
self.hifigan_super_resolution_denoiser = denoiser
self.hifigan_super_resolution_h = h
self.hifigan_super_resolution_filename = checkpoint_path
def maybe_load_waveglow_model(self, checkpoint_path):
if self.waveglow_filename == checkpoint_path:
print('Waveglow model already loaded into memory: {}'.format(checkpoint_path))
return
print('Dumping previous waveglow model from memory: {}'.format(self.waveglow_filename))
print('Loading waveglow model into memory: {}'.format(checkpoint_path), flush=True)
self.waveglow = load_waveglow_model(checkpoint_path)
self.waveglow_filename = checkpoint_path
def maybe_load_tacotron_model_to_cache(self, checkpoint_path):
if checkpoint_path in self.tacotron_model_cache:
print('Tacotron model already loaded into memory: {}'.format(checkpoint_path))
return
print('Loading tacotron model into memory: {}'.format(checkpoint_path), flush=True)
model = load_tacotron_model(checkpoint_path)
self.tacotron_model_cache[checkpoint_path] = model
print('Now {} synthesizer models in memory'.format(len(self.tacotron_model_cache)))
def uncache_tacotron_model(self, checkpoint_path):
print('Currently {} synthesizer models in memory'.format(
len(self.tacotron_model_cache)))
print('Clearing synthesizer from memory: {}'.format(checkpoint_path))
self.tacotron_model_cache.pop(checkpoint_path, None)
print('Now reduced to {} synthesizer models in memory'.format(
len(self.tacotron_model_cache)))
def infer(self, args):
assert('synthesizer_checkpoint_path' in args)
assert('vocoder_type' in args)
assert('raw_text' in args)
assert('output_audio_filename' in args)
assert('output_spectrogram_filename' in args)
assert('output_metadata_filename' in args)
# ======================= BEGIN TALKNET EXPERIMENT ========================
print('load models')
with torch.no_grad():
# TODO: Singer model?
# tnmodel = TalkNetSingerModel.restore_from(singer_path
talknet_path = "/home/bt/models/talknet/david-attenborough/TalkNetSpect.nemo"
tnmodel = TalkNetSpectModel.restore_from(talknet_path)
durs_path = "/home/bt/models/talknet/david-attenborough/TalkNetDurs.nemo"
pitch_path = "/home/bt/models/talknet/david-attenborough/TalkNetPitch.nemo"
if os.path.exists(durs_path):
print('loading duration and pitch models')
tndurs = TalkNetDursModel.restore_from(durs_path)
tnmodel.add_module("_durs_model", tndurs)
tnpitch = TalkNetPitchModel.restore_from(pitch_path)
tnmodel.add_module("_pitch_model", tnpitch)
tnmodel.eval()
print('done loading model?')
# ==== ARPABET ====
print('Raw text: {}'.format(args['raw_text']))
text = args['raw_text']
token_list = arpa_parse(text, tnmodel)
print('token_list', token_list)
DEVICE = "cuda:0"
tokens = torch.IntTensor(token_list).view(1, -1).to(DEVICE)
print('tokens', tokens)
arpa = to_arpa(token_list)
print('arpa', arpa)
print('done with arpabet')
# ==== END ARPABET ====
# TODO: Either pitch prediction (if model given), or use a source audio file (TODO)
print("spectrogram estimation")
spect = tnmodel.generate_spectrogram(tokens=tokens)
# ==== HIFIGAN ====
print("load hifigan")
hifigan_path = "/home/bt/models/talknet/david-attenborough/hifiganmodel"
hifigan, h, denoiser = load_hifigan(hifigan_path, "config_v1")
hifipath = hifigan_path
y_g_hat = hifigan(spect.float())
audio = y_g_hat.squeeze()
audio = audio * MAX_WAV_VALUE
audio_denoised = denoiser(audio.view(1, -1), strength=35)[:, 0]
audio_np = (
audio_denoised.detach().cpu().numpy().reshape(-1).astype(np.int16)
)
# ==== HIFIGAN SR ====
print("load hifigan SR")
sr_path = "/home/bt/models/hifigan/Superres_Twilight_33000"
hifigan_sr, h2, denoiser_sr = load_hifigan(sr_path, "config_32k")
# TODO
# TODO
# TODO -- Auto-tuning
# TODO
# TODO
print("resample 32k")
# Resample to 32k
wave = resampy.resample(
audio_np,
h.sampling_rate,
h2.sampling_rate,
filter="sinc_window",
window=scipy.signal.windows.hann,
num_zeros=8,
)
wave_out = wave.astype(np.int16)
print("hifi SR")
# HiFi-GAN super-resolution
wave = wave / MAX_WAV_VALUE
wave = torch.FloatTensor(wave).to(torch.device(DEVICE))
new_mel = mel_spectrogram(
wave.unsqueeze(0),
h2.n_fft,
h2.num_mels,
h2.sampling_rate,
h2.hop_size,
h2.win_size,
h2.fmin,
h2.fmax,
)
y_g_hat2 = hifigan_sr(new_mel)
audio2 = y_g_hat2.squeeze()
audio2 = audio2 * MAX_WAV_VALUE
audio2_denoised = denoiser(audio2.view(1, -1), strength=35)[:, 0]
# High-pass filter, mixing and denormalizing
audio2_denoised = audio2_denoised.detach().cpu().numpy().reshape(-1)
b = scipy.signal.firwin(
101, cutoff=10500, fs=h2.sampling_rate, pass_zero=False
)
y = scipy.signal.lfilter(b, [1.0], audio2_denoised)
y *= 4.0 # superres strength
y_out = y.astype(np.int16)
y_padded = np.zeros(wave_out.shape)
y_padded[: y_out.shape[0]] = y_out
sr_mix = wave_out + y_padded
#buffer = io.BytesIO()
#wavfile.write(buffer, 32000, sr_mix.astype(np.int16))
#b64 = base64.b64encode(buffer.getvalue())
#sound = "data:audio/x-wav;base64," + b64.decode("ascii")
print("done!")
output_audio = sr_mix.astype(np.int16)
#rate = self.hifigan_super_resolution_h.sampling_rate
rate = h2.sampling_rate
write_wav(args['output_audio_filename'], rate, output_audio)
print('Saving spectrogram as JSON...')
save_spectorgram_json_file(spect, args['output_spectrogram_filename'])
print('Generating metadata file...')
generate_metadata_file(args['output_audio_filename'], args['output_metadata_filename'])
# ======================= END TALKNET EXPERIMENT ========================
#tacotron = self.tacotron_model_cache[args['synthesizer_checkpoint_path']]
#with torch.no_grad():
# mel_outputs, mel_outputs_postnet, _, alignments = tacotron.inference(sequence)
#if args['vocoder_type'] == 'hifigan-superres':
# self.vocode_hifigan_superres(args, mel_outputs_postnet)
#elif args['vocoder_type'] == 'waveglow':
# self.vocode_waveglow(args, mel_outputs_postnet)
#else:
# raise Exception('Wrong vocoder type: {}'.format(args['vocoder_type']))
def vocode_waveglow(self, args, mel_outputs_postnet):
print('Running waveglow...')
with torch.no_grad():
sigma = 0.8
audio = self.waveglow.infer(mel_outputs_postnet, sigma=sigma)
print('Encoding and saving audio...')
save_wav_audio_file(audio, args['output_audio_filename'])
def vocode_hifigan_superres(self, args, mel_outputs_postnet):
print('Running hifigan...')
with torch.no_grad():
y_g_hat = self.hifigan(mel_outputs_postnet.float())
MAX_WAV_VALUE = 32768.0
audio = y_g_hat.squeeze()
audio = audio * MAX_WAV_VALUE
audio_denoised = self.hifigan_denoiser(audio.view(1, -1), strength=35)[:, 0]
# Resample to 32k
audio_denoised = audio_denoised.cpu().numpy().reshape(-1)
normalize = (MAX_WAV_VALUE / np.max(np.abs(audio_denoised))) ** 0.9
audio_denoised = audio_denoised * normalize
wave = resampy.resample(
audio_denoised,
self.hifigan_h.sampling_rate,
self.hifigan_super_resolution_h.sampling_rate,
filter="sinc_window",
window=scipy.signal.windows.hann,
num_zeros=8,
)
wave_out = wave.astype(np.int16)
# HiFi-GAN super-resolution
print('Running hifigan super resolution...')
wave = wave / MAX_WAV_VALUE
wave = torch.FloatTensor(wave).to(torch.device("cuda"))
new_mel = mel_spectrogram(
wave.unsqueeze(0),
self.hifigan_super_resolution_h.n_fft,
self.hifigan_super_resolution_h.num_mels,
self.hifigan_super_resolution_h.sampling_rate,
self.hifigan_super_resolution_h.hop_size,
self.hifigan_super_resolution_h.win_size,
self.hifigan_super_resolution_h.fmin,
self.hifigan_super_resolution_h.fmax,
)
y_g_hat2 = self.hifigan_super_resolution(new_mel)
audio2 = y_g_hat2.squeeze()
audio2 = audio2 * MAX_WAV_VALUE
audio2_denoised = self.hifigan_denoiser(audio2.view(1, -1), strength=35)[:, 0]
# High-pass filter, mixing and denormalizing
audio2_denoised = audio2_denoised.cpu().numpy().reshape(-1)
b = scipy.signal.firwin(
101, cutoff=10500, fs=self.hifigan_super_resolution_h.sampling_rate, pass_zero=False
)
y = scipy.signal.lfilter(b, [1.0], audio2_denoised)
superres_strength = 4.0
y *= superres_strength
y_out = y.astype(np.int16)
y_padded = np.zeros(wave_out.shape)
y_padded[: y_out.shape[0]] = y_out
sr_mix = wave_out + y_padded
sr_mix = sr_mix / normalize
print('Encoding and saving audio...')
rate = self.hifigan_super_resolution_h.sampling_rate
output_audio = sr_mix.astype(np.int16)
write_wav(args['output_audio_filename'], rate, output_audio)
tts/NeMo/.dockerignore
+19
View File
@@ -0,0 +1,19 @@
__pycache__
*.pyc
*.pyo
*.pyd
.Python
env
pip-log.txt
pip-delete-this-directory.txt
.tox
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
*.log
.git
**/*.nemo
**/*.ckpt
tts/NeMo/.github/ISSUE_TEMPLATE/bug_report.md
Vendored
+42
View File
@@ -0,0 +1,42 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**Steps/Code to reproduce bug**
Please list *minimal* steps or code snippet for us to be able to reproduce the bug.
A helpful guide on on how to craft a minimal bug report http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment overview (please complete the following information)**
- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
- Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
**Environment details**
If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version
**Additional context**
Add any other context about the problem here.
Example: GPU model
tts/NeMo/.github/ISSUE_TEMPLATE/feature_request.md
Vendored
+25
View File
@@ -0,0 +1,25 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: feature request
assignees: okuchaiev
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
Provide a code snippet on how new APIs/changes would be used by others.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
tts/NeMo/.gitignore
+176
View File
@@ -0,0 +1,176 @@
# log and data files
*.model
*.pkl
#*.ipynb
output
result
*.pt
tests/data/asr
.DS_Store
bert.pt.json
work
runs
fastspeech_output
.hydra
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
**.pyc
# C extensions
*.so
# Distribution / packaging
.idea
.Python
wandb
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
#parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/build
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don’t work, or not
# install all needed dependencies.
#Pipfile.lock
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# VSCode project settins
.vscode/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
/docs/html
/docs/docs_zh/zh
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# Emacs backup files
*~
cifar-10-batches-py
*.tar.gz
# Test data.
tests/.data
tests/data
# outputs folder
examples/*/outputs
examples/*/NeMo_experiments
examples/*/nemo_experiments
examples/*/.hydra
examples/*/wandb
examples/*/data
wandb
dump.py
docs/sources/source/test_build/
# Checkpoints, config files and temporary files created in tutorials.
examples/neural_graphs/*.chkpt
examples/neural_graphs/*.yml
.hydra/
nemo_experiments/
tts/NeMo/.readthedocs.yml
+31
View File
@@ -0,0 +1,31 @@
# =============================================================================
# Copyright (c) 2020 NVIDIA. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =============================================================================
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required field.
version: 2
# Build documentation in the docs/ directory with Sphinx.
sphinx:
configuration: docs/source/conf.py
# Set the version of Python and requirements required to build your docs
python:
version: 3.7
install:
- requirements: requirements/requirements_docs.txt
tts/NeMo/CONTRIBUTING.md
+68
View File
@@ -0,0 +1,68 @@
# Contributions are welcome!
We do all of NeMo's development in the open. Contributions from NeMo community are welcome.
# Pull Requests (PR) Guidelines
**Send your PRs to the `main` branch**
1) Make sure your PR does one thing. Have a clear answer to "What does this PR do?".
2) Read General Principles and style guide below
3) Make sure you sign your commits. E.g. use ``git commit -s`` when before your commit
4) Make sure all unittests finish successfully before sending PR ``pytest`` or (if yor dev box does not have GPU) ``pytest --cpu`` from NeMo's root folder
5) Send your PR and request a review
## Unit tests
Quick tests (locally, while developing)
```
pytest
# If you don't have NVIDIA GPU do:
# pytest --cpu
```
Full tests, including pre-trained model downloads
```
pytest --with_downloads
```
## Whom should you ask for review:
1. For changes to NeMo's core: @ericharper, @titu1994, @blisc, or @okuchaiev
1. For changes to NeMo's ASR collection: @titu1994, @redoctopus, @jbalam-nv, or @okuchaiev
1. For changes to NeMo's NLP collection: @MaximumEntropy, @ericharper, @ekmb, @yzhang123, @VahidooX, @vladgets, or @okuchaiev
1. For changes to NeMo's TTS collection: @blisc or @stasbel, or @okuchaiev
Note that some people may self-assign to review your PR - in which case, please wait for them to add a review.
Your pull requests must pass all checks and peer-review before they can be merged.
# General principles
1. **User-oriented**: make it easy for end users, even at the cost of writing more code in the background
1. **Robust**: make it hard for users to make mistakes.
1. **Well-tested**: please add simple, fast unittests. Consider adding CI tests for end-to-end functionality.
1. **Reusable**: for every piece of code, think about how it can be reused in the future and make it easy to be reused.
1. **Readable**: code should be easier to read.
1. **Legal**: if you copy even one line of code from the Internet, make sure that the code allows the license that NeMo supports. Give credit and link back to the code.
1. **Sensible**: code should make sense. If you think a piece of code might be confusing, write comments.
## Python style
We use ``black`` as our style guide. To check whether your code will pass style check (from the NeMo's repo folder) run:
``python setup.py style`` and if it does not pass run ``python setup.py style --fix``.
1. Include docstrings for every class and method exposed to the user.
1. Use Python 3 type hints for every class and method exposed to the user.
1. Avoid wild import: ``from X import *`` unless in ``X.py``, ``__all__`` is defined.
1. Minimize the use of ``**kwargs``.
1. ``RaiseError`` is preferred to ``assert``. Write: ```if X: raise Error``` instead of ```assert X```.
1. Classes are preferred to standalone methods.
1. Methods should be atomic. A method shouldn't be longer than 75 lines, e.g. can be fit into the computer screen without scrolling.
1. If a method has arguments that don't fit into one line, each argument should be in its own line for readability.
1. Add ``__init__.py`` for every folder.
1. F-strings are prefered to formatted strings.
1. Loggers are preferred to print. In NeMo, you can use logger from ``from nemo.utils import logging``
1. Private functions (functions start with ``_``) shouldn't be called outside its host file.
1. If a comment lasts multiple lines, use ``'''`` instead of ``#``.
# Collections
Collection is a logical grouping of related Neural Modules. It is a grouping of modules that share a domain area or semantics.
When contributing module to a collection, please make sure it belongs to that category.
If you would like to start a new one and contribute back to the platform, you are very welcome to do so.
tts/NeMo/Dockerfile
+104
View File
@@ -0,0 +1,104 @@
# syntax=docker/dockerfile:experimental
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ARG BASE_IMAGE=nvcr.io/nvidia/pytorch:21.05-py3
# build an image that includes only the nemo dependencies, ensures that dependencies
# are included first for optimal caching, and useful for building a development
# image (by specifying build target as `nemo-deps`)
FROM ${BASE_IMAGE} as nemo-deps
# Ensure apt-get won't prompt for selecting options
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y \
libsndfile1 sox \
libfreetype6 \
python-setuptools swig \
python-dev ffmpeg && \
rm -rf /var/lib/apt/lists/*
# uninstall stuff from base container
RUN pip uninstall -y sacrebleu torchtext
# build torchaudio (change latest release version to match pytorch)
WORKDIR /tmp/torchaudio_build
RUN git clone --depth 1 --branch release/0.7 https://github.com/pytorch/audio.git && \
cd audio && \
BUILD_SOX=1 python setup.py install && \
cd .. && rm -r audio
# TODO: remove when 21.04 container is released
# build torchtext
WORKDIR /tmp/torchtext_build
RUN git clone --branch v0.8.1 https://github.com/pytorch/text.git && \
cd text && \
git submodule update --init --recursive && \
python setup.py clean install && \
cd .. && rm -r text
#install TRT tools: PT quantization support and ONNX graph optimizer
WORKDIR /tmp/trt_build
RUN git clone https://github.com/NVIDIA/TensorRT.git && \
cd TensorRT/tools/onnx-graphsurgeon && python setup.py install && \
cd ../pytorch-quantization && \
python setup.py install && \
rm -fr /tmp/trt_build
# install nemo dependencies
WORKDIR /tmp/nemo
COPY requirements .
RUN for f in $(ls requirements*.txt); do pip install --disable-pip-version-check --no-cache-dir -r $f; done
# install nemo_text_processing dependencies
COPY nemo_text_processing /tmp/nemo/nemo_text_processing/
RUN /bin/bash /tmp/nemo/nemo_text_processing/setup.sh
# copy nemo source into a scratch image
FROM scratch as nemo-src
COPY . .
# start building the final container
FROM nemo-deps as nemo
ARG NEMO_VERSION=1.0.2
# Check that NEMO_VERSION is set. Build will fail without this. Expose NEMO and base container
# version information as runtime environment variable for introspection purposes
RUN /usr/bin/test -n "$NEMO_VERSION" && \
/bin/echo "export NEMO_VERSION=${NEMO_VERSION}" >> /root/.bashrc && \
/bin/echo "export BASE_IMAGE=${BASE_IMAGE}" >> /root/.bashrc
RUN --mount=from=nemo-src,target=/tmp/nemo cd /tmp/nemo && pip install ".[all]" && \
python -c "import nemo.collections.asr as nemo_asr" && \
python -c "import nemo.collections.nlp as nemo_nlp" && \
python -c "import nemo.collections.tts as nemo_tts" && \
python -c "import nemo_text_processing.text_normalization as text_normalization"
# TODO: Remove once 21.04 container is base container
# install latest numba version
RUN conda update -c numba numba -y
# copy scripts/examples/tests into container for end user
WORKDIR /workspace/nemo
COPY scripts /workspace/nemo/scripts
COPY examples /workspace/nemo/examples
COPY tests /workspace/nemo/tests
COPY tutorials /workspace/nemo/tutorials
# COPY README.rst LICENSE /workspace/nemo/
RUN printf "#!/bin/bash\njupyter lab --no-browser --allow-root --ip=0.0.0.0" >> start-jupyter.sh && \
chmod +x start-jupyter.sh
tts/NeMo/Jenkinsfile
Vendored
+1451
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/LICENSE
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
tts/NeMo/README.rst
+174
View File
@@ -0,0 +1,174 @@
|status| |documentation| |license| |lgtm_grade| |lgtm_alerts| |black|
.. |status| image:: http://www.repostatus.org/badges/latest/active.svg
:target: http://www.repostatus.org/#active
:alt: Project Status: Active – The project has reached a stable, usable state and is being actively developed.
.. |documentation| image:: https://readthedocs.com/projects/nvidia-nemo/badge/?version=main
:alt: Documentation
:target: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/
.. |license| image:: https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg
:target: https://github.com/NVIDIA/NeMo/blob/master/LICENSE
:alt: NeMo core license and license for collections in this repo
.. |lgtm_grade| image:: https://img.shields.io/lgtm/grade/python/g/NVIDIA/NeMo.svg?logo=lgtm&logoWidth=18
:target: https://lgtm.com/projects/g/NVIDIA/NeMo/context:python
:alt: Language grade: Python
.. |lgtm_alerts| image:: https://img.shields.io/lgtm/alerts/g/NVIDIA/NeMo.svg?logo=lgtm&logoWidth=18
:target: https://lgtm.com/projects/g/NVIDIA/NeMo/alerts/
:alt: Total alerts
.. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
:alt: Code style: black
.. _main-readme:
**NVIDIA NeMo**
===============
Introduction
------------
NVIDIA NeMo is a conversational AI toolkit built for researchers working on automatic speech recognition (ASR), natural language processing (NLP), and text-to-speech synthesis (TTS).
The primary objective of NeMo is to help researchers from industry and academia to reuse prior work (code and pretrained models and make it easier to create new `conversational AI models <https://developer.nvidia.com/conversational-ai#started>`_.
`Introductory video. <https://www.youtube.com/embed/wBgpMf_KQVw>`_
Key Features
------------
* Speech processing
* `Automatic Speech recognition (ASR) <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/intro.html>`_: Jasper, QuartzNet, CitriNet, Conformer
* `Speech Classification <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speech_classification/intro.html>`_: MatchboxNet (command recognition), MarbleNet (voice activity detection)
* `Speaker Recognition <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_recognition/intro.html>`_: SpeakerNet
* `Speaker Diarization <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_diarization/intro.html>`_: MarbleNet + SpeakerNet
* `NGC collection of pre-trained speech processing models. <https://ngc.nvidia.com/catalog/collections/nvidia:nemo_asr>`_
* Natural Language Processing
* `Compatible with Hugging Face Transformers and NVIDIA Megatron <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/megatron_finetuning.html>`_
* `Neural Machine Translation (NMT) <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/machine_translation.html>`_
* `Punctuation and Capitalization <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/punctuation_and_capitalization.html>`_
* `Token classification (named entity recognition) <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/token_classification.html>`_
* `Text classification <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/text_classification.html>`_
* `Joint Intent and Slot Classification <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/joint_intent_slot.html>`_
* `BERT pre-training <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/bert_pretraining.html>`_
* `Question answering <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/question_answering.html>`_
* `GLUE benchmark <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/glue_benchmark.html>`_
* `Information retrieval <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/information_retrieval.html>`_
* `NGC collection of pre-trained NLP models. <https://ngc.nvidia.com/catalog/collections/nvidia:nemo_nlp>`_
* `Speech synthesis (TTS) <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tts/intro.html#>`_
* Spectrogram generation: Tacotron2, GlowTTS, FastSpeech2, FastPitch,
* Vocoders: WaveGlow, SqueezeWave, UniGlow, MelGAN, HiFiGAN
* End-to-end speech generation: FastPitch_HifiGan_E2E, FastSpeech2_HifiGan_E2E
* `NGC collection of pre-trained TTS models. <https://ngc.nvidia.com/catalog/collections/nvidia:nemo_tts>`_
Built for speed, NeMo can utilize NVIDIA's Tensor Cores and scale out training to multiple GPUs and multiple nodes.
Requirements
------------
1) Python 3.6, 3.7 or 3.8
2) Pytorch 1.8.1 or above
3) NVIDIA GPU for training
Documentation
-------------
.. |main| image:: https://readthedocs.com/projects/nvidia-nemo/badge/?version=main
:alt: Documentation Status
:scale: 100%
:target: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/
.. |v1.0.2| image:: https://readthedocs.com/projects/nvidia-nemo/badge/?version=v1.0.2
:alt: Documentation Status
:scale: 100%
:target: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.2/
.. |stable| image:: https://readthedocs.com/projects/nvidia-nemo/badge/?version=stable
:alt: Documentation Status
:scale: 100%
:target: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/
+---------+-------------+----------------------------------------------------------------------------------------------------------------------------------+
| Version | Status | Description |
+=========+=============+==================================================================================================================================+
| Latest | |main| | `Documentation of the latest (i.e. main) branch. <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/>`_ |
+---------+-------------+----------------------------------------------------------------------------------------------------------------------------------+
| Next | |v1.0.2| | `Documentation of the most recent release: v1.0.2 <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.2/>`_ |
+---------+-------------+----------------------------------------------------------------------------------------------------------------------------------+
| Stable | |stable| | `Documentation of the stable (i.e. stable) branch. <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/>`_ |
+---------+-------------+----------------------------------------------------------------------------------------------------------------------------------+
Tutorials
---------
A great way to start with NeMo is by checking `one of our tutorials <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.2/starthere/tutorials.html>`_.
Getting help with NeMo
----------------------
FAQ can be found on NeMo's `Discussions board <https://github.com/NVIDIA/NeMo/discussions>`_. You are welcome to ask questions or start discussions there.
Installation
------------
Pip
~~~
Use this installation mode if you want the latest released version.
.. code-block:: bash
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython
pip install nemo_toolkit['all']
Pip from source
~~~~~~~~~~~~~~~
Use this installation mode if you want the a version from particular GitHub branch (e.g main).
.. code-block:: bash
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython
python -m pip install git+https://github.com/NVIDIA/NeMo.git@{BRANCH}#egg=nemo_toolkit[all]
From source
~~~~~~~~~~~
Use this installation mode if you are contributing to NeMo.
.. code-block:: bash
apt-get update && apt-get install -y libsndfile1 ffmpeg
git clone https://github.com/NVIDIA/NeMo
cd NeMo
./reinstall.sh
Docker containers:
~~~~~~~~~~~~~~~~~~
If you chose to work with main branch, we recommend using NVIDIA's PyTorch container version 21.05-py3 and then installing from GitHub.
.. code-block:: bash
docker run --gpus all -it --rm -v <nemo_github_folder>:/NeMo --shm-size=8g \
-p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit \
stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:21.05-py3
Examples
--------
Many example can be found under `"Examples" <https://github.com/NVIDIA/NeMo/tree/main/examples>`_ folder.
Contributing
------------
We welcome community contributions! Please refer to the `CONTRIBUTING.md <https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md>`_ CONTRIBUTING.md for the process.
License
-------
NeMo is under `Apache 2.0 license <https://github.com/NVIDIA/NeMo/blob/main/LICENSE>`_.
tts/NeMo/__init__.py
View File
tts/NeMo/docs/.nojekyll
View File
tts/NeMo/docs/Makefile
+216
View File
@@ -0,0 +1,216 @@
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
.PHONY: help
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
.PHONY: clean
clean:
rm -rf $(BUILDDIR)/*
.PHONY: html
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
.PHONY: dirhtml
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
.PHONY: singlehtml
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
.PHONY: pickle
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
.PHONY: json
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
.PHONY: htmlhelp
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
.PHONY: qthelp
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/OpenSeq2Seq.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/OpenSeq2Seq.qhc"
.PHONY: applehelp
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
.PHONY: devhelp
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/OpenSeq2Seq"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/OpenSeq2Seq"
@echo "# devhelp"
.PHONY: epub
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
.PHONY: latex
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
.PHONY: latexpdf
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: latexpdfja
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: text
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
.PHONY: man
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
.PHONY: texinfo
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
.PHONY: info
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
.PHONY: gettext
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
.PHONY: changes
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
.PHONY: linkcheck
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
.PHONY: doctest
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
.PHONY: coverage
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
.PHONY: xml
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
.PHONY: pseudoxml
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
tts/NeMo/docs/source/_static/css/custom.css
+199
View File
@@ -0,0 +1,199 @@
/* Width of template */
.wy-nav-content {
max-width: 1200px !important;
}
/* Standard Text Formatting */
h1
{
color: #76b900;
text-align: center;
background-color: #333333;
}
h2
{
color: #ffffff;
background-color: #76b900;
Padding: 5px;
}
h3
{
padding-top: 0px;
border-top: solid 3px #76b900;
border-bottom: solid 3px #76b900;
}
p
{
margin-bottom: 24px;
}
/* Link Colors */
a
{
color: #76b900;
}
a:visited
{
color: #218219;
}
/* ----------------------------------------------TABLES--------------------------------------- */
/* Table head Color */
thead td
{
background-color: #333333 !important;
}
.row-odd p
{
/*padding-bottom: 0px;*/
/*margin-bottom: 0px;*/
}
/* even rows*/
.row-even tr
{
background-color: #e5f1e6 !important;
}
/* odd rows*/
.wy-table-responsive table tr {
background-color: #ffffff !important;
}
.wy-table-responsive table td {
white-space: normal;
}
/* Removes bottom margin in tables*/
.rst-content .line-block {
margin-bottom: 0px;
}
.wy-table-responsive {
overflow: visible !important;
}
/* reduces the size of text in multiline table columns. */
.rst-content table.docutils td
{
font-size: 80%;
}
.rst-content dl:not(.docutils) dt
{
background-color: inherit;
color: #000000;
border-top: solid 0px #000000;
}
.rst-content dl:not(.docutils) dt:before {
color: #333333;
}
.rst-content .line-block {
margin-bottom: 0px;
}
.wy-side-nav-search, .wy-nav-top
{
background-color: #000000;
padding: 0;
}
.wy-side-nav-search img
{
padding: 0px;
padding: 0px 0px;
margin-bottom: 0;
}
.wy-side-nav-search input[type=text]
{
border-radius: 0px;
}
.wy-menu-vertical p.caption
{
color: #76b900;
}
.wy-side-nav-search>a img.logo, .wy-side-nav-search .wy-dropdown>a img.logo
{
margin: 0px 0px 0px 0px;
}
.wy-nav-content
{
margin: 0;
min-height: 100%;
height: 100%;
background: #ffffff;
}
/* List (numbered, bulleted) padding Fix */
.wy-plain-list-decimal li
{
margin-top: -6px;
margin-bottom: -6px;
}
.rst-content .section ol.loweralpha
{
margin-top: -6px;
margin-bottom: 12px;
}
.wy-plain-list-disc, .rst-content .toctree-wrapper ul, article ul
{
margin-top: 0px !important;
margin-bottom: 12px;
}
/* Alert Boxes */
/* Background color of Alert Box Title */
.rst-content .section ul
{
margin-top: -12px;
margin-bottom: 16px;
}
.wy-alert.wy-alert-info .wy-alert-title, .rst-content .note .wy-alert-title, .rst-content .wy-alert-info.attention .wy-alert-title, .rst-content .wy-alert-info.caution .wy-alert-title, .rst-content .wy-alert-info.danger .wy-alert-title, .rst-content .wy-alert-info.error .wy-alert-title, .rst-content .wy-alert-info.hint .wy-alert-title, .rst-content .wy-alert-info.important .wy-alert-title, .rst-content .wy-alert-info.tip .wy-alert-title, .rst-content .wy-alert-info.warning .wy-alert-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-info.admonition-todo .wy-alert-title, .rst-content .wy-alert-info.admonition .wy-alert-title, .wy-alert.wy-alert-info .rst-content .admonition-title, .rst-content .wy-alert.wy-alert-info .admonition-title, .rst-content .note .admonition-title, .rst-content .wy-alert-info.attention .admonition-title, .rst-content .wy-alert-info.caution .admonition-title, .rst-content .wy-alert-info.danger .admonition-title, .rst-content .wy-alert-info.error .admonition-title, .rst-content .wy-alert-info.hint .admonition-title, .rst-content .wy-alert-info.important .admonition-title, .rst-content .wy-alert-info.tip .admonition-title, .rst-content .wy-alert-info.warning .admonition-title, .rst-content .seealso .admonition-title, .rst-content .wy-alert-info.admonition-todo .admonition-title, .rst-content .wy-alert-info.admonition .admonition-title
{
background: #76b900;
}
/* Background and Font Color of Alert Box Main Body*/
.wy-alert.wy-alert-info, .rst-content .note, .rst-content .wy-alert-info.attention, .rst-content .wy-alert-info.caution, .rst-content .wy-alert-info.danger, .rst-content .wy-alert-info.error, .rst-content .wy-alert-info.hint, .rst-content .wy-alert-info.important, .rst-content .wy-alert-info.tip, .rst-content .wy-alert-info.warning, .rst-content .seealso, .rst-content .wy-alert-info.admonition-todo, .rst-content .wy-alert-info.admonition {
background: #333333;
color: #999999;
}
.section
{
margin-top: 50px;
}
tts/NeMo/docs/source/_static/js/pk_scripts.js
+19
View File
@@ -0,0 +1,19 @@
document.addEventListener("DOMContentLoaded", function () {
var params = window.location.search.substring(1).split("&").reduce(function (params, param) {
if (!param) {
return params;
}
var values = param.split("=");
var name = values[0];
var value = values[1];
params[name] = value;
return params;
}, {});
var form = document.getElementById("feedback-form");
for (var name in params) {
var input = form.querySelector("[name=" + name + "]");
input.value = params[name];
}
});
tts/NeMo/docs/source/asr/api.rst
+203
View File
@@ -0,0 +1,203 @@
NeMo ASR collection API
=======================
Model Classes
-------------
.. autoclass:: nemo.collections.asr.models.EncDecCTCModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecCTCModelBPE
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTBPEModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecSpeakerLabelModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
Modules
-------
.. autoclass:: nemo.collections.asr.modules.ConvASREncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConvASRDecoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConvASRDecoderClassification
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.SpeakerDecoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConformerEncoder
:show-inheritance:
:members:
Parts
-----
.. autoclass:: nemo.collections.asr.parts.submodules.jasper.JasperBlock
:show-inheritance:
:members:
Mixins
------
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin
:show-inheritance:
:members:
Datasets
--------
Character Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToCharDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToCharWithDursF0Dataset
:show-inheritance:
:members:
Subword Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToBPEDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset
:show-inheritance:
:members:
Audio Preprocessors
-------------------
.. autoclass:: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.AudioToMFCCPreprocessor
:show-inheritance:
:members:
Audio Augmentors
----------------
.. autoclass:: nemo.collections.asr.modules.SpectrogramAugmentation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.CropOrPadSpectrogramAugmentation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.SpeedPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TimeStretchPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.GainPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ImpulsePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ShiftPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.NoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.WhiteNoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.RirAndNoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TranscodePerturbation
:show-inheritance:
:members:
Miscellaneous Classes
---------------------
RNNT Decoding
~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.metrics.rnnt_wer.RNNTDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.metrics.rnnt_wer_bpe.RNNTBPEDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamRNNTInfer
:show-inheritance:
:members:
Hypotheses
~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis
:show-inheritance:
:no-members:
.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses
:show-inheritance:
:no-members:
tts/NeMo/docs/source/asr/asr_all.bib
+980
View File
@@ -0,0 +1,980 @@
@article{matchboxnet,
title={{MatchboxNet}: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition},
author={Majumdar, Somshubra and Ginsburg, Boris},
journal={Proc. Interspeech 2020},
year={2020}
}
@article{marblenet,
title={MarbleNet: Deep 1D Time-Channel Separable Convolutional Neural Network for Voice Activity Detection},
author={Jia, Fei and Majumdar, Somshubra and Ginsburg, Boris},
journal={arXiv preprint arXiv:2010.13886},
year={2020}
}
@inproceedings{panayotov2015librispeech,
title={Librispeech: an ASR corpus based on public domain audio books},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on},
pages={5206--5210},
year={2015},
organization={IEEE}
}
@article{luong17,
author = {Minh{-}Thang Luong and Eugene Brevdo and Rui Zhao},
title = {Neural Machine Translation (seq2seq) Tutorial},
journal = {https://github.com/tensorflow/nmt},
year = {2017},
}
@INPROCEEDINGS{LaurentSeqWiseBN,
author={C. {Laurent} and G. {Pereyra} and P. {Brakel} and Y. {Zhang} and Y. {Bengio}},
booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={Batch normalized recurrent neural networks},
year={2016},
volume={},
number={},
pages={2657-2661},
keywords={feedforward neural nets;learning (artificial intelligence);recurrent neural nets;speech recognition;batch normalized recurrent neural networks;RNN;sequential data;long-term dependency learning;convergence rate improvement;intermediate representation normalization;feedforward neural networks;speech recognition task;language modeling;training criterion;Training;Recurrent neural networks;Convergence;Speech recognition;Computer architecture;Speech;batch normalization;RNN;LSTM;optimization},
doi={10.1109/ICASSP.2016.7472159},
ISSN={2379-190X},
month={March},}
@article{graves2005,
author = {Alex Graves and Jurgen Schmidhuber},
title = {Framewise phoneme classification with bidirectional LSTM and other neural network architectures},
journal = {Neural Networks, vol. 18},
pages={602–-610},
year = {2005},
}
@inproceedings{graves2006,
title={Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks},
author={Graves, Alex and Fern{\'a}ndez, Santiago and Gomez, Faustino and Schmidhuber, J{\"u}rgen},
booktitle={Proceedings of the 23rd international conference on Machine learning},
pages={369--376},
year={2006},
organization={ACM}
}
@article{li2019jasper,
title={Jasper: An End-to-End Convolutional Neural Acoustic Model},
author={Li, Jason and Lavrukhin, Vitaly and Ginsburg, Boris and Leary, Ryan and Kuchaiev, Oleksii and Cohen, Jonathan M and Nguyen, Huyen and Gadde, Ravi Teja},
journal={arXiv preprint arXiv:1904.03288},
year={2019}
}
@misc{ardila2019common,
title={Common Voice: A Massively-Multilingual Speech Corpus},
author={Rosana Ardila and Megan Branson and Kelly Davis and Michael Henretty and Michael Kohler and Josh Meyer and Reuben Morais and Lindsay Saunders and Francis M. Tyers and Gregor Weber},
year={2019},
eprint={1912.06670},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@article{graves2012,
title={Sequence Transduction with Recurrent Neural Networks},
author={Graves, Alex},
journal={arXiv preprint arXiv:1211.3711},
year={2012}
}
@article{graves2013,
title={Generating sequences with recurrent neural networks},
author={Graves, Alex},
journal={arXiv preprint arXiv:1308.0850},
year={2013}
}
@article{sergeev2018horovod,
title={Horovod: fast and easy distributed deep learning in TensorFlow},
author={Sergeev, Alexander and Del Balso, Mike},
journal={arXiv preprint arXiv:1802.05799},
year={2018}
}
@misc{NVVolta,
title = {NVIDIA TESLA V100 GPU ARCHITECTURE},
howpublished = {\url{http://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf}},
note = {Accessed: 2018-10-09}
}
@article{NVTuring,
title = {NVIDIA TURING GPU ARCHITECTURE},
howpublished = {\url{https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf}},
author = {NVIDIA},
year = {2018},
note = {Accessed: 2018-10-09}
}
@misc{Rygaard2015,
title = {Using Synthesized Speech to Improve Speech Recognition for Low-Resource Languages},
author = {Luise Valentin Rygaard},
howpublished = {\url{https://parasol.tamu.edu/dreu2015/Rygaard/report.pdf}},
year = {2015},
}
@misc{OpenSeq2Seq,
title = {OpenSeq2Seq: extensible toolkit for distributed and mixed precision training of sequence-to-sequence models},
author = {Kuchaiev, Oleksii and Ginsburg, Boris and Gitman, Igor and Lavrukhin,Vitaly and Case, Carl and Micikevicius, Paulius},
howpublished = {\url{https://arxiv.org/abs/1805.10387}},
year = {2018},
}
@misc{MPGuide,
title = {Training with Mixed Precision},
howpublished = {\url{http://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/}},
note = {Accessed: 2018-04-06},
}
@misc{Mozilla,
title = {Mozilla: A Journey to less than 10\% Word Error Rate},
howpublished = {\url{https://hacks.mozilla.org/2017/11/a-journey-to-10-word-error-rate/}},
note = {Accessed: 2018-04-06},
}
@article{Waibel1989,
title={A time-delay neural network architecture for isolated word recognition},
author={Waibel, Alexander, and Hanazawa, Toshiyki and Hinton,Geoffrey and Shirano, Kiyohiro and Lang, Kevin },
journal={IEEE Trans. on Acoustics, Speech and Signal Processing},
year={1989}
}
@article{Lang1990,
title={A time-delay neural network architecture for isolated word recognition},
author={Lang, Kevin and Waibel, Alexander, and Hinton,Geoffrey },
journal={Neural Networks},
year={1990}
}
@book{Bengio1996,
Author = {Bengio, Y.},
Publisher = {International Thomson Computer Press},
Title = {Neural Networks for Speech and Sequence Recognition},
Year = {1996}
}
@article{Bengio1992,
title={Global optimization of a neural network-hidden Markov model hybrid},
author={Bengio, Y., and De Mori, R., and Flammia, G., and Kompe, R. },
journal={IEEE Transactions on Neural Networks, 3(2), 252–259},
year={1992}
}
@article{Bourlard1994,
title={Connectionist speech recognition: a hybrid approach},
author={Bourlard, H. A. and Morgan, N.},
journal={volume 247 Springer },
year={1994}
}
@article{srivastava14a,
author = {Nitish Srivastava, and Geoffrey Hinton, and Alex Krizhevsky, and Ilya Sutskever, and Ruslan Salakhutdinov},
title = {Dropout: A Simple Way to Prevent Neural Networks from Overfitting},
journal = {Journal of Machine Learning Research},
year = {2014},
volume = {15},
pages = {1929-1958},
url = {http://jmlr.org/papers/v15/srivastava14a.html}
}
@article{Hinton2012,
title={Deep Neural Networks for Acoustic Modeling in Speech Recognition},
author={ Hinton,Geoffrey and Deng, Li and Yu, Dong and Dahl,George and Mohamed,Abdel-rahman and Jaitly, Navdeep and Senior, Andrew and Vanhoucke, Vincent and Nguyen, Patrick and Kingsbury, Brian and Sainath, Tara},
journal={IEEE Signal Processing Magazine},
year={2012}
}
@article{Graves2014,
title={Towards End-to-End Speech Recognition with Recurrent Neural Networks},
author={Graves, Alex and Jaitly, Navdeep},
journal={International Conference on Machine Learning},
year={2014}
}
@article{Chorowski2014,
title={End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results},
author={ Chorowski, Jan, and Bahdanau, Dzmitry , and Cho, Kyunghyun , and Bengio, Yoshua },
journal={Neural Information Processing Systems: Workshop Deep Learning and Representation Learning Workshop },
year={2014}
}
@article{Sak2014,
title={Long short-term memory recurrent neural network architectures for large scale acoustic modeling},
author={Sak, Hasim and Senior, Andrew and Beaufays, Francoise },
journal={Interspeech 2014},
year={2014}
}
@article{Ko2015,
title={Audio Augmentation for Speech Recognition},
author={Tom, Ko and Vijayaditya, Peddinti and Daniel, Povey
and Sanjeev, Khudanpur },
journal={Interspeech 2015},
year={2015}
}
@article{Tjandra2017,
title={Listening while Speaking: Speech Chain by Deep Learning},
author={Andros, Tjandra and Sakriani, Sakti and Satoshi, Nakamura },
journal={ASRU 2017},
year={2017}
}
@article{Tjandra2018,
title={Machine Speech Chain with One-shot Speaker Adaptation},
author={Andros, Tjandra and Sakriani, Sakti and Satoshi, Nakamura },
journal={Interspeech 2018},
year={2018}
}
@article{bahdanau2014neural,
title={Neural machine translation by jointly learning to align and translate},
author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
journal={arXiv preprint arXiv:1409.0473},
year={2014}
}
@article{cho2014learning,
title={Learning phrase representations using RNN encoder-decoder for statistical machine translation},
author={Cho, Kyunghyun and Van Merri{\"e}nboer, Bart and Gulcehre, Caglar and Bahdanau, Dzmitry and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua},
journal={arXiv preprint arXiv:1406.1078},
year={2014}
}
@article{rush2015neural,
title={A neural attention model for abstractive sentence summarization},
author={Rush, Alexander M and Chopra, Sumit and Weston, Jason},
journal={arXiv preprint arXiv:1509.00685},
year={2015}
}
@article{micikevicius2017mixed,
title={Mixed precision training},
author={Micikevicius, Paulius and Narang, Sharan and Alben, Jonah and Diamos, Gregory and Elsen, Erich and Garcia, David and Ginsburg, Boris and Houston, Michael and Kuchaev, Oleksii and Venkatesh, Ganesh and others},
journal={arXiv preprint arXiv:1710.03740},
year={2017}
}
@ARTICLE{Britz:2017,
author = {{Britz}, Denny and {Goldie}, Anna and {Luong}, Thang and {Le}, Quoc},
title = "{Massive Exploration of Neural Machine Translation Architectures}",
journal = {ArXiv e-prints arXiv:1703.03906},
archivePrefix = "arXiv",
eprinttype = {arxiv},
eprint = {1703.03906},
primaryClass = "cs.CL",
keywords = {Computer Science - Computation and Language},
year = 2017,
month = mar,
}
@inproceedings{vaswani2017attention,
title={Attention is all you need},
author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia},
booktitle={Advances in Neural Information Processing Systems},
pages={6000--6010},
year={2017}
}
@inproceedings{abadi2016tensorflow,
title={TensorFlow: A System for Large-Scale Machine Learning.},
author={Abadi, Mart{\'\i}n and Barham, Paul and Chen, Jianmin and Chen, Zhifeng and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Irving, Geoffrey and Isard, Michael and others},
booktitle={OSDI},
volume={16},
pages={265--283},
year={2016}
}
@article{tensor2tensor,
author = {Ashish Vaswani and Samy Bengio and Eugene Brevdo and Francois Chollet and Aidan N. Gomez and Stephan Gouws and Llion Jones and \L{}ukasz Kaiser and Nal Kalchbrenner and Niki Parmar and Ryan Sepassi and
Noam Shazeer and Jakob Uszkoreit},
title = {Tensor2Tensor for Neural Machine Translation},
journal = {CoRR},
volume = {abs/1803.07416},
year = {2018},
url = {http://arxiv.org/abs/1803.07416},
}
@article{gehring2017convs2s,
author = {Gehring, Jonas, and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N},
title = "{Convolutional Sequence to Sequence Learning}",
journal = {ArXiv e-prints arXiv:1705.03122},
archivePrefix = "arXiv",
eprinttype = {arxiv},
eprint = {1705.03122},
primaryClass = "cs.CL",
keywords = {Computer Science - Computation and Language},
year = 2017,
month = May,
}
@inproceedings{chan2015,
title={Listen, attend and spell},
author={Chan, William and Jaitly, Navdeep and Le, Quoc V and Vinyals, Oriol},
booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on},
pages={5206--5210},
year={2016},
organization={IEEE}
}
@inproceedings{xu2015show,
title={Show, attend and tell: Neural image caption generation with visual attention},
author={Xu, Kelvin and Ba, Jimmy and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhudinov, Ruslan and Zemel, Rich and Bengio, Yoshua},
booktitle={International Conference on Machine Learning},
pages={2048--2057},
year={2015}
}
@incollection{Sutskever2014,
title = {Sequence to Sequence Learning with Neural Networks},
author = {Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V},
booktitle = {Advances in Neural Information Processing Systems 27},
editor = {Z. Ghahramani and M. Welling and C. Cortes and N. D. Lawrence and K. Q. Weinberger},
pages = {3104--3112},
year = {2014},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf}
}
@article{DeepSpeech2014,
title = {Deep Speech: Scaling up end-to-end speech recognition},
author = {Awni Y. Hannun and Carl Case and Jared Casper and Bryan Catanzaro and Greg Diamos and Erich Elsen and Ryan Prenger and Sanjeev Satheesh and Shubho Sengupta and Adam Coates and Andrew Y. Ng},
journal = {CoRR},
volume = {abs/1412.5567},
year = {2014},
url = {http://arxiv.org/abs/1412.5567},
archivePrefix = {arXiv},
eprint = {1412.5567},
timestamp = {Mon, 13 Aug 2018 16:48:07 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/HannunCCCDEPSSCN14},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{DeepSpeech2,
author = {Amodei, Dario and Ananthanarayanan, Sundaram and Anubhai, Rishita and Bai, Jingliang and Battenberg, Eric and Case, Carl and Casper, Jared and Catanzaro, Bryan and Cheng, Qiang and Chen, Guoliang and Chen, Jie and Chen, Jingdong and Chen, Zhijie and Chrzanowski, Mike and Coates, Adam and Diamos, Greg and Ding, Ke and Du, Niandong and Elsen, Erich and Engel, Jesse and Fang, Weiwei and Fan, Linxi and Fougner, Christopher and Gao, Liang and Gong, Caixia and Hannun, Awni and Han, Tony and Johannes, Lappi Vaino and Jiang, Bing and Ju, Cai and Jun, Billy and LeGresley, Patrick and Lin, Libby and Liu, Junjie and Liu, Yang and Li, Weigao and Li, Xiangang and Ma, Dongpeng and Narang, Sharan and Ng, Andrew and Ozair, Sherjil and Peng, Yiping and Prenger, Ryan and Qian, Sheng and Quan, Zongfeng and Raiman, Jonathan and Rao, Vinay and Satheesh, Sanjeev and Seetapun, David and Sengupta, Shubho and Srinet, Kavya and Sriram, Anuroop and Tang, Haiyuan and Tang, Liliang and Wang, Chong and Wang, Jidong and Wang, Kaifu and Wang, Yi and Wang, Zhijian and Wang, Zhiqian and Wu, Shuang and Wei, Likai and Xiao, Bo and Xie, Wen and Xie, Yan and Yogatama, Dani and Yuan, Bin and Zhan, Jun and Zhu, Zhenyao},
title = {Deep Speech 2: End-to-end Speech Recognition in English and Mandarin},
booktitle = {Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48},
series = {ICML'16},
year = {2016},
location = {New York, NY, USA},
pages = {173--182},
numpages = {10},
url = {http://dl.acm.org/citation.cfm?id=3045390.3045410},
acmid = {3045410},
publisher = {JMLR.org},
}
@inproceedings{prabhavalkar2017comparison,
title={A comparison of sequence-to-sequence models for speech recognition},
author={Prabhavalkar, Rohit and Rao, Kanishka and Sainath, Tara N and Li, Bo and Johnson, Leif and Jaitly, Navdeep},
booktitle={Proc. Interspeech},
pages={939--943},
year={2017}
}
@article{chiu2017state,
title={State-of-the-art speech recognition with sequence-to-sequence models},
author={Chiu, Chung-Cheng and Sainath, Tara N and Wu, Yonghui and Prabhavalkar, Rohit and Nguyen, Patrick and Chen, Zhifeng and Kannan, Anjuli and Weiss, Ron J and Rao, Kanishka and Gonina, Katya and others},
journal={arXiv preprint arXiv:1712.01769},
year={2017}
}
@misc{NVMixed,
title = {{NVIDA's Mixed-Precision Training - TensorFlow example}},
howpublished = {\url{https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/#example_tensorflow}},
author={NVIDIA},
note = {Accessed: 2018-10-09},
year={2018}
}
@article{gehring2017,
title={Convolutional sequence to sequence learning},
author={Gehring, Jonas and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N},
journal={arXiv preprint arXiv:1705.03122},
year={2017}
}
@article{collobert2016,
title={Wav2letter: an end-to-end convnet-based speech recognition system},
author={Collobert, Ronan and Puhrsch, Christian and Synnaeve, Gabriel},
journal={arXiv preprint arXiv:1609.03193},
year={2016}
}
@inproceedings{Zhang2016,
author={Ying Zhang and Mohammad Pezeshki and Philémon Brakel and Saizheng Zhang and César Laurent and Yoshua Bengio and Aaron Courville},
title={Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-1446},
url={http://dx.doi.org/10.21437/Interspeech.2016-1446},
pages={410--414}
}
@inproceedings{Zhang2017,
title={Very deep convolutional networks for end-to-end speech recognition},
author={Zhang, Yu, and Chan, William, and Jaitly, Navdeep},
booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on},
year={2017},
organization={IEEE}
}
@article{Wang2017,
title={Tacotron: Towards End-to-End Speech Synthesis},
author={ Wang, Yuxuan, and Skerry-Ryan, RJ, and Stanton, Daisy and Wu, Yonghui and Weiss, Ron, and Jaitly, Navdeep and Yang, Zongheng and Xiao, Ying and Chen,Zhifeng and Bengio, Samy and Le, Quoc and Agiomyrgiannakis, Yannis and Clark,Rob and Saurous, Rif A.},
journal={arXiv preprint arXiv:1703.10135},
year={2017}
}
@inproceedings{shen2018natural,
title={Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions},
author={Shen, Jonathan and Pang, Ruoming and Weiss, Ron J and Schuster, Mike and Jaitly, Navdeep and Yang, Zongheng and Chen, Zhifeng and Zhang, Yu and Wang, Yuxuan and Skerrv-Ryan, Rj and others},
booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages={4779--4783},
year={2018},
organization={IEEE}
}
@article{griffin1984signal,
title={Signal estimation from modified short-time Fourier transform},
author={Griffin, Daniel and Lim, Jae},
journal={IEEE Transactions on Acoustics, Speech, and Signal Processing},
volume={32},
number={2},
pages={236--243},
year={1984},
publisher={IEEE}
}
@misc{ito2017lj,
title={The LJ speech dataset},
author={Ito, Keith and others},
year={2017}
}
@misc{mailabs,
title = {{The M-AILABS Speech Dataset}},
howpublished = {\url{http://www.m-ailabs.bayern/en/the-mailabs-speech-dataset/}},
author={M-AILABS},
note = {Accessed: 2018-10-09},
year={2018}
}
@article{merity2016pointer,
title={Pointer sentinel mixture models},
author={Merity, Stephen and Xiong, Caiming and Bradbury, James and Socher, Richard},
journal={arXiv preprint arXiv:1609.07843},
year={2016}
}
@inproceedings{socher2013recursive,
title={Recursive deep models for semantic compositionality over a sentiment treebank},
author={Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D and Ng, Andrew and Potts, Christopher},
booktitle={Proceedings of the 2013 conference on empirical methods in natural language processing},
pages={1631--1642},
year={2013}
}
@InProceedings{maas-EtAl:2011:ACL-HLT2011,
author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher},
title = {Learning Word Vectors for Sentiment Analysis},
booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
month = {June},
year = {2011},
address = {Portland, Oregon, USA},
publisher = {Association for Computational Linguistics},
pages = {142--150},
url = {http://www.aclweb.org/anthology/P11-1015}
}
@inproceedings{Povey2018SemiOrthogonalLM,
title={Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks},
author={Daniel Povey and Gaofeng Cheng and Yiming Wang and Ke Li and Hainan Xu and Mahsa Yarmohammadi and Sanjeev Khudanpur},
booktitle={Interspeech},
year={2018}
}
@article{CAPIO2017,
author = {Kyu J. Han and Akshay Chandrashekaran and Jungsuk Kim and Ian R. Lane},
title = {The {CAPIO} 2017 Conversational Speech Recognition System},
journal = {CoRR},
volume = {abs/1801.00059},
year = {2018},
url = {http://arxiv.org/abs/1801.00059},
archivePrefix = {arXiv},
eprint = {1801.00059},
timestamp = {Mon, 13 Aug 2018 16:49:10 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1801-00059},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{WaveNet,
author = {A{\"{a}}ron van den Oord and Sander Dieleman and Heiga Zen and Karen Simonyan and Oriol Vinyals and Alex Graves and Nal Kalchbrenner and Andrew W. Senior and Koray Kavukcuoglu},
title = {WaveNet: {A} Generative Model for Raw Audio},
journal = {CoRR},
volume = {abs/1609.03499},
year = {2016},
url = {http://arxiv.org/abs/1609.03499},
archivePrefix = {arXiv},
eprint = {1609.03499},
timestamp = {Mon, 13 Aug 2018 16:49:15 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/OordDZSVGKSK16},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{FacebookGERENGBackTranslation,
author = {Rico Sennrich and Barry Haddow and Alexandra Birch},
title = {Improving Neural Machine Translation Models with Monolingual Data},
journal = {CoRR},
volume = {abs/1511.06709},
year = {2015},
url = {http://arxiv.org/abs/1511.06709},
archivePrefix = {arXiv},
eprint = {1511.06709},
timestamp = {Mon, 13 Aug 2018 16:47:05 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/SennrichHB15a},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{GlobalStyleTokens,
author = {Yuxuan Wang and Daisy Stanton and Yu Zhang and R. J. Skerry{-}Ryan and Eric Battenberg and Joel Shor and Ying Xiao and Fei Ren and Ye Jia and Rif A. Saurous},
title = {Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis},
journal = {CoRR},
volume = {abs/1803.09017},
year = {2018},
url = {http://arxiv.org/abs/1803.09017},
archivePrefix = {arXiv},
eprint = {1803.09017},
timestamp = {Mon, 13 Aug 2018 16:46:53 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1803-09017},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{IoffeS15BatchNorm,
author = {Sergey Ioffe and Christian Szegedy},
title = {Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift},
journal = {CoRR},
volume = {abs/1502.03167},
year = {2015},
url = {http://arxiv.org/abs/1502.03167},
archivePrefix = {arXiv},
eprint = {1502.03167},
timestamp = {Mon, 13 Aug 2018 16:47:06 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/IoffeS15},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{kingma,
author = {Diederik P. Kingma and
Jimmy Ba},
title = {Adam: {A} Method for Stochastic Optimization},
journal = {CoRR},
volume = {abs/1412.6980},
year = {2014},
url = {http://arxiv.org/abs/1412.6980},
archivePrefix = {arXiv},
eprint = {1412.6980},
timestamp = {Mon, 13 Aug 2018 01:00:00 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/KingmaB14},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@incollection{Salimans2016WeightNorm,
title = {Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks},
author = {Salimans, Tim and Kingma, Durk P},
booktitle = {Advances in Neural Information Processing Systems 29},
editor = {D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett},
pages = {901--909},
year = {2016},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf}
}
@article{wu2016google,
title={Google's neural machine translation system: Bridging the gap between human and machine translation},
author={Wu, Yonghui and Schuster, Mike and Chen, Zhifeng and Le, Quoc V and Norouzi, Mohammad and Macherey, Zolfgang and Krikun, Maxim and Cao, Yuan and Gao, Qin and Macherey, Klaus and others},
journal={arXiv preprint arXiv:1609.08144},
year={2016}
}
@inproceedings{opennmt,
author = {Guillaume Klein and Yoon Kim and Yuntian Deng and Jean Senellart and Alexander M. Rush},
title = {OpenNMT: Open-Source Toolkit for Neural Machine Translation},
booktitle = {Proc. ACL},
year = {2017},
url = {https://doi.org/10.18653/v1/P17-4012},
doi = {10.18653/v1/P17-4012}
}
@article{paszke2017automatic,
title={Automatic differentiation in PyTorch},
author={Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory and Yang, Edward and DeVito, Zachary and Lin, Zeming and Desmaison, Alban and Antiga, Luca and Lerer, Adam},
year={2017}
}
@article{yu2014introduction,
title={An introduction to computational networks and the computational network toolkit},
author={Yu, Dong and Eversole, Adam and Seltzer, Mike and Yao, Kaisheng and Huang, Zhiheng and Guenter, Brian and Kuchaiev, Oleksii and Zhang, Yu and Seide, Frank and Wang, Huaming and others},
journal={Microsoft Technical Report MSR-TR-2014--112},
year={2014}
}
@article{nvidia2017v100,
title={V100 GPU architecture. The world’s most advanced data center GPU. Version WP-08608-001\_v1. 1},
author={NVIDIA, Tesla},
journal={NVIDIA. Aug},
pages={108},
year={2017}
}
@article{post2018call,
title={A call for clarity in reporting bleu scores},
author={Post, Matt},
journal={arXiv preprint arXiv:1804.08771},
year={2018}
}
@article{Ba2016LayerNorm,
author = {Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E Hinton},
title = {Layer normalization},
journal = {CoRR},
volume = {abs/1607.06450},
year = {2016},
url = {http://arxiv.org/abs/1607.06450},
archivePrefix = {arXiv},
}
@inproceedings{Dauphin2017GLU,
author = {Dauphin, Yann N. and Fan, Angela and Auli, Michael and Grangier, David},
title = {Language Modeling with Gated Convolutional Networks},
booktitle = {Proceedings of the 34th International Conference on Machine Learning - Volume 70},
series = {ICML'17},
year = {2017},
location = {Sydney, NSW, Australia},
pages = {933--941},
numpages = {9},
url = {http://dl.acm.org/citation.cfm?id=3305381.3305478},
acmid = {3305478},
publisher = {JMLR.org},
}
@incollection{Oord2016PixelCNN,
title = {Conditional Image Generation with PixelCNN Decoders},
author = {van den Oord, Aaron and Kalchbrenner, Nal and Espeholt, Lasse and kavukcuoglu, koray and Vinyals, Oriol and Graves, Alex},
booktitle = {Advances in Neural Information Processing Systems 29},
editor = {D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett},
pages = {4790--4798},
year = {2016},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/6527-conditional-image-generation-with-pixelcnn-decoders.pdf}
}
@article{he2015,
title={Deep residual learning for image recognition},
author={K. He, and X. Zhang, and S. Ren, and J. Sun},
journal={arXiv preprint arXiv:1512.03385},
year={2015}
}
@article{huang2016,
title={Densely Connected Convolutional Networks},
author={Gao Huang, and Zhuang Liu, and Laurens van der Maaten, and Kilian Q. Weinberger},
journal={arXiv preprint arXiv:1608.06993},
year={2016}
}
@inproceedings{heafield2011kenlm,
title={KenLM: Faster and smaller language model queries},
author={Heafield, Kenneth},
booktitle={Proceedings of the sixth workshop on statistical machine translation},
pages={187--197},
year={2011},
organization={Association for Computational Linguistics}
}
@article{dai2018transformer,
title={Transformer-XL: Language Modeling with Longer-Term Dependency},
author={Dai, Zihang and Yang, Zhilin and Yang, Yiming and Cohen, William W and Carbonell, Jaime and Le, Quoc V and Salakhutdinov, Ruslan},
year={2018},
journal = {CoRR},
volume = {abs/1901.02860},
url = {http://arxiv.org/abs/1901.02860},
archivePrefix = {arXiv},
eprint = {1901.02860},
timestamp = {Fri, 01 Feb 2019 13:39:59 +0100},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1901-02860},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{Saon+2016,
author={George Saon and Tom Sercu and Steven Rennie and Hong-Kwang J. Kuo},
title={The IBM 2016 English Conversational Telephone Speech Recognition System},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-1460},
url={http://dx.doi.org/10.21437/Interspeech.2016-1460},
pages={7--11}
}
@INPROCEEDINGS{Sercu-2016,
author={T. {Sercu} and C. {Puhrsch} and B. {Kingsbury} and Y. {LeCun}},
booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={Very deep multilingual convolutional neural networks for LVCSR},
year={2016},
volume={},
number={},
pages={4955-4959},
keywords={natural language processing;neural nets;speech recognition;very deep multilingual convolutional neural networks;LVCSR;CNN;large vocabulary continuous speech recognition systems;word error rate;Training;Context;Hidden Markov models;Neural networks;Computer architecture;Kernel;Training data;Convolutional Networks;Multilingual;Acoustic Modeling;Speech Recognition;Neural Networks},
doi={10.1109/ICASSP.2016.7472620},
ISSN={2379-190X},
month={March},}
@inproceedings{Sercu+2016,
author={Tom Sercu and Vaibhava Goel},
title={Advances in Very Deep Convolutional Neural Networks for LVCSR},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-1033},
url={http://dx.doi.org/10.21437/Interspeech.2016-1033},
pages={3429--3433}
}
@INPROCEEDINGS{Xiong-2018,
author={W. {Xiong} and L. {Wu} and F. {Alleva} and J. {Droppo} and X. {Huang} and A. {Stolcke}},
booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={The Microsoft 2017 Conversational Speech Recognition System},
year={2018},
volume={},
number={},
pages={5934-5938},
keywords={convolution;feedforward neural nets;natural language processing;speaker recognition;speech processing;language model rescoring step;senone level;switchboard domains;character-based LSTM language models;NIST 2000 switchboard test set;frame level;word-level voting;acoustic model posteriors;dialog session aware LSTM language models;CNN-BLSTM acoustic model;Microsoft 2017 conversational speech recognition system;Acoustics;Error analysis;Training;Speech recognition;Switches;Computational modeling;Context modeling;Conversational speech recognition;CNN;LACE;BLSTM;LSTM-LM;system combination;human parity},
doi={10.1109/ICASSP.2018.8461870},
ISSN={2379-190X},
month={April},}
@inproceedings{zeyer2018improved,
author={Albert Zeyer and Kazuki Irie and Ralf Schlüter and Hermann Ney},
title={Improved Training of End-to-end Attention Models for Speech Recognition},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={7--11},
doi={10.21437/Interspeech.2018-1616},
url={http://dx.doi.org/10.21437/Interspeech.2018-1616}
}
@article{Wav2LetterV2,
author = {Vitaliy Liptchinsky and
Gabriel Synnaeve and
Ronan Collobert},
title = {Letter-Based Speech Recognition with Gated ConvNets},
journal = {CoRR},
volume = {abs/1712.09444},
year = {2017},
url = {http://arxiv.org/abs/1712.09444},
archivePrefix = {arXiv},
eprint = {1712.09444},
timestamp = {Mon, 13 Aug 2018 16:46:33 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-09444},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{zeghidour2018,
author = {Neil Zeghidour and
Qiantong Xu and
Vitaliy Liptchinsky and
Nicolas Usunier and
Gabriel Synnaeve and
Ronan Collobert},
title = {Fully Convolutional Speech Recognition},
journal = {CoRR},
volume = {abs/1812.06864},
year = {2018},
url = {http://arxiv.org/abs/1812.06864},
archivePrefix = {arXiv},
eprint = {1812.06864},
timestamp = {Tue, 01 Jan 2019 15:01:25 +0100},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1812-06864},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{Hadian2018,
author={Hossein Hadian and Hossein Sameti and Daniel Povey and Sanjeev Khudanpur},
title={End-to-end Speech Recognition Using Lattice-free MMI},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={12--16},
doi={10.21437/Interspeech.2018-1423},
url={http://dx.doi.org/10.21437/Interspeech.2018-1423}
}
@inproceedings{Tang2018,
author={Jian Tang and Yan Song and Lirong Dai and Ian McLoughlin},
title={Acoustic Modeling with Densely Connected Residual Network for Multichannel Speech Recognition},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={1783--1787},
doi={10.21437/Interspeech.2018-1089},
url={http://dx.doi.org/10.21437/Interspeech.2018-1089}
}
@article{Kurata2017LanguageMW,
title={Language modeling with highway LSTM},
author={Gakuto Kurata and Bhuvana Ramabhadran and George Saon and Abhinav Sethy},
journal={2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
year={2017},
pages={244-251}
}
@inproceedings{Saon2017,
author={George Saon and Gakuto Kurata and Tom Sercu and Kartik Audhkhasi and Samuel Thomas and Dimitrios Dimitriadis and Xiaodong Cui and Bhuvana Ramabhadran and Michael Picheny and Lynn-Li Lim and Bergul Roomi and Phil Hall},
title={English Conversational Telephone Speech Recognition by Humans and Machines},
year=2017,
booktitle={Proc. Interspeech 2017},
pages={132--136},
doi={10.21437/Interspeech.2017-405},
url={http://dx.doi.org/10.21437/Interspeech.2017-405}
}
@inproceedings{Povey+2016,
author={Daniel Povey and Vijayaditya Peddinti and Daniel Galvez and Pegah Ghahremani and Vimal Manohar and Xingyu Na and Yiming Wang and Sanjeev Khudanpur},
title={Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-595},
url={http://dx.doi.org/10.21437/Interspeech.2016-595},
pages={2751--2755}
}
@article{Yang2018,
author = {Xuerui Yang and
Jiwei Li and
Xi Zhou},
title = {A novel pyramidal-FSMN architecture with lattice-free {MMI} for speech
recognition},
journal = {CoRR},
volume = {abs/1810.11352},
year = {2018},
url = {http://arxiv.org/abs/1810.11352},
archivePrefix = {arXiv},
eprint = {1810.11352},
timestamp = {Wed, 31 Oct 2018 14:24:29 +0100},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1810-11352},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{liptchinsky2017based,
title={Letter-Based Speech Recognition with Gated ConvNets},
author={Liptchinsky, Vitaliy and Synnaeve, Gabriel and Collobert, Ronan},
journal={arXiv preprint arXiv:1712.09444},
year={2017}
}
@inproceedings{Weng2018,
author={Chao Weng and Jia Cui and Guangsen Wang and Jun Wang and Chengzhu Yu and Dan Su and Dong Yu},
title={Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={761--765},
doi={10.21437/Interspeech.2018-1030},
url={http://dx.doi.org/10.21437/Interspeech.2018-1030}
}
@INPROCEEDINGS{Battenberg2017,
author={E. {Battenberg} and J. {Chen} and R. {Child} and A. {Coates} and Y. G. Y. {Li} and H. {Liu} and S. {Satheesh} and A. {Sriram} and Z. {Zhu}},
booktitle={2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
title={Exploring neural transducers for end-to-end speech recognition},
year={2017},
volume={},
number={},
pages={206-213},
keywords={recurrent neural nets;speech recognition;Hub500 benchmark;CTC models;speech recognition pipeline;RNN-Transducer models;language model;Seq2Seq models;end-to-end speech recognition;neural transducers;Decoding;Hidden Markov models;Transducers;Task analysis;Speech;Mathematical model;Neural networks},
doi={10.1109/ASRU.2017.8268937},
ISSN={},
month={Dec},
}
@inproceedings{
loshchilov2018,
title={Decoupled Weight Decay Regularization},
author={Ilya Loshchilov and Frank Hutter},
booktitle={International Conference on Learning Representations},
year={2019},
url={https://openreview.net/forum?id=Bkg6RiCqY7},
}
@article{zhang2017ndadam,
author = {Zijun Zhang and Lin Ma and Zongpeng Li and Chuan Wu},
title = {Normalized Direction-preserving Adam},
journal = {arXiv e-prints arXiv:1709.04546},
year = {2017},
}
@article{park2019,
author = {{Park}, Daniel S. and {Chan}, William and {Zhang}, Yu and
{Chiu}, Chung-Cheng and {Zoph}, Barret and {Cubuk}, Ekin D. and
{Le}, Quoc V.},
title = "{SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition}",
journal = {arXiv e-prints},
year = "2019",
eid = {arXiv:1904.08779},
eprint = {1904.08779},
}
@article{novograd2019,
author = {{Ginsburg}, Boris and {Castonguay}, Patrice and {Hrinchuk}, Oleksii and
{Kuchaiev}, Oleksii and {Lavrukhin}, Vitaly and {Leary}, Ryan and
{Li}, Jason and {Nguyen}, Huyen and {Cohen}, Jonathan M.},
title = "{Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks}",
journal = {arXiv e-prints},
year = "2019",
eid = {arXiv:1905.11286},
eprint = {1905.11286},
}
@article{kriman2019quartznet,
title={Quartznet: {Deep} automatic speech recognition with 1d time-channel separable convolutions},
author={Kriman, Samuel and Beliaev, Stanislav and Ginsburg, Boris and Huang, Jocelyn and Kuchaiev, Oleksii and Lavrukhin, Vitaly and Leary, Ryan and Li, Jason and Zhang, Yang},
journal={arXiv preprint arXiv:1910.10261},
year={2019}
}
@misc{itu1988g711,
title={{ITU-T} {G.711} - {Pulse} code modulation ({PCM}) of voice frequencies},
author={ITU-T Geneva Switzerland},
year={1988},
}
@article{han2020contextnet,
title={ContextNet: Improving convolutional neural networks for automatic speech recognition with global context},
author={Han, Wei and Zhang, Zhengdong and Zhang, Yu and Yu, Jiahui and Chiu, Chung-Cheng and Qin, James and Gulati, Anmol and Pang, Ruoming and Wu, Yonghui},
journal={arXiv:2005.03191},
year={2020}
}
@inproceedings{hu2018squeeze,
title={Squeeze-and-excitation networks},
author={Hu, Jie and Shen, Li and Sun, Gang},
booktitle={ICVPR},
year={2018}
}
@article{koluguri2020speakernet,
title={SpeakerNet: 1D Depth-wise Separable Convolutional Network for Text-Independent Speaker Recognition and Verification},
author={Koluguri, Nithin Rao and Li, Jason and Lavrukhin, Vitaly and Ginsburg, Boris},
journal={arXiv preprint arXiv:2010.12653},
year={2020}
}
@article{gulati2020conformer,
title={Conformer: Convolution-augmented transformer for speech recognition},
author={Gulati, Anmol and Qin, James and Chiu, Chung-Cheng and Parmar, Niki and Zhang, Yu and Yu, Jiahui and Han, Wei and Wang, Shibo and Zhang, Zhengdong and Wu, Yonghui and others},
journal={arXiv preprint arXiv:2005.08100},
year={2020}
}
tts/NeMo/docs/source/asr/asr_language_modeling.rst
+292
View File
@@ -0,0 +1,292 @@
#####################
ASR Language Modeling
#####################
Language models have shown to help the accuracy of ASR models. NeMo supports the following two approaches to incorporate language models into the ASR models:
* :ref:`ngram_modeling`
* :ref:`neural_rescoring`
It is possible to use both approaches on the same ASR model.
.. _ngram_modeling:
************************
N-gram Language Modeling
************************
In this approach, an N-gram LM is trained on text data, then it is used in fusion with beam search decoding to find the
best candidates. The beam search decoders in NeMo support language models trained with KenLM library (
`https://github.com/kpu/kenlm <https://github.com/kpu/kenlm>`__).
The beam search decoders and KenLM library are not installed by default in NeMo, and you need to install them to be
able to use beam search decoding and N-gram LM.
Please refer to `scripts/ngram_lm/install_beamsearch_decoders.sh` on how to install them.
NeMo supports both character-based and BPE-based models for N-gram LMs. An N-gram LM can be used with beam search
decoders on top of the ASR models to produce more accurate candidates. The beam search decoder would incorporate
the scores produced by the N-gram LM into its score calculations as the following:
.. code::
final_score = acoustic_score + beam_alpha*lm_score + beam_beta*seq_length
where acoustic_score is the score predicted by the acoustic encoder and lm_score is the one estimated by the LM.
Parameter 'beam_alpha' specifies amount of importance to place on the N-gram language model, and 'beam_beta' is a
penalty term to consider the sequence length in the scores. Larger alpha means more importance on the LM and less
importance on the acoustic model. Negative values for beta will give penalty to longer sequences and make the decoder
to prefer shorter predictions, while positive values would result in longer candidates.
Train N-gram LM
===============
The script to train an N-gram language model with KenLM can be found at
`scripts/asr_language_modeling/ngram_lm/train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
This script would train an N-gram language model with KenLM library which can be used with the beam search decoders
on top of the ASR models. This script supports both character level and BPE level encodings and models which is
detected automatically from the type of the model.
You may train the N-gram model as the following:
.. code::
python train_kenlm.py --nemo_model_file <path to the .nemo file of the model> \
--train_file <path to the training text or JSON manifest file \
--kenlm_bin_path <path to the bin folder of KenLM library> \
--kenlm_model_file <path to store the binary KenLM model> \
--ngram_length <order of N-gram model>
The train file specified by `--train_file` can be a text file or JSON manifest. If the file's extension is anything
other than `.json`, it assumes that data format is plain text. For plain text format, each line should contain one
sample. For JSON manifest file, the file need to contain json formatted samples per each line like this:
.. code::
{"audio_filepath": "/data_path/file1.wav", "text": "The transcript of the audio file."}
It just extracts the `text` field from each line to create the training text file. After the N-gram model is trained,
it is stored at the path specified by `--kenlm_model_file`.
The following is the list of the arguments for the training script:
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| **Argument** | **Type** | **Default** | **Description** |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| nemo_model_file | str | Required | The path of the `.nemo` file of the ASR model. It is needed to extract the tokenizer. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| train_file | str | Required | Path to the training file, it can be a text file or JSON manifest. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| kenlm_model_file | str | Required | The path to store the KenLM binary model file. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| kenlm_bin_path | str | Required | The path to the bin folder of KenLM. It is a folder named `bin` under where KenLM is installed.|
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| ngram_length** | int | Required | Specifies order of N-gram LM. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| do_lower_case | bool | ``False`` | Whether to make the training text all lower case. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
** Note: Recommend to use 6 as the order of the N-gram model for BPE-based models. Higher orders may need the re-compilation of KenLM to support it.
Evaluate by Beam Search Decoding and N-gram LM
==============================================
NeMo's beam search decoders are capable of using the KenLM's N-gram models to find the best candidates.
The script to evaluate an ASR model with beam search decoding and N-gram models can be found at
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py>`__.
You may evaluate an ASR model as the following:
.. code::
python eval_beamsearch_ngram.py --nemo_model_file <path to the .nemo file of the model> \
--input_manifest <path to the evaluation JSON manifest file \
--kenlm_model_file <path to the binary KenLM model> \
--acoustic_batch_size <batch size for calculating log probabilities> \
--beam_width <list of the beam widths> \
--beam_alpha <list of the beam alphas> \
--beam_width <list of the beam betas> \
--preds_output_folder <optional folder to store the predictions> \
--decoding_mode beam_search_ngram
It can evaluate a model in the three following modes by setting the argument `--decoding_mode`:
* greedy: Just greedy decoding is done, and no beam search decoding is performed.
* beamsearch: The beam search decoding is done but without using the N-gram language model, final results would be equivalent to setting the weight of LM (beam_beta) to zero.
* beamsearch_ngram: The beam search decoding is done with N-gram LM.
The `beamsearch` mode would evaluate by beam search decoding without any language model.
It would report the performances in terms of Word Error Rate (WER) and Character Error Rate (CER). Moreover,
the WER/CER of the model when the best candidate is selected among the candidates is also reported as the best WER/CER.
It can be an indicator of how good the predicted candidates are.
The script would initially load the ASR model and predict the outputs of the model's encoder as log probabilities.
This part would be computed in batches on a device selected by `--device`, which can be CPU (`--device=cpu`) or a
single GPU (`--device=cuda:0`). The batch size of this part can get specified by `--acoustic_batch_size`. You may use
the largest batch size feasible to speed up the step of calculating the log probabilities. You may also use `--use_amp`
to speed up the calculation of log probabilities and make it possible to use larger sizes for `--acoustic_batch_size`.
Currently multi-GPU is not supported for calculating the log probabilities, but using `--probs_cache_file` can help.
It stores the log probabilities produced from the model's encoder into a pickle file so that next time the first step
can get skipped.
The following is the list of the arguments for the evaluation script:
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| **Argument** |**Type**| **Default** | **Description** |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| nemo_model_file | str | Required | The path of the `.nemo` file of the ASR model to extract the tokenizer. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| input_manifest | str | Required | Path to the training file, it can be a text file or JSON manifest. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| kenlm_model_file | str | Required | The path to store the KenLM binary model file. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| preds_output_folder | str | None | The path to an optional folder to store the predictions. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| probs_cache_file | str | None | The cache file for storing the outputs of the model. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| acoustic_batch_size | int | 16 | The batch size to calculate log probabilities. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| use_amp | bool | ``False`` | Whether to use AMP if available to calculate log probabilities. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| device | str | cuda | The device to load the model onto to calculate log probabilities. |
| | | | It can `cpu`, `cuda`, `cuda:0`, `cuda:1`, ... |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| decoding_mode | str | beamsearch_ngram | The decoding scheme to be used for evaluation. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_width | float | Required | The width or list of the widths of the beam search decoding. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_alpha | float | Required | The alpha parameter or list of the alphas for the beam search decoding. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_beta | float | Required | The beta parameter or list of the betas for the beam search decoding. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_batch_size | int | 128 | The batch size to be used for beam search decoding. |
| | | | Larger batch size can be a little faster, but uses larger memory. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
Width of the beam search (`--beam_width`) specifies the number of top candidates/predictions the beam search decoder
would search for. Larger beams result in more accurate but slower predictions.
There is also a tutorial to learn more about evaluating the ASR models with N-gram LM here:
`Offline ASR Inference with Beam Search and External Language Model Rescoring <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/Offline_ASR.ipynb>`_
Hyperparameter Grid Search
--------------------------
Beam search decoding with N-gram LM has three main hyperparameters: `beam_width`, `beam_alpha`, and `beam_beta`.
The accuracy of the model is dependent to the values of these parameters, specially beam_alpha and beam_beta.
You may specify a single or list of values for each of these parameters to perform grid search. It would perform the
beam search decoding on all the combinations of the these three hyperparameters.
For instance, the following set of parameters would results in 2*1*2=4 beam search decodings:
.. code::
python eval_beamsearch_ngram.py ... \
--beam_width 64 128 \
--beam_alpha 1.0 \
--beam_beta 1.0 0.5
.. _neural_rescoring:
****************
Neural Rescoring
****************
In this approach a neural network is used which can gives scores to a candidate. A candidate is the text transcript predicted by the decoder of the ASR model.
The top K candidates produced by the beam search decoding (beam width of K) are given to a neural language model to rank them.
Ranking can be done by a language model which gives a score to each candidate.
This score is usually combined with the scores from the beam search decoding to produce the final scores and rankings.
Train Neural Rescorer
=====================
An example script to train such a language model with Transformer can be found at `examples/nlp/language_modeling/transformer_lm.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/transformer_lm.py>`__.
It trains a TransformerLMModel which can be used as a neural rescorer for an ASR system.
Evaluation
==========
Given a trained TransformerLMModel `.nemo` file, the script available at
`scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__
can be used to re-score beams obtained with ASR model. You need the `.tsv` file containing the candidates produced
by the acoustic model and the beam search decoding to use this script. The candidates can be the result of just the beam
search decoding or the result of fusion with an N-gram LM. You may generate this file by specifying `--preds_output_folder' for
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py>`__.
The neural rescorer would rescore the beams/candidates by using two parameters of `rescorer_alpha` and `rescorer_beta` as the following:
.. code::
final_score = beam_search_score + rescorer_alpha*neural_rescorer_score + rescorer_beta*seq_length
Parameter `rescorer_alpha` specifies amount of importance to place on the neural rescorer model, and `rescorer_beta` is
a penalty term to consider the sequence length in the scores. They have similar effects like the parameters
`beam_alpha` and `beam_beta` of beam search decoder and N-gram LM.
You may follow the following steps to evaluate a neural LM:
#. Obtain `.tsv` file with beams and their corresponding scores. Scores can be from a regular beam search decoder or
in fusion with an N-gram LM scores. For a given beam size `beam_size` and a number of examples
for evaluation `num_eval_examples`, it should contain (`num_eval_examples` x `beam_size`) lines of
form `beam_candidate_text \t score`. This file can be generated by `scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py>`__
#. Rescore the candidates by `scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__.
.. code::
python eval_neural_rescorer.py
--lm_model=[path to .nemo file of the LM]
--beams_file=[path to beams .tsv file]
--beam_size=[size of the beams]
--eval_manifest=[path to eval manifest .json file]
--batch_size=[batch size used for inference on the LM model]
--alpha=[the value for the parameter rescorer_alpha]
--beta=[the value for the parameter rescorer_beta]
--scores_output_file=[the optional path to store the rescored candidates]
The candidates along with their new scores would be stored at the file specified by `--scores_output_file`.
The following is the list of the arguments for the evaluation script:
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| **Argument** |**Type**| **Default** | **Description** |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| lm_model | str | Required | The path of the '.nemo' file of the ASR model to extract the tokenizer. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| eval_manifest | str | Required | Path to the evaluation manifest file (.json manifest file) |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beams_file | str | Required | path to beams file (.tsv) containing the candidates and their scores |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_size | int | Required | The width of the beams (number of candidates) generated by the decoder |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| alpha | float | None | The value for parameter rescorer_alpha |
| | | | Not passing value would enable linear search for rescorer_alpha |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beta | float | None | The value for parameter rescorer_beta |
| | | | Not passing value would enable linear search for rescorer_beta |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| batch_size | int | 16 | The batch size used to calculate the scores |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| scores_output_file | str | None | The optional file to store the rescored beams |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| use_amp | bool | ``False`` | Whether to use AMP if available calculate the scores |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| device | str | cuda | The device to load LM model onto to calculate the scores |
| | | | It can be 'cpu', 'cuda', 'cuda:0', 'cuda:1', ... |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
Hyperparameter Linear Search
----------------------------
This script also supports linear search for parameters `rescorer_alpha` and `rescorer_beta`. If any of the two is not
provided, a linear search is performed to find the best value for that parameter. When linear search is used, initially
`rescorer_beta` is set to zero and the best value for `rescorer_alpha` is found, then `rescorer_alpha` is fixed with
that value and another linear search is done to find the best value for `rescorer_beta`.
If any of the of these two parameters is already specified, then search for that one is skipped. After each search for a
parameter, the plot of WER% for different values of the parameter is also shown.
It is recommended to first use the linear search for both parameters on a validation set by not providing any values for `--alpha` and `--beta`.
Then check the WER curves and decide on the best values for each parameter. Finally, evaluate the best values on the test set.
tts/NeMo/docs/source/asr/configs.rst
+528
View File
@@ -0,0 +1,528 @@
NeMo ASR Configuration Files
============================
This section describes the NeMo configuration file setup that is specific to models in the ASR collection. For general information
about how to set up and run experiments that is common to all NeMo models (e.g. Experiment Manager and PyTorch Lightning trainer
parameters), see the :doc:`../core/core` section.
The model section of the NeMo ASR configuration files generally requires information about the dataset(s) being used, the preprocessor
for audio files, parameters for any augmentation being performed, as well as the model architecture specification. The sections on
this page cover each of these in more detail.
Example configuration files for all of the NeMo ASR scripts can be found in the
`config directory of the examples <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples/asr/conf>`_.
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections in the configuration file, respectively. Depending on the task, there may be arguments specifying the sample rate
of the audio files, the vocabulary of the dataset (for character prediction), whether or not to shuffle the dataset, and so on. You may
also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command-line at runtime.
Any initialization parameter that is accepted for the Dataset class used in the experiment can be set in the config file.
Refer to the `Datasets <./api.html#Datasets>`__ section of the API for a list of Datasets and their respective parameters.
An example ASR train and validation configuration should look similar to the following:
.. code-block:: yaml
# Specified at the beginning of the config file
labels: &labels [" ", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m",
"n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "'"]
model:
train_ds:
manifest_filepath: ???
sample_rate: 16000
labels: *labels # Uses the labels above
batch_size: 32
trim_silence: True
max_duration: 16.7
shuffle: True
is_tarred: False # If set to true, uses the tarred version of the Dataset
tarred_audio_filepaths: null # Not used if is_tarred is false
tarred_shard_strategy: "scatter" # Not used if is_tarred is false
num_workers: 8
pin_memory: true
validation_ds:
manifest_filepath: ???
sample_rate: 16000
labels: *labels # Uses the labels above
batch_size: 32
shuffle: False # No need to shuffle the validation data
num_workers: 8
pin_memory: true
Preprocessor Configuration
--------------------------
If you are loading audio files for your experiment, you will likely want to use a preprocessor to convert from the
raw audio signal to features (e.g. mel-spectrogram or MFCC). The ``preprocessor`` section of the config specifies the audio
preprocessor to be used via the ``_target_`` field, as well as any initialization parameters for that preprocessor.
An example of specifying a preprocessor is as follows:
.. code-block:: yaml
model:
...
preprocessor:
# _target_ is the audio preprocessor module you want to use
_target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
normalize: "per_feature"
window_size: 0.02
...
# Other parameters for the preprocessor
Refer to the `Audio Preprocessors <./api.html#Audio Preprocessors>`__ API section for the preprocessor options, expected arguments,
and defaults.
Augmentation Configurations
---------------------------
There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
configuration file using a ``spec_augment`` section.
For example, there are options for `Cutout <https://arxiv.org/abs/1708.04552>`_ and
`SpecAugment <https://arxiv.org/abs/1904.08779>`_ available via the ``SpectrogramAugmentation`` module.
The following example sets up both ``Cutout`` (via the ``rect_*`` parameters) and ``SpecAugment`` (via the ``freq_*``
and ``time_*`` parameters).
.. code-block:: yaml
model:
...
spec_augment:
_target_: nemo.collections.asr.modules.SpectrogramAugmentation
# Cutout parameters
rect_masks: 5 # Number of rectangles to cut from any given spectrogram
rect_freq: 50 # Max cut of size 50 along the frequency dimension
rect_time: 120 # Max cut of size 120 along the time dimension
# SpecAugment parameters
freq_masks: 2 # Cut two frequency bands
freq_width: 15 # ... of width 15 at maximum
time_masks: 5 # Cut out 10 time bands
time_width: 25 # ... of width 25 at maximum
You can use any combination of ``Cutout``, frequency/time ``SpecAugment``, or neither of them.
With NeMo ASR, you can also add augmentation pipelines that can be used to simulate various kinds of noise
added to audio in the channel. Augmentors in a pipeline are applied on the audio data read in the data layer. Online
augmentors can be specified in the config file using an ``augmentor`` section in ``train_ds``. The following example
adds an augmentation pipeline that first adds white noise to an audio sample with a probability of 0.5 and at a level
randomly picked between -50 dB and -10 dB and then passes the resultant samples through a room impulse response randomly
picked from the manifest file provided for ``impulse`` augmentation in the config file.
.. code-block:: yaml
model:
...
train_ds:
...
augmentor:
white_noise:
prob: 0.5
min_level: -50
max_level: -10
impulse:
prob: 0.3
manifest_path: /path/to/impulse_manifest.json
Refer to the `Audio Augmentors <./api.html#Audio Augmentors>`__ API section for more details.
Tokenizer Configurations
------------------------
Some models utilize sub-word encoding via an external tokenizer instead of explicitly defining their vocabulary.
For such models, a ``tokenizer`` section is added to the model config. ASR models currently support two types of
custom tokenizers:
- Google Sentencepiece tokenizers (tokenizer type of ``bpe`` in the config)
- HuggingFace WordPiece tokenizers (tokenizer type of ``wpe`` in the config)
In order to build custom tokenizers, refer to the ``ASR_with_Subword_Tokenization`` notebook available in the
ASR tutorials directory.
The following example sets up a ``SentencePiece Tokenizer`` at a path specified by the user:
.. code-block:: yaml
model:
...
tokenizer:
dir: "<path to the directory that contains the custom tokenizer files>"
type: "bpe" # can be "bpe" or "wpe"
For models which utilize sub-word tokenization, we share the decoder module (``ConvASRDecoder``) with character tokenization models.
All parameters are shared, but for models which utilize sub-word encoding, there are minor differences when setting up the config. For
such models, the tokenizer is utilized to fill in the missing information when the model is constructed automatically.
For example, a decoder config corresponding to a sub-word tokenization model should look similar to the following:
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoder
feat_in: *enc_final
num_classes: -1 # filled with vocabulary size from tokenizer at runtime
vocabulary: [] # filled with vocabulary from tokenizer at runtime
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment. Models in the NeMo ASR collection need
an ``encoder`` section and a ``decoder`` section, with the ``_target_`` field specifying the module to use for each.
Here is the list of the parameters in the model section which are shared among most of the ASR models:
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=================================+
| :code:`log_prediction` | bool | Whether a random sample should be printed in the output at each step, along with its predicted transcript. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`ctc_reduction` | string | Specifies the reduction type of CTC loss. Defaults to ``mean_batch`` which would take the average over the | :code:`none`, |
| | | batch after taking the average over the length of each sample. | :code:`mean_batch` |
| | | | :code:`mean`, :code:`sum` |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
The following sections go into more detail about the specific configurations of each model architecture.
For more information about the ASR models, refer to the :doc:`Models <./models>` section.
Jasper and QuartzNet
~~~~~~~~~~~~~~~~~~~~
The `Jasper <./models.html#Jasper>`__ and `QuartzNet <./models.html#QuartzNet>`__ models are very similar, and as such the components in their
configs are very similar as well.
Both architectures use the ``ConvASREncoder`` for the ``encoder``, with parameters detailed in the table below. The encoder parameters
include details about the Jasper/QuartzNet ``[BxR]`` encoder architecture, including how many blocks to use (``B``), how many times
to repeat each sub-block (``R``), and the convolution parameters for each block.
The number of blocks ``B`` is determined by the number of list elements under ``jasper`` minus the one prologue and two epilogue blocks.
The number of sub-blocks ``R`` is determined by setting the ``repeat`` parameter.
To use QuartzNet (which uses more compact time-channel separable convolutions) instead of Jasper, add :code:`separable: true` to all
but the last block in the architecture.
Change the parameter name ``jasper``.
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=====================================+
| :code:`feat_in` | int | The number of input features. Should be equal to :code:`features` in the preprocessor parameters. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| :code:`activation` | string | Which activation function to use in the encoder. | :code:`hardtanh`, :code:`relu`, |
| | | | :code:`selu`, :code:`swish` |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| :code:`conv_mask` | bool | Whether to use masked convolutions in the encoder. Defaults to ``true``. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| :code:`jasper` | | A list of blocks that specifies your encoder architecture. Each entry in this list represents one block in | |
| | | the architecture and contains the parameters for that block, including convolution parameters, dropout, and | |
| | | the number of times the block is repeated. Refer to the `Jasper <https://arxiv.org/pdf/1904.03288.pdf>`_ and | |
| | | `QuartzNet <https://arxiv.org/pdf/1910.10261.pdf>`_ papers for details about specific model configurations. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
A QuartzNet 15x5 (fifteen blocks, each sub-block repeated five times) encoder configuration should look similar to the following example:
.. code-block:: yaml
# Specified at the beginning of the file for convenience
n_mels: &n_mels 64 # Used for both the preprocessor and encoder as number of input features
repeat: &repeat 5 # R=5
dropout: &dropout 0.0
separable: &separable true # Set to true for QN. Set to false for Jasper.
model:
...
encoder:
_target_: nemo.collections.asr.modules.ConvASREncoder
feat_in: *n_mels # Should match "features" in the preprocessor.
activation: relu
conv_mask: true
jasper: # This field name should be "jasper" for both types of models.
# Prologue block
- dilation: [1]
dropout: *dropout
filters: 256
kernel: [33]
repeat: 1 # Prologue block is not repeated.
residual: false
separable: *separable
stride: [2]
# Block 1
- dilation: [1]
dropout: *dropout
filters: 256
kernel: [33]
repeat: *repeat
residual: true
separable: *separable
stride: [1]
... # Entries for blocks 2~14
# Block 15
- dilation: [1]
dropout: *dropout
filters: 512
kernel: [75]
repeat: *repeat
residual: true
separable: *separable
stride: [1]
# Two epilogue blocks
- dilation: [2]
dropout: *dropout
filters: 512
kernel: [87]
repeat: 1 # Epilogue blocks are not repeated
residual: false
separable: *separable
stride: [1]
- dilation: [1]
dropout: *dropout
filters: &enc_filters 1024
kernel: [1]
repeat: 1 # Epilogue blocks are not repeated
residual: false
stride: [1]
Both Jasper and QuartzNet use the ``ConvASRDecoder`` as the decoder. The decoder parameters are detailed in the following table.
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=================================+
| :code:`feat_in` | int | The number of input features to the decoder. Should be equal to the number of filters in the last block of | |
| | | the encoder. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`vocabulary` | list | A list of the valid output characters for your model. For example, for an English dataset, this could be a | |
| | | list of all lowercase letters, space, and apostrophe. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`num_classes` | int | Number of output classes, i.e. the length of :code:`vocabulary`. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
For example, a decoder config corresponding to the encoder above should look similar to the following:
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoder
feat_in: *enc_filters
vocabulary: *labels
num_classes: 28 # Length of the vocabulary list
Citrinet
~~~~~~~~
The `Citrinet <./models.html#Citrinet>`__ and `QuartzNet <./models.html#QuartzNet>`__ models are very similar, and as such the
components in their configs are very similar as well. Citrinet utilizes Squeeze and Excitation, as well as sub-word tokenization, in
contrast to QuartzNet. Depending on the dataset, we utilize different tokenizers. For Librispeech, we utilize the HuggingFace WordPiece
tokenizer, and for all other datasets we utilize the Google Sentencepiece tokenizer - usually the ``unigram`` tokenizer type.
Both architectures use the ``ConvASREncoder`` for the ``encoder``, with parameters detailed above. The encoder parameters include
details about the Citrinet-C encoder architecture, including how many filters are used per channel (``C``). The Citrinet-C
configuration is a shortform notation for Citrinet-21x5xC, such that ``B = 21`` and ``R = 5`` are the default and should generally
not be changed.
To use Citrinet instead of QuartzNet, refer to the ``citrinet_512.yaml`` configuration found inside the ``examples/asr/conf/citrinet``
directory. Citrinet is primarily comprised of the same :class:`~nemo.collections.asr.parts.submodules.jasper.JasperBlock` as ``Jasper`` or
``QuartzNet`.
While the configs for Citrinet and QuartzNet are similar, we note the additional flags used for Citrinet below. Refer to the
``JasperBlock`` documentation for the meaning of these arguments.
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+===========================+==================+===========================================================================================================+===================================+
| :code:`se` | bool | Whether to apply squeeze-and-excitation mechanism or not. | :code:`true` or :code:`false` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| :code:`se_context_size` | int | SE context size. -1 means global context. | :code:`-1` or :code:`+ve int` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| :code:`stride_last` | bool | Stride on the final repeated block or all repeated blocks. | :code:`true` or :code:`false` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| :code:`residual_mode` | str | Type of residual branch to construct. | :code:`"add"` or |
| | | Can be pointwise residual addition or pointwise strided residual attention | :code:`"stride_add"` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
A Citrinet-512 config should look similar to the following:
.. code-block:: yaml
model:
...
# Specify some defaults across the entire model
model_defaults:
repeat: 5
dropout: 0.1
separable: true
se: true
se_context_size: -1
...
encoder:
_target_: nemo.collections.asr.modules.ConvASREncoder
feat_in: *n_mels # Should match "features" in the preprocessor.
activation: relu
conv_mask: true
jasper: # This field name should be "jasper" for the JasperBlock (which constructs Citrinet).
# Prologue block
- filters: 512
repeat: 1
kernel: [5]
stride: [1]
dilation: [1]
dropout: 0.0
residual: false
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
# Block 1
- filters: 512
repeat: ${model.model_defaults.repeat}
kernel: [11]
stride: [2]
dilation: [1]
dropout: ${model.model_defaults.dropout}
residual: true
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
stride_last: true
residual_mode: "stride_add"
... # Entries for blocks 2~21
# Block 22
- filters: 512
repeat: ${model.model_defaults.repeat}
kernel: [39]
stride: [1]
dilation: [1]
dropout: ${model.model_defaults.dropout}
residual: true
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
# Epilogue block
- filters: &enc_final 640
repeat: 1
kernel: [41]
stride: [1]
dilation: [1]
dropout: 0.0
residual: false
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
As mentioned above, Citrinet uses the ``ConvASRDecoder`` as the decoder layer similar to QuartzNet. Only the configuration must be
changed slightly as Citrinet utilizes sub-word tokenization.
.. note::
The following information is relevant to any of the above models that implements its encoder as an :class:`~nemo.collections.asr.modules.conv_asr.ConvASREncoder`, and utilizes the ``SqueezeExcite`` mechanism.
The ``SqueezeExcite`` block within a :class:`~nemo.collections.asr.modules.conv_asr.ConvASREncoder` network can be modified to utilize a different context window after the model has been instantiated (even after the model has been trained) so as to evaluate the model with limited context. This can be achieved using the :meth:`~nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin.change_conv_asr_se_context_window`
.. code-block:: python
# Here, model can be any model that has a `ConvASREncoder` as its encoder, and utilized `SqueezeExcite` blocks
# `context_window` : It is an integer representing the number of timeframes (each corresponding to some window stride).
# `update_config` : Bool flag which determines whether the config of the model should be updated to reflect the new context window.
# Here, we specify that 128 timeframes of 0.01s stride should be the context window
# This is equivalent to 128 * 0.01s context window for `SqueezeExcite`
model.change_conv_asr_se_context_window(context_window=128, update_config=True)
Conformer-CTC
~~~~~~~~~~~~~
The config files for Conformer-CTC model contain character-based encoding and sub-word encoding at
``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_char.yaml`` and ``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_bpe.yaml``
respectively. Some components of the configs of `Conformer-CTC <./models.html#Conformer-CTC>`__ include the following datasets:
- ``train_ds``, ``validation_ds``, and ``test_ds``
- opimizer (``optim``)
- augmentation (``spec_augment``)
- ``decoder``
- ``trainer``
``exp_manager``
These datasets are similar to other ASR models like `QuartzNet <./models.html#QuartzNet>`__. There should be a tokenizer section where you can
specify the tokenizer if you want to use sub-word encoding instead of character-based encoding.
The encoder section includes the details about the Conformer-CTC encoder architecture. You may find more information in the
config files and also :doc:`nemo.collections.asr.modules.ConformerEncoder<./api.html#nemo.collections.asr.modules.ConformerEncoder>`.
Fine-tuning Configurations
-------------------------
All ASR scripts support easy fine-tuning by partially/fully loading the pretrained weights from a checkpoint into the currently instantiated model. Pre-trained weights can be provided in multiple ways -
1) Providing a path to a NeMo model (via ``init_from_nemo_model``)
2) Providing a name of a pretrained NeMo model (which will be downloaded via the cloud) (via ``init_from_pretrained_model``)
3) Providing a path to a Pytorch Lightning checkpoint file (via ``init_from_ptl_ckpt``)
Fine-tuning via a NeMo model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: sh
python examples/asr/script_to_<script_name>.py \
--config-path=<path to dir of configs> \
--config-name=<name of config without .yaml>) \
model.train_ds.manifest_filepath="<path to manifest file>" \
model.validation_ds.manifest_filepath="<path to manifest file>" \
trainer.gpus=-1 \
trainer.max_epochs=50 \
+init_from_nemo_model="<path to .nemo model file>"
Fine-tuning via a NeMo pretrained model name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: sh
python examples/asr/script_to_<script_name>.py \
--config-path=<path to dir of configs> \
--config-name=<name of config without .yaml>) \
model.train_ds.manifest_filepath="<path to manifest file>" \
model.validation_ds.manifest_filepath="<path to manifest file>" \
trainer.gpus=-1 \
trainer.max_epochs=50 \
+init_from_pretrained_model="<name of pretrained checkpoint>"
Fine-tuning via a Pytorch Lightning checkpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: sh
python examples/asr/script_to_<script_name>.py \
--config-path=<path to dir of configs> \
--config-name=<name of config without .yaml>) \
model.train_ds.manifest_filepath="<path to manifest file>" \
model.validation_ds.manifest_filepath="<path to manifest file>" \
trainer.gpus=-1 \
trainer.max_epochs=50 \
+init_from_ptl_ckpt="<name of pytorch lightning checkpoint>"
tts/NeMo/docs/source/asr/data/asr_results.csv
+21
View File
@@ -0,0 +1,21 @@
Model Name,Model Base Class,Model Card
QuartzNet15x5Base-En,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels"
stt_en_jasper10x5dr,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr"
stt_en_citrinet_256,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256"
stt_en_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512"
stt_en_citrinet_1024,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024"
stt_ca_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5"
stt_it_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5"
stt_fr_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5"
stt_es_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5"
stt_de_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5"
stt_pl_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5"
stt_ru_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5"
stt_zh_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5"
stt_zh_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512"
stt_en_conformer_ctc_small,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small"
stt_en_conformer_ctc_medium,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium"
stt_en_conformer_ctc_large,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large"
stt_en_conformer_ctc_small_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls"
stt_en_conformer_ctc_medium_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls"
stt_en_conformer_ctc_large_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls"
1 Model Name Model Base Class Model Card
2 QuartzNet15x5Base-En EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels
3 stt_en_jasper10x5dr EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr
4 stt_en_citrinet_256 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256
5 stt_en_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512
6 stt_en_citrinet_1024 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024
7 stt_ca_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5
8 stt_it_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5
9 stt_fr_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5
10 stt_es_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5
11 stt_de_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5
12 stt_pl_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5
13 stt_ru_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5
14 stt_zh_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5
15 stt_zh_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512
16 stt_en_conformer_ctc_small EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small
17 stt_en_conformer_ctc_medium EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium
18 stt_en_conformer_ctc_large EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large
19 stt_en_conformer_ctc_small_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls
20 stt_en_conformer_ctc_medium_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls
21 stt_en_conformer_ctc_large_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls
tts/NeMo/docs/source/asr/data/asrlm_results.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Model Base Class,Model Card
asrlm_en_transformer_large_ls,TransformerLMModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls"
1 Model Name Model Base Class Model Card
2 asrlm_en_transformer_large_ls TransformerLMModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls
tts/NeMo/docs/source/asr/data/benchmark_ca.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class,Model Card
stt_ca_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_ca_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_de.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_de_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_de_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_en.csv
+14
View File
@@ -0,0 +1,14 @@
Model Name,Model Base Class,Model Card
QuartzNet15x5Base-En,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels"
stt_zh_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5"
stt_en_jasper10x5dr,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr"
stt_en_citrinet_256,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256"
stt_en_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512"
stt_en_citrinet_1024,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024"
stt_zh_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512"
stt_en_conformer_ctc_small,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small"
stt_en_conformer_ctc_medium,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium"
stt_en_conformer_ctc_large,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large"
stt_en_conformer_ctc_small_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls"
stt_en_conformer_ctc_medium_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls"
stt_en_conformer_ctc_large_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls"
1 Model Name Model Base Class Model Card
2 QuartzNet15x5Base-En EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels
3 stt_zh_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5
4 stt_en_jasper10x5dr EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr
5 stt_en_citrinet_256 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256
6 stt_en_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512
7 stt_en_citrinet_1024 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024
8 stt_zh_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512
9 stt_en_conformer_ctc_small EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small
10 stt_en_conformer_ctc_medium EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium
11 stt_en_conformer_ctc_large EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large
12 stt_en_conformer_ctc_small_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls
13 stt_en_conformer_ctc_medium_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls
14 stt_en_conformer_ctc_large_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls
tts/NeMo/docs/source/asr/data/benchmark_es.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_es_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_es_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_fr.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_fr_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_fr_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_it.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_it_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_it_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_pl.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class,Model Card
stt_pl_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_pl_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_ru.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_ru_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_ru_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_zh.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_zh_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5"
stt_zh_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512"
1 Model Model Base Class Model Card
2 stt_zh_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5
3 stt_zh_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512
tts/NeMo/docs/source/asr/datasets.rst
+268
View File
@@ -0,0 +1,268 @@
Datasets
========
NeMo has scripts to convert several common ASR datasets into the format expected by the ``nemo_asr`` collection. You can get started
with those datasets by following the instructions to run those scripts in the section appropriate to each dataset below.
If the user has their own data and want to preprocess it to use with NeMo ASR models, refer to the `Preparing Custom ASR Data`_ section.
If the user already has a dataset that you want to convert to a tarred format, refer to the `Tarred Datasets`_ section.
.. _LibriSpeech_dataset:
LibriSpeech
-----------
Run the following scripts to download the LibriSpeech data and convert it into the format expected by `nemo_asr`. At least 250GB free
space is required.
.. code-block:: bash
# install sox
sudo apt-get install sox
mkdir data
python get_librispeech_data.py --data_root=data --data_set=ALL
After this, the ``data`` folder should contain wav files and ``.json`` manifests for NeMo ASR datalayer.
Each line is a training example. ``audio_filepath`` contains the path to the wav file, ``duration`` is the duration in seconds, and ``text`` is the transcript:
.. code-block:: json
{"audio_filepath": "<absolute_path_to>/1355-39947-0000.wav", "duration": 11.3, "text": "psychotherapy and the community both the physician and the patient find their place in the community the life interests of which are superior to the interests of the individual"}
{"audio_filepath": "<absolute_path_to>/1355-39947-0001.wav", "duration": 15.905, "text": "it is an unavoidable question how far from the higher point of view of the social mind the psychotherapeutic efforts should be encouraged or suppressed are there any conditions which suggest suspicion of or direct opposition to such curative work"}
Fisher English Training Speech
------------------------------
Run these scripts to convert the Fisher English Training Speech data into a format expected by the ``nemo_asr`` collection.
In brief, the following scripts convert the ``.sph`` files to ``.wav``, slices those files into smaller audio samples, matches the
smaller slices with their corresponding transcripts, and splits the resulting audio segments into train, validation, and test sets
(with one manifest each).
.. note::
- 106 GB of space is required to run the ``.wav`` conversion
- additional 105 GB is required for the slicing and matching
- ``sph2pipe`` is required in order to run the ``.wav`` conversion
**Instructions**
The following scripts assume that you already have the Fisher dataset from the Linguistic Data Consortium, with a directory structure
that looks similar to the following:
.. code-block:: bash
FisherEnglishTrainingSpeech/
├── LDC2004S13-Part1
│   ├── fe_03_p1_transcripts
│   ├── fisher_eng_tr_sp_d1
│   ├── fisher_eng_tr_sp_d2
│   ├── fisher_eng_tr_sp_d3
│   └── ...
└── LDC2005S13-Part2
├── fe_03_p2_transcripts
├── fe_03_p2_sph1
├── fe_03_p2_sph2
├── fe_03_p2_sph3
└── ...
The transcripts that will be used are located in the ``fe_03_p<1,2>_transcripts/data/trans`` directory. The audio files (``.sph``)
are located in the remaining directories in an ``audio`` subdirectory.
#. Convert the audio files from ``.sph`` to ``.wav`` by running:
.. code-block:: bash
cd <nemo_root>/scripts
python fisher_audio_to_wav.py \
--data_root=<fisher_root> --dest_root=<conversion_target_dir>
This will place the unsliced ``.wav`` files in ``<conversion_target_dir>/LDC200[4,5]S13-Part[1,2]/audio-wav/``. It will take several
minutes to run.
#. Process the transcripts and slice the audio data.
.. code-block:: bash
python process_fisher_data.py \
--audio_root=<conversion_target_dir> --transcript_root=<fisher_root> \
--dest_root=<processing_target_dir> \
--remove_noises
This script splits the full dataset into train, validation, test sets, and places the audio slices in the corresponding folders
in the destination directory. One manifest is written out per set, which includes each slice's transcript, duration, and path.
This will likely take around 20 minutes to run. Once finished, delete the 10 minute long ``.wav`` files.
2000 HUB5 English Evaluation Speech
-----------------------------------
Run the following script to convert the HUB5 data into a format expected by the ``nemo_asr`` collection.
Similarly, to the Fisher dataset processing scripts, this script converts the ``.sph`` files to ``.wav``, slices the audio files and
transcripts into utterances, and combines them into segments of some minimum length (default is 10 seconds). The resulting segments
are all written out to an audio directory and the corresponding transcripts are written to a manifest JSON file.
.. note::
- 5 GB of free space is required to run this script
- ``sph2pipe`` is also required to be installed
This script assumes you already have the 2000 HUB5 dataset from the Linguistic Data Consortium.
Run the following command to process the 2000 HUB5 English Evaluation Speech samples:
.. code-block:: bash
python process_hub5_data.py \
--data_root=<path_to_HUB5_data> \
--dest_root=<target_dir>
You can optionally include ``--min_slice_duration=<num_seconds>`` if you would like to change the minimum audio segment duration.
AN4 Dataset
-----------
This is a small dataset recorded and distributed by Carnegie Mellon University. It consists of recordings of people spelling out
addresses, names, etc. Information about this dataset can be found on the `official CMU site <http://www.speech.cs.cmu.edu/databases/an4/>`_.
#. `Download and extract the dataset <http://www.speech.cs.cmu.edu/databases/an4/an4_sphere.tar.gz>`_ (which is labeled "NIST's Sphere audio (.sph) format (64M)".
#. Convert the ``.sph`` files to ``.wav`` using sox, and build one training and one test manifest.
.. code-block:: bash
python process_an4_data.py --data_root=<path_to_extracted_data>
After the script finishes, the ``train_manifest.json`` and ``test_manifest.json`` can be found in the ``<data_root>/an4/`` directory.
Aishell-1
---------
To download the Aishell-1 data and convert it into a format expected by ``nemo_asr``, run:
.. code-block:: bash
# install sox
sudo apt-get install sox
mkdir data
python get_aishell_data.py --data_root=data
After the script finishes, the ``data`` folder should contain a ``data_aishell`` folder which contains a wav file, a transcript folder, and related ``.json`` and ``vocab.txt`` files.
Aishell-2
---------
To process the AIShell-2 dataset, in the command below, set the data folder of AIShell-2 using ``--audio_folder`` and where to push
these files using ``--dest_folder``. In order to generate files in the supported format of ``nemo_asr``, run:
.. code-block:: bash
python process_aishell2_data.py --audio_folder=<data directory> --dest_folder=<destination directory>
After the script finishes, the ``train.json``, ``dev.json``, ``test.json``, and ``vocab.txt`` files can be found in the ``dest_folder`` directory.
Preparing Custom ASR Data
-------------------------
The ``nemo_asr`` collection expects each dataset to consist of a set of utterances in individual audio files plus
a manifest that describes the dataset, with information about one utterance per line (``.json``).
The audio files can be of any format supported by `Pydub <https://github.com/jiaaro/pydub>`_, though we recommend
WAV files as they are the default and have been most thoroughly tested.
There should be one manifest file per dataset that will be passed in, therefore, if the user wants separate training and validation
datasets, they should also have separate manifests. Otherwise, thay will be loading validation data with their training data and vice
versa.
Each line of the manifest should be in the following format:
.. code::
{"audio_filepath": "/path/to/audio.wav", "text": "the transcription of the utterance", "duration": 23.147}
The :code:`audio_filepath` field should provide an absolute path to the ``.wav`` file corresponding to the utterance.
The :code:`text` field should contain the full transcript for the utterance, and the :code:`duration` field should
reflect the duration of the utterance in seconds.
Each entry in the manifest (describing one audio file) should be bordered by '{' and '}' and must
be contained on one line. The fields that describe the file should be separated by commas, and have the form :code:`"field_name": value`,
as shown above. There should be no extra lines in the manifest, i.e. there should be exactly as many lines in the manifest as
there are audio files in the dataset.
Since the manifest specifies the path for each utterance, the audio files do not have to be located
in the same directory as the manifest, or even in any specific directory structure.
Once there is a manifest that describes each audio file in the dataset, use the dataset by passing
in the manifest file path in the experiment config file, e.g. as ``training_ds.manifest_filepath=<path/to/manifest,json>``.
Tarred Datasets
---------------
If experiments are run on a cluster with datasets stored on a distributed file system, the user will likely
want to avoid constantly reading multiple small files and would prefer tarring their audio files.
There are tarred versions of some NeMo ASR dataset classes for this case, such as the ``TarredAudioToCharDataset``
(corresponding to the ``AudioToCharDataset``) and the ``TarredAudioToBPEDataset`` (corresponding to the
``AudioToBPEDataset``). The tarred audio dataset classes in NeMo use `WebDataset <https://github.com/tmbdev/webdataset>`_.
To use an existing tarred dataset instead of a non-tarred dataset, set ``is_tarred: true`` in
the experiment config file. Then, pass in the paths to all of the audio tarballs in ``tarred_audio_filepaths``, either as a list
of filepaths, e.g. ``['/data/shard1.tar', '/data/shard2.tar']``, or in a single brace-expandable string, e.g.
``'/data/shard_{1..64}.tar'`` or ``'/data/shard__OP_1..64_CL_'`` (recommended, see note below).
.. note::
For brace expansion, there may be cases where ``{x..y}`` syntax cannot be used due to shell interference. This occurs most commonly
inside SLURM scripts. Therefore, we provide a few equivalent replacements. Supported opening braces (equivalent to ``{``) are ``(``,
``[``, ``<`` and the special tag ``_OP_``. Supported closing braces (equivalent to ``}``) are ``)``, ``]``, ``>`` and the special
tag ``_CL_``. For SLURM based tasks, we suggest the use of the special tags for ease of use.
As with non-tarred datasets, the manifest file should be passed in ``manifest_filepath``. The dataloader assumes that the length
of the manifest after filtering is the correct size of the dataset for reporting training progress.
The ``tarred_shard_strategy`` field of the config file can be set if you have multiple shards and are running an experiment with
multiple workers. It defaults to ``scatter``, which preallocates a set of shards per worker which do not change during runtime.
For more information about the individual tarred datasets and the parameters available, including shuffling options,
see the corresponding class APIs in the `Datasets <./api.html#Datasets>`__ section.
.. warning::
If using multiple workers, the number of shards should be divisible by the world size to ensure an even
split among workers. If it is not divisible, logging will give a warning but training will proceed, but likely hang at the last epoch.
In addition, if using distributed processing, each shard must have the same number of entries after filtering is
applied such that each worker ends up with the same number of files. We currently do not check for this in any dataloader, but the user's
program may hang if the shards are uneven.
Conversion to Tarred Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can easily convert your existing NeMo-compatible ASR datasets using the
`conversion script here <https://github.com/NVIDIA/NeMo/blob/v1.0.2/scripts/speech_recognition/convert_to_tarred_audio_dataset.py>`_.
.. code::
python convert_to_tarred_audio_dataset.py \
--manifest_path=<path to the manifest file> \
--target_dir=<path to output directory> \
--num_shards=<number of tarfiles that will contain the audio>
--max_duration=<float representing maximum duration of audio samples> \
--min_duration=<float representing minimum duration of audio samples> \
--shuffle --shuffle_seed=0
This script shuffles the entries in the given manifest (if ``--shuffle`` is set, which we recommend), filter
audio files according to ``min_duration`` and ``max_duration``, and tar the remaining audio files to the directory
``--target_dir`` in ``n`` shards, along with separate manifest and metadata files.
The files in the target directory should look similar to the following:
.. code::
target_dir/
├── audio_1.tar
├── audio_2.tar
├── ...
├── metadata.yaml
└── tarred_audio_manifest.json
Note that file structures are flattened such that all audio files are at the top level in each tarball. This ensures that
filenames are unique in the tarred dataset and the filepaths do not contain "-sub" and forward slashes in each ``audio_filepath`` are
simply converted to underscores. For example, a manifest entry for ``/data/directory1/file.wav`` would be ``_data_directory1_file.wav``
in the tarred dataset manifest, and ``/data/directory2/file.wav`` would be converted to ``_data_directory2_file.wav``.
tts/NeMo/docs/source/asr/images/citrinet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 135 KiB

tts/NeMo/docs/source/asr/images/conformer_ctc.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 103 KiB

tts/NeMo/docs/source/asr/images/ctc_asr.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

tts/NeMo/docs/source/asr/images/jasper_layers.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

tts/NeMo/docs/source/asr/images/jasper_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 273 KiB

tts/NeMo/docs/source/asr/images/quartz_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 257 KiB

tts/NeMo/docs/source/asr/intro.rst
+59
View File
@@ -0,0 +1,59 @@
Automatic Speech Recognition (ASR)
==================================
ASR, or Automatic Speech Recognition, refers to the problem of getting a program to automatically transcribe spoken language
(speech-to-text). Our goal is usually to have a model that minimizes the Word Error Rate (WER) metric when transcribing speech input.
In other words, given some audio file (e.g. a WAV file) containing speech, how do we transform this into the corresponding text with
as few errors as possible?
Traditional speech recognition takes a generative approach, modeling the full pipeline of how speech sounds are produced in order to
evaluate a speech sample. We would start from a language model that encapsulates the most likely orderings of words that are generated
(e.g. an n-gram model), to a pronunciation model for each word in that ordering (e.g. a pronunciation table), to an acoustic model that
translates those pronunciations to audio waveforms (e.g. a Gaussian Mixture Model).
Then, if we receive some spoken input, our goal would be to find the most likely sequence of text that would result in the given audio
according to our generative pipeline of models. Overall, with traditional speech recognition, we try to model ``Pr(audio|transcript)*Pr(transcript)``,
and take the argmax of this over possible transcripts.
Over time, neural nets advanced to the point where each component of the traditional speech recognition model could be replaced by a
neural model that had better performance and that had a greater potential for generalization. For example, we could replace an n-gram
model with a neural language model, and replace a pronunciation table with a neural pronunciation model, and so on. However, each of
these neural models need to be trained individually on different tasks, and errors in any model in the pipeline could throw off the
whole prediction.
Thus, we can see the appeal of end-to-end ASR architectures: discriminative models that simply take an audio input and give a textual
output, and in which all components of the architecture are trained together towards the same goal. The model's encoder would be
akin to an acoustic model for extracting speech features, which can then be directly piped to a decoder which outputs text. If desired,
we could integrate a language model that would improve our predictions, as well.
And the entire end-to-end ASR model can be trained at once--a much easier pipeline to handle!
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
datasets
asr_language_modeling
results
configs
api
Resources and Documentation
---------------------------
Hands-on speech recognition tutorial notebooks can be found under `the ASR tutorials folder <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/asr/>`_.
If you are a beginner to NeMo, consider trying out the `ASR with NeMo <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb>`_ tutorial.
This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular ASR model, or would like to find out more about the model
architectures available in the `nemo_asr` collection, refer to the :doc:`Models <./models>` section.
NeMo includes preprocessing scripts for several common ASR datasets. The :doc:`Datasets <./datasets>` section contains instructions on
running those scripts. It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), as well as a list of the checkpoints
available on NGC are located on the :doc:`Checkpoints <./results>` section.
Documentation regarding the configuration files specific to the ``nemo_asr`` models can be found on the :doc:`Configuration Files <./configs>` section.
tts/NeMo/docs/source/asr/models.rst
+97
View File
@@ -0,0 +1,97 @@
Models
======
This section gives a brief overview of the models that NeMo's ASR collection currently supports.
Each of these models can be used with the example ASR scripts (in the ``<NeMo_git_root>/examples/asr`` directory) by
specifying the model architecture in the config file used. Examples of config files for each model can be found in
the ``<NeMo_git_root>/examples/asr/conf`` directory.
For more information about the config files and how they should be structured, refer to the :doc:`./configs` section.
Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found in the :doc:`./results`
section. You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets. The checkpoints section
also contains benchmark results for the available ASR models.
.. _Jasper_model:
Jasper
------
Jasper ("Just Another Speech Recognizer") :cite:`asr-models-li2019jasper` is a deep time delay neural network (TDNN) comprising of
blocks of 1D-convolutional layers. The Jasper family of models are denoted as ``Jasper_[BxR]`` where ``B`` is the number of blocks
and ``R`` is the number of convolutional sub-blocks within a block. Each sub-block contains a 1-D convolution, batch normalization,
ReLU, and dropout:
.. image:: images/jasper_vertical.png
:align: center
:alt: japer model
:scale: 50%
Jasper models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModel` class.
QuartzNet
---------
QuartzNet :cite:`asr-models-kriman2019quartznet` is a version of Jasper :cite:`asr-models-li2019jasper` model with separable
convolutions and larger filters. It can achieve performance similar to Jasper but with an order of magnitude fewer parameters.
Similarly to Jasper, the QuartzNet family of models are denoted as ``QuartzNet_[BxR]`` where ``B`` is the number of blocks and ``R``
is the number of convolutional sub-blocks within a block. Each sub-block contains a 1-D *separable* convolution, batch normalization,
ReLU, and dropout:
.. image:: images/quartz_vertical.png
:align: center
:alt: quartznet model
:scale: 40%
QuartzNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModel` class.
Citrinet
--------
Citrinet is a version of QuartzNet :cite:`asr-models-kriman2019quartznet` that extends ContextNet :cite:`asr-models-han2020contextnet`,
utilizing subword encoding (via Word Piece tokenization) and Squeeze-and-Excitation mechanism :cite:`asr-models-hu2018squeeze` to
obtain highly accurate audio transcripts while utilizing a non-autoregressive CTC based decoding scheme for efficient inference.
.. image:: images/citrinet_vertical.png
:align: center
:alt: citrinet model
:scale: 50%
Citrinet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModelBPE` class.
.. _Conformer-CTC_model:
Conformer-CTC
-------------
Conformer-CTC is a CTC-based variant of the Conformer model introduced in :cite:`asr-models-gulati2020conformer`. Conformer-CTC has a
similar encoder as the original Conformer but uses CTC loss and decoding instead of RNNT loss, which makes it a non-autoregressive model.
We also drop the LSTM decoder and instead use a linear decoder on the top of the encoder. This model uses the combination of
self-attention and convolution modules to achieve the best of the two approaches, the self-attention layers can learn the global
interaction while the convolutions efficiently capture the local correlations. The self-attention modules support both regular
self-attention with absolute positional encoding, and also Transformer-XL's self-attention with relative positional encodings.
Here is the overall architecture of the encoder of Conformer-CTC:
.. image:: images/conformer_ctc.png
:align: center
:alt: Conformer-CTC Model
:scale: 50%
This model supports both the sub-word level and character level encodings. You can find more details on the config files for the
Conformer-CTC models at `Conformer-CTC <./configs.html#conformer-ctc>`. The variant with sub-word encoding is a BPE-based model
which can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModelBPE` class, while the
character-based variant is based on :class:`~nemo.collections.asr.models.EncDecCTCModel`.
You may find the example config files of Conformer-CTC model with character-based encoding at
``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_char.yaml`` and
with sub-word encoding at ``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_bpe.yaml``.
References
----------
.. bibliography:: asr_all.bib
:style: plain
:labelprefix: ASR-MODELS
:keyprefix: asr-models-
tts/NeMo/docs/source/asr/results.rst
+181
View File
@@ -0,0 +1,181 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo:
* Using the :code:`restore_from()` method to load a local checkpoint file (``.nemo``), or
* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
Refer to the following sections for instructions and examples for each.
Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning. For resuming an unfinished
training experiment, use the Experiment Manager to do so by setting the ``resume_if_exists`` flag to ``True``.
Loading Local Checkpoints
-------------------------
NeMo automatically saves checkpoints of a model that is trained in a ``.nemo`` format. Alternatively, to manually save the model at any
point, issue :code:`model.save_to(<checkpoint_path>.nemo)`.
If there is a local ``.nemo`` checkpoint that you'd like to load, use the :code:`restore_from()` method:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
Where the model base class is the ASR model class of the original checkpoint, or the general ``ASRModel`` class.
NGC Pretrained Checkpoints
--------------------------
The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks. These checkpoints are
obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
The model cards on NGC contain more information about each of the checkpoints available.
The tables below list the ASR models available from NGC. The models can be accessed via the :code:`from_pretrained()` method inside
the ASR Model class. In general, you can load any of these models with code in the following format:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.from_pretrained(model_name="<MODEL_NAME>")
Where the model name is the value under "Model Name" entry in the tables below.
For example, to load the base English QuartzNet model for speech recognition, run:
.. code-block:: python
model = nemo_asr.models.ASRModel.from_pretrained(model_name="QuartzNet15x5Base-En")
You can also call :code:`from_pretrained()` from the specific model class (such as :code:`EncDecCTCModel`
for QuartzNet) if you need to access a specific model functionality.
If you would like to programatically list the models available for a particular base class, you can use the
:code:`list_available_models()` method.
.. code-block:: python
nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
Transcribing/Inference
^^^^^^^^^^^^^^^^^^^^^^
To perform inference and transcribe a sample of speech after loading the model, use the ``transcribe()`` method:
.. code-block:: python
model.transcribe(paths2audio_files=[list of audio files], batch_size=BATCH_SIZE, logprobs=False)
Setting the argument ``logprobs`` to ``True`` returns the log probabilities instead of transcriptions. For more information, see :doc:`./api.html#modules`.
The audio files should be 16KHz monochannel wav files.
Fine-tuning on Different Datasets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are multiple ASR tutorials provided in the :ref:`Tutorials <tutorials>` section. Most of these tutorials explain how to instantiate a pre-trained model, prepare the model for fine-tuning on some dataset (in the same language) as a demonstration.
Automatic Speech Recognition Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/asr_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
Language Models for ASR
^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/asrlm_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
Speech Recognition (Languages)
------------------------------
English
^^^^^^^
.. csv-table::
:file: data/benchmark_en.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Mandarin
^^^^^^^^
.. csv-table::
:file: data/benchmark_zh.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
German
^^^^^^
.. csv-table::
:file: data/benchmark_de.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Polish
^^^^^^
.. csv-table::
:file: data/benchmark_pl.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Italian
^^^^^^^
.. csv-table::
:file: data/benchmark_it.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Russian
^^^^^^^
.. csv-table::
:file: data/benchmark_ru.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Spanish
^^^^^^^
.. csv-table::
:file: data/benchmark_es.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Catalan
^^^^^^^
.. csv-table::
:file: data/benchmark_ca.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
tts/NeMo/docs/source/asr/speaker_diarization/api.rst
+17
View File
@@ -0,0 +1,17 @@
NeMo Speaker Diarization API
=============================
Model Classes
-------------
.. autoclass:: nemo.collections.asr.models.ClusteringDiarizer
:show-inheritance:
:members: init_speaker_model, set_vad_model, init_vad_model, setup_vad_test_data, setup_spkr_test_data, run_vad, extract_embeddings, path2audio_files_to_manifest, diarize, make_nemo_file_from_folder, save_to, unpack_nemo_file, restore_from
Mixins
------
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.DiarizationMixin
:show-inheritance:
:members:
tts/NeMo/docs/source/asr/speaker_diarization/configs.rst
+65
View File
@@ -0,0 +1,65 @@
NeMo Speaker Diarization Configuration Files
============================================
Since speaker diarization model here is not a fully-trainble End-to-End model but an inference pipeline, we use **diarizer** instead of **model** which is used in other tasks.
The diarizer section will generally require information about the dataset(s) being
used, models used in this pipline, as well as inference related parameters such as postprocessing of each models.
The sections on this page cover each of these in more detail.
Example configuration files for speaker diarization can be found in the
``<NeMo_git_root>/examples/speaker_recognition/conf/speaker_diarization.yaml>``
.. note::
For model details and deep understanding about configs, finetuning, tuning threshold, and evaluation,
please refer to ``<NeMo_git_root>/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb``;
for other applications such as possible integration with ASR, have a look at ``<NeMo_git_root>/tutorials/speaker_recognition/ASR_with_SpeakerDiarization.ipynb``.
Dataset Configuration
-----------------------
In contrast to other ASR related tasks or models in NeMo, speaker diarization supported in NeMo is a modular inference pipeline.
Datasets here denotes the data you would like to perform speaker diarization on.
An example Speaker Diarization dataset configuration could look like:
.. code-block:: yaml
diarizer:
oracle_num_speakers: ??? # if number of speakers is known input it or pass null if not known. Accepts int or path to file containing uniq-id and num of speakers of that session
max_num_speakers: 8 # max number of speakers for each recording. If oracle num speakers is passed this value is ignored
out_dir: ???
paths2audio_files: ??? # file containing paths to audio files for diarzation
path2groundtruth_rttm_files: null # (Optional) file containing paths to rttm files (this can be passed if we need to calculate DER rate based on ground truth rttm files).
...
.. note::
We expect audio and the corresponding RTTM to have the same base name and the name should be unique.
Diarizer Architecture Configurations
-------------------------------------
.. code-block:: yaml
diarizer:
...
vad:
model_path: null #.nemo local model path or pretrained model name or none
window_length_in_sec: 0.15
shift_length_in_sec: 0.01
threshold: 0.5 # tune threshold on dev set. Check <NeMo_git_root>/scripts/voice_activity_detection/vad_tune_threshold.py
vad_decision_smoothing: True
smoothing_params:
method: "median"
overlap: 0.875
speaker_embeddings:
oracle_vad_manifest: null # leave it null if to perform diarization with above VAD model else path to manifest file genrerated as shown in Datasets section
model_path: ??? #.nemo local model path or pretrained model name
window_length_in_sec: 1.5 # for better performace try with 3 sec and 1.5 sec
shift_length_in_sec: 0.75 # for better performace try with 1.5 sec and 0.75 sec
tts/NeMo/docs/source/asr/speaker_diarization/data/diarization_results.csv
+4
View File
@@ -0,0 +1,4 @@
Model Name,Model Base Class,Model Card
vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
speakerdiarization_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet"
1 Model Name Model Base Class Model Card
2 vad_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet
3 vad_telephony_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet
4 speakerdiarization_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet
tts/NeMo/docs/source/asr/speaker_diarization/datasets.rst
+72
View File
@@ -0,0 +1,72 @@
Datasets
========
Check out page :doc:`Speech Classification Datasets <../speech_classification/datasets>` and :doc:`Speaker Recogniton Datasets <../speaker_recognition/datasets>`
for preparing datasets for training and validating VAD and speaker embedding models respectively.
For Speaker Diarization inference, ``diarizer`` expects following arguments :
- paths2audio_files
- oracle_num_speakers
- path2groundtruth_rttm_files [Optional]
- oracle_vad_manifest (if vad/rttm files are known)[Optional but suggested for good performace. This avoids FPR and TPR]
paths2audio_files:
- a file containing absolute paths to audio files or list of paths to audio files.
For generating a file that contains paths to audio files, you can simply use ``find`` bash command as shown below:
.. code-block:: bash
find $PWD/{data_dir} -iname '*.wav' > path_to_audiofiles.txt
.. note::
We expect audio and the corresponding RTTM files to have the same base name and the name should be unique (uniq-id).
oracle_num_speakers:
- if number of speakers is known input it or pass null if not known. Accepts int or path to file containing uniq-id with num of speakers of that session
Sample line (uniq-id `iaaa` session has 2 unique speakers )::
iaaa 2
path2groundtruth_rttm_files [Optional]:
- To evaluate the diarizer system with known rttm files, One needs to provide a txt file (path2groundtruth_rttm_files) for groundtruth label files.
use above mentioned ``find`` command to get all reference rttm files (use '*.rttm' as search pattern)
Each groundtruth label file should be in NIST Rich Transcription Time Marked (RTTM) format. Take one line from a RTTM file for example:
.. code-block:: bash
SPEAKER TS3012d.Mix-Headset 1 331.573 0.671 <NA> <NA> MTD046ID <NA> <NA>
oracle_vad_manifest [Optional but suggested for good performace. This reduces FPR and TPR]:
- To perform just oracle diarization, that is taking speech activity time stamps from groundtruth RTTMs or another VAD from wild instead of NeMo VAD output, ``diarizer`` expects an oracle manifest json file that contains paths to audio files with offset for start time and duration of segment.
To prepare an oracle manifest file, use the script from ``scripts`` folder as shown below:
.. code-block:: bash
python $NeMo/scripts/ --paths2rttm_files=<path2groundtruth_rttm_files.txt> --paths2audio_files=<paths2audio_files.txt> --manifest_file=<output_oracle_manifest_file.json>
Here ``paths2audio_files`` and ``path2groundtruth_rttm_files`` are files containing paths to audio files as shown above.
AMI Meeting Corpus
------------------
The following are the suggested parameters for getting Speaker Error Rate (SER) of 4.1% on AMI Lapel test set corpus:
- diarizer.oracle_num_speakers = 4 (since there are exactly 4 speakers per each Lapel test set session)
- diarizer.speaker_embeddings.model_path = ``speakerverification_speakernet`` (This model is trained on voxceleb dataset. ``Use this model for simialr non-telephonic speech datasets``)
- diarizer.speaker_embeddings.window_length_in_sec = 3
- diarizer.speaker_embeddings.shift_length_in_sec = 1.5
Input paths2audio_files, paths2rttm_files and oracle_vad_manifest by following steps as shown above
CallHome LDC97S42 (CH109)
-------------------------
The following are the suggested parameters for getting Speaker Error Rate (SER) of 5.4% on CH109 set:
- diarizer.oracle_num_speakers = 2 (since there are exactly 2 speakers per each ch109 session)
- diarizer.speaker_embeddings.model_path = ``speakerdiarization_speakernet`` (This model is trained on voxceleb and telephonic speech Fisher and SWBD. ``Use this model for similar telephonic speech datasets``)
- diarizer.speaker_embeddings.window_length_in_sec = 1.5
- diarizer.speaker_embeddings.shift_length_in_sec = 0.75
Input paths2audio_files, paths2rttm_files and oracle_vad_manifest by following steps as shown above
tts/NeMo/docs/source/asr/speaker_diarization/images/sd_pipeline.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

tts/NeMo/docs/source/asr/speaker_diarization/intro.rst
+52
View File
@@ -0,0 +1,52 @@
Speaker Diarization
==================================
Speaker Diarization (SD) is the task of segmenting audio recordings by speaker labels, that is Who Speaks When?
.. image:: images/sd_pipeline.png
:align: center
:alt: Speaker Diarization pipeline [todo]
:scale: 50%
A diarization system consists of a **Voice Activity Detection (VAD)** model to get the time stamps of audio where speech is being spoken while ignoring the background noise and
a **Speaker Embeddings** model to get speaker embeddings on speech segments obtained from VAD time stamps.
These speaker embeddings would then be clustered into clusters based on number of speakers present in the audio recording.
In NeMo we support both **oracle VAD** and **non-oracle VAD** diarization.
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
datasets
results
configs
api
Resource and Documentation Guide
--------------------------------
Hands-on speaker diarization tutorial notebooks can be found under ``<NeMo_git_root>/tutorials/speaker_recognition/``.
There are tutorials for peformming inference using :ref:`MarbleNet_model` and :ref:`SpeakerNet_model`,
and how one can get ASR transcriptions combined with Speaker labels along with voice activity time stamps with NeMo asr collections.
Most of the tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular model used for speaker diarization inference, or would like to find out more about the model
architectures available in the `nemo_asr` collection, check out the :doc:`Models <./models>` page.
Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
NeMo includes preprocessing scripts for several common ASR datasets, and this page contains instructions on running
those scripts.
It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
:doc:`Configuration Files <./configs>` page.
tts/NeMo/docs/source/asr/speaker_diarization/models.rst
+5
View File
@@ -0,0 +1,5 @@
Models
======
Currenlty NeMo's Speaker Diarization pipeline uses `MarbleNet <../speech_classification/models.html#marblenet-vad>`__ model for Voice Activity Detection (VAD) and `SpeakerNet <../speaker_recognition/models.html#speakernet>`__ model for Speaker Embedding Extraction.
tts/NeMo/docs/source/asr/speaker_diarization/results.rst
+76
View File
@@ -0,0 +1,76 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo as introduced in `loading ASR checkpoints <../results.html#checkpoints>`__.
In speaker diarization, the diarizer loads checkpoints that are passed through the config file. For example:
Loading Local Checkpoints
---------------------------
Load VAD model
.. code-block:: bash
pretrained_vad_model='/path/to/vad_marblenet.nemo' # local .nemo or pretrained vad model name
...
# pass with hydra config
config.diarizer.vad.model_path=pretrained_vad_model
Load Speaker Embedding model
.. code-block:: bash
pretrained_speaker_model='/path/to/speakerdiarization_speakernet.nemo' # local .nemo or pretrained speakernet model name
...
# pass with hydra config
config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
Inference
----------
.. note::
For details and deep understanding, please refer to ``<NeMo_git_root>/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb``.
Check out :doc:`Datasets <./datasets>` for preparing audio files and optional label files.
Run and evaluate speaker diarizer with below command:
.. code-block:: bash
# Have a look at the instruction inside the script and pass the arguments you might need.
python <NeMo_git_root>/examples/speaker_recognition/speaker_diarize.py
NGC Pretrained Checkpoints
----------------------------
The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
The model cards on NGC contain more information about each of the checkpoints available.
In general, you can load models with model name in the following format,
.. code-block:: python
pretrained_vad_model='vad_telephony_marblenet'
pretrained_speaker_model='speakerdiarization_speakernet'
...
config.diarizer.vad.model_path=retrained_vad_model \
config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
where the model name is the value under "Model Name" entry in the tables below.
Models for Speaker Diarization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/diarization_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
tts/NeMo/docs/source/asr/speaker_recognition/configs.rst
+117
View File
@@ -0,0 +1,117 @@
NeMo ASR Configuration Files
============================
This page covers NeMo configuration file setup that is specific to speaker recognition models.
For general information about how to set up and run experiments that is common to all NeMo models (e.g.
experiment manager and PyTorch Lightning trainer parameters), see the :doc:`../../core/core` page.
The model section of NeMo ASR configuration files will generally require information about the dataset(s) being
used, the preprocessor for audio files, parameters for any augmentation being performed, as well as the
model architecture specification.
The sections on this page cover each of these in more detail.
Example configuration files for all of the Speaker related scripts can be found in the
config directory of the examples ``{NEMO_ROOT/examples/speaker_recognition/conf}``.
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections of your configuration file, respectively.
Depending on the task, you may have arguments specifying the sample rate of your audio files, max time length to consider for each audio file , whether or not to shuffle the dataset, and so on.
You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command line
at runtime.
Any initialization parameters that are accepted for the Dataset class used in your experiment
can be set in the config file.
An example SpeakerNet train and validation configuration could look like:
.. code-block:: yaml
model:
train_ds:
manifest_filepath: ???
sample_rate: 16000
labels: None # finds labels based on manifest file
batch_size: 32
trim_silence: False
time_length: 8
shuffle: True
validation_ds:
manifest_filepath: ???
sample_rate: 16000
labels: None # Keep None, to match with labels extracted during training
batch_size: 32
shuffle: False # No need to shuffle the validation data
If you would like to use tarred dataset, have a look at `Datasets Configuration <../configs.html#dataset-configuration>`__.
Preprocessor Configuration
--------------------------
Preprocessor helps to compute MFCC or mel spectrogram features that are given as inputs to model.
For details on how to write this section, refer to `Preprocessor Configuration <../configs.html#preprocessor-configuration>`__
Augmentation Configurations
---------------------------
For SpeakerNet training we use on-the fly augmentations with MUSAN and RIR impulses using ``noise`` augmentor section
The following example sets up musan augmentation with audio files taken from manifest path and
minimum and maximum SNR specified with min_snr and max_snr respectively. This section can be added to
``train_ds`` part in model
.. code-block:: yaml
model:
...
train_ds:
...
augmentor:
noise:
manifest_path: /path/to/musan/manifest_file
prob: 0.2 # probability to augment the incoming batch audio with augmentor data
min_snr_db: 5
max_snr_db: 15
See the :class:`nemo.collections.asr.parts.preprocessing.perturb.AudioAugmentor` API section for more details.
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment.
Models in the NeMo ASR collection need a ``encoder`` section and a ``decoder`` section, with the ``_target_`` field
specifying the module to use for each.
The following sections go into more detail about the specific configurations of each model architecture.
For more information about the SpeakerNet Encoder models, see the :doc:`Models <./models>` page and at `Jasper and QuartzNet <../configs.html#jasper-and-quartznet>`__
Decoder Configurations
------------------------
After features have been computed from speakernet encoder, we pass these features to the decoder to compute embeddings and then to compute log probabilities
for training models.
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.SpeakerDecoder
feat_in: *enc_final
num_classes: 7205 # Total number of classes in training manifest file
pool_mode: xvector # xvector for variance and mean bases statistics pooling
emb_sizes: 256 # number of inermediate emb layers. can be comma separated for additional layers like 512,512
angular: true # if true then loss will be changed to angular softmax loss and consider scale and margin from loss section else train with cross-entrophy loss
loss:
scale: 30
margin 0.2
tts/NeMo/docs/source/asr/speaker_recognition/data/speaker_results.csv
+4
View File
@@ -0,0 +1,4 @@
Model Name,Model Base Class,Model Card
speakerverification_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet"
speakerrecognition_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet"
speakerdiarization_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet"
1 Model Name Model Base Class Model Card
2 speakerverification_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet
3 speakerrecognition_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet
4 speakerdiarization_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet
tts/NeMo/docs/source/asr/speaker_recognition/datasets.rst
+57
View File
@@ -0,0 +1,57 @@
Datasets
========
.. _HI-MIA:
HI-MIA
--------
Run the script to download and process ``hi-mia`` dataset in order to generate files in the supported format of ``nemo_asr``. You should set the data folder of
hi-mia using ``--data_root``. These scripts are present in ``<nemo_root>/scripts``
.. code-block:: bash
python get_hi-mia_data.py --data_root=<data directory>
After download and conversion, your `data` folder should contain directories with following set of files as:
* `data/<set>/train.json`
* `data/<set>/dev.json`
* `data/<set>/{set}_all.json`
* `data/<set>/utt2spk`
All-other Datasets
------------------
These methods can be applied to any dataset to get similar training manifest files.
First we prepare scp file(s) containing absolute paths to all the wav files required for each of the train, dev, and test set. This can be easily prepared by using ``find`` bash command as follows:
.. code-block:: bash
!find {data_dir}/{train_dir} -iname "*.wav" > data/train_all.scp
!head -n 3 data/train_all.scp
Since we created the scp file for the train, we use `scp_to_manifest.py` to convert this scp file to a manifest file and then optionally split the files to train \& dev for evaluating the models while training by using the --split flag.
We wont be needing the --split option for the test folder. Accordingly please mention the id number, which is the field num separated by / to be considered as the speaker label.
After the download and conversion, your data folder should contain directories with manifest files as:
* `data/<path>/train.json`
* `data/<path>/dev.json`
* `data/<path>/train_all.json`
Each line in the manifest file describes a training sample - audio_filepath contains the path to the wav file, duration it's duration in seconds, and label is the speaker class label:
.. code-block:: json
{"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": 3.9, "label": "speaker_id"}
Tarred Datasets
---------------
Similarly to ASR, you can tar your audio files and use ASR Dataset class ``TarredAudioToSpeechLabelDataset`` (corresponding to the ``AudioToSpeechLabelDataset``) for this case.
If you want to use tarred dataset, have a look at `ASR Tarred Datasets <../datasets.html#tarred-datasets>`__.
tts/NeMo/docs/source/asr/speaker_recognition/images/ICASPP_SpeakerNet.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 207 KiB

tts/NeMo/docs/source/asr/speaker_recognition/intro.rst
+48
View File
@@ -0,0 +1,48 @@
Speaker Recognition (SR)
========================
Speaker Recognition (SR) is a broad research area which solves two major tasks: speaker identification (who is speaking?) and speaker verification (is the speaker who they claim to be?).
We focus on far-field, text-independent speaker recognition when the identity of the speaker is based on how the speech is spoken, not necessarily in what is being said.
Typically such SR systems operate on unconstrained speech utterances, which are converted into vectors of fixed length, called speaker embeddings.
Speaker embeddings can also be used in automatic speech recognition (ASR) and speech synthesis.
As the goal of most speaker recognition systems is to get good speaker level embeddings that could help distinguish from other speakers, we shall first train these embeddings in end-to-end manner optimizing the QuatzNet based encoder model on cross-entropy loss.
We modify the decoder to get these fixed size embeddings irrespective of the length of the input audio and employ a mean and variance based statistics pooling method to grab these embeddings.
In Speaker Identification we typically train on a larger training set with cross-entrophy loss and finetune later on your preferred set of labels where one would want to classify only known set of speakers.
And in Speaker verification we train with Angular Softmax loss and compare embedings extracted from one audio file coming from a single speaker with
embeddings extracted from another file of same or another speaker by employing backend scoring techniques like cosine similarity.
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
configs
datasets
results
Resource and Documentation Guide
--------------------------------
Hands-on speaker recognition tutorial notebooks can be found under
`the speaker recognition tutorials folder <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/speaker_recognition/>`_. This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular SpeakerNet model, or would like to find out more about the model
architectures available in the ``nemo_asr`` collection, check out the :doc:`Models <./models>` page.
Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
NeMo includes preprocessing and other scripts for speaker_recognition in <nemo/scripts/speaker_recognition/> folder, and this page contains instructions on running
those scripts. It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
:doc:`Configuration Files <./configs>` page.
For a clear step-by-step tutorial we advice you to refer tutorials found in `folder <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/speaker_recognition/>`_.
tts/NeMo/docs/source/asr/speaker_recognition/models.rst
+52
View File
@@ -0,0 +1,52 @@
Models
======
Examples of config files for the following quartznet model can be found in the ``<NeMo_git_root>/examples/speaker_recogniton/conf`` directory.
For more information about the config files and how they should be structured, see the :doc:`./configs` page.
Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found on the :doc:`./results` page.
You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets.
The Checkpoints page also contains benchmark results for the available speaker recognition models.
.. _SpeakerNet_model:
SpeakerNet
-----------
The model is based on the QuartzNet ASR architecture :cite:`sr-models-koluguri2020speakernet`
comprising of an encoder and decoder structure. We use the encoder of the QuartzNet model as a top-level feature extractor, and feed the output to the statistics pooling layer, where
we compute the mean and variance across channel dimensions to capture the time-independent utterance-level speaker features.
The QuartzNet encoder used for speaker embeddings shown in figure below has the following structure: a QuartzNet BxR
model has B blocks, each with R sub-blocks. Each sub-block applies the following operations: a 1D convolution, batch norm, ReLU, and dropout. All sub-blocks in a block have the same number of output channels. These blocks are connected with residual connections. We use QuartzNet with 3 blocks, 2 sub-blocks, and 512 channels, as the Encoder for Speaker Embeddings. All conv layers have stride 1 and dilation 1.
.. image:: images/ICASPP_SpeakerNet.png
:align: center
:alt: speakernet model
:scale: 40%
Top level acoustic Features, obtained from the output of
encoder are used to compute intermediate features that are
then passed to the decoder for getting utterance level speaker
embeddings. The intermediate time-independent features are
computed using a statistics pooling layer, where we compute the mean and standard deviation of features across
time-channels, to get a time-independent feature representation S of size Batch_size × 3000.
The intermediate features, S are passed through the Decoder consisting of two layers each of output size 512 for a
linear transformation from S to the final number of classes
N for the larger (L) model, and a single linear layer of output size 256 to the final number of classes N for the medium
(M) model. We extract q-vectors after the final linear layer
of fixed size 512, 256 for SpeakerNet-L and SpeakerNet-M
models respectively.
SpeakerNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecSpeakerLabelModel` class.
References
-----------
.. bibliography:: ../asr_all.bib
:style: plain
:labelprefix: SR-MODELS
:keyprefix: sr-models-
tts/NeMo/docs/source/asr/speaker_recognition/results.rst
+85
View File
@@ -0,0 +1,85 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo:
* Using the :code:`restore_from()` method to load a local checkpoint file (`.nemo`), or
* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
See the following sections for instructions and examples for each.
Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning.
For resuming an unfinished training experiment, please use the experiment manager to do so by setting the
``resume_if_exists`` flag to True.
Loading Local Checkpoints
-------------------------
NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
If you have a local ``.nemo`` checkpoint that you'd like to load, simply use the :code:`restore_from()` method:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
Where the model base class is the ASR model class of the original checkpoint, or the general `ASRModel` class.
Inference
----------
The audio files should be 16KHz monochannel wav files.
Write audio files to a ``manifest.json`` file with lines as in format:
.. code-block:: json
{"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": "duration of file in sec", "label": "speaker_id"}
This python call will download best pretrained model from NGC and writes embeddings pickle file to current working directory
.. code-block:: bash
python examples/speaker_recognition/extract_speaker_embeddings.py --manifest=manifest.json
NGC Pretrained Checkpoints
--------------------------
The SpeakerNet-ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
`Speaker_Recognition <https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet>`_ and `Speaker_Verification <https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet>`_ model cards on NGC contain more information about each of the checkpoints available.
The tables below list the SpeakerNet models available from NGC, and the models can be accessed via the
:code:`from_pretrained()` method inside the EncDecSpeakerLabelModel Model class.
.. note:: While loading, remember to use EncDecSpeakerLabelModel for recognition tasks and ExtractSpeakerEmbeddingsModel while extracting embeddings.
In general, you can load any of these models with code in the following format.
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_CLASS_NAME>.from_pretrained(model_name="<MODEL_NAME>")
Where the model name is the value under "Model Name" entry in the tables below.
If you would like to programatically list the models available for a particular base class, you can use the
:code:`list_available_models()` method.
.. code-block:: python
nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
Speaker Recognition Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/speaker_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
tts/NeMo/docs/source/asr/speech_classification/configs.rst
+104
View File
@@ -0,0 +1,104 @@
NeMo Speech Classification Configuration Files
================================================
This page covers NeMo configuration file setup that is specific to models in the Speech Classification collection.
For general information about how to set up and run experiments that is common to all NeMo models (e.g.
experiment manager and PyTorch Lightning trainer parameters), see the :doc:`../../core/core` page.
The model section of NeMo Speech Classification configuration files will generally require information about the dataset(s) being
used, the preprocessor for audio files, parameters for any augmentation being performed, as well as the
model architecture specification.
The sections on this page cover each of these in more detail.
Example configuration files for all of the NeMo ASR scripts can be found in the
``<NeMo_git_root>/examples/asr/conf>``.
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections of your configuration file, respectively.
Depending on the task, you may have arguments specifying the sample rate of your audio files, labels, whether or not to shuffle the dataset, and so on.
You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command line
at runtime.
Any initialization parameters that are accepted for the Dataset class used in your experiment
can be set in the config file.
See the `Datasets <../api.html#Datasets>`__ section of the API for a list of Datasets and their respective parameters.
An example Speech Classification train and validation configuration could look like:
.. code-block:: yaml
model:
sample_rate: 16000
repeat: 2 # number of convolutional sub-blocks within a block, R in <MODEL>_[BxRxC]
dropout: 0.0
kernel_size_factor: 1.0
labels: ['bed', 'bird', 'cat', 'dog', 'down', 'eight', 'five', 'four', 'go', 'happy', 'house', 'left', 'marvin',
'nine', 'no', 'off', 'on', 'one', 'right', 'seven', 'sheila', 'six', 'stop', 'three', 'tree', 'two', 'up',
'wow', 'yes', 'zero']
train_ds:
manifest_filepath: ???
sample_rate: ${model.sample_rate}
labels: ${model.labels} # Uses the labels above
batch_size: 128
shuffle: True
validation_ds:
manifest_filepath: ???
sample_rate: ${model.sample_rate}
labels: ${model.labels} # Uses the labels above
batch_size: 128
shuffle: False # No need to shuffle the validation data
If you would like to use tarred dataset, have a look at `Datasets Configuration <../configs.html#dataset-configuration>`__.
Preprocessor Configuration
--------------------------
Preprocessor helps to compute MFCC or mel spectrogram features that are given as inputs to model.
For details on how to write this section, refer to `Preprocessor Configuration <../configs.html#preprocessor-configuration>`__
Check config yaml files in ``<NeMo_git_root>/examples/asr/conf`` to find the processors been used by speech classification models.
Augmentation Configurations
---------------------------
There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
configuration file using the ``augmentor`` and ``spec_augment`` section.
For details on how to write this section, refer to `Augmentation Configuration <../configs.html#augmentation-configurations>`__
Check config yaml files in ``<NeMo_git_root>/tutorials/asr/conf`` to find the processors been used by speech classification models.
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment.
Models in the NeMo ASR collection need a ``encoder`` section and a ``decoder`` section, with the ``_target_`` field
specifying the module to use for each.
The following sections go into more detail about the specific configurations of each model architecture.
The `MatchboxNet <./models.html#matchboxnet-speech-commands>`__ and `MarbleNet <./models.html#marblenet-vad>`__ models are very similar, and they are based on `QuartzNet <../models.html#quartznet>`__ and as such the components in their
configs are very similar as well.
Decoder Configurations
------------------------
After features have been computed from ConvASREncoder, we pass the features to decoder to compute embeddings and then to compute log_probs
for training models.
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoderClassification
feat_in: *enc_final_filters
return_logits: true # return logits if true, else return softmax output
pooling_type: 'avg' # AdaptiveAvgPool1d 'avg' or AdaptiveMaxPool1d 'max'
tts/NeMo/docs/source/asr/speech_classification/data/classification_results.csv
+9
View File
@@ -0,0 +1,9 @@
Model Name,Model Base Class,Model Card
vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
commandrecognition_en_matchboxnet3x1x64_v1,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v1"
commandrecognition_en_matchboxnet3x2x64_v1,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v1"
commandrecognition_en_matchboxnet3x1x64_v2,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2"
commandrecognition_en_matchboxnet3x2x64_v2,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2"
commandrecognition_en_matchboxnet3x1x64_v2_subset_task,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2_subset_task"
commandrecognition_en_matchboxnet3x2x64_v2_subset_task,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2_subset_task"
1 Model Name Model Base Class Model Card
2 vad_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet
3 vad_telephony_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet
4 commandrecognition_en_matchboxnet3x1x64_v1 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v1
5 commandrecognition_en_matchboxnet3x2x64_v1 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v1
6 commandrecognition_en_matchboxnet3x1x64_v2 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2
7 commandrecognition_en_matchboxnet3x2x64_v2 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2
8 commandrecognition_en_matchboxnet3x1x64_v2_subset_task EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2_subset_task
9 commandrecognition_en_matchboxnet3x2x64_v2_subset_task EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2_subset_task
tts/NeMo/docs/source/asr/speech_classification/datasets.rst
+131
View File
@@ -0,0 +1,131 @@
Datasets
========
NeMo has scripts to convert several common ASR datasets into the format expected by the `nemo_asr` collection.
You can get started with those datasets by following the instructions to run those scripts in the section appropriate to each dataset below.
If you have your own data and want to preprocess it to use with NeMo ASR models, check out the `Preparing Custom Speech Classification Data`_ section at the bottom of the page.
.. _Freesound-dataset:
Freesound
-----------
`Freesound <http://www.freesound.org/>`_ is a website that aims to create a huge open collaborative database of audio snippets, samples, recordings, bleeps.
Most audio samples are released under Creative Commons licenses that allow their reuse.
Researchers and developers can access Freesound content using the Freesound API to retrieve meaningful sound information such as metadata, analysis files, and the sounds themselves.
**Instructions**
Go to ``<NeMo_git_root>/scripts/freesound_download_resample`` and follow the below steps to download and convert freedsound data into a format expected by the `nemo_asr` collection.
1. We will need some required libraries including freesound, requests, requests_oauthlib, joblib, librosa and sox. If they are not installed, please run `pip install -r freesound_requirements.txt`
2. Create an API key for freesound.org at https://freesound.org/help/developers/
3. Create a python file called `freesound_private_apikey.py` and add lined `api_key = <your Freesound api key> and client_id = <your Freesound client id>`
4. Authorize by run `python freesound_download.py --authorize` and visit the website and paste response code
5. Feel free to change any arguments in `download_resample_freesound.sh` such as max_samples and max_filesize
6. Run `bash download_resample_freesound.sh <numbers of files you want> <download data directory> <resampled data directory>` . For example:
.. code-block:: bash
bash download_resample_freesound.sh 4000 ./freesound ./freesound_resampled_background
Note that downloading this dataset may take hours. Change categories in download_resample_freesound.sh to include other (speech) categories audio files.
Then, you should have 16khz mono wav files in `<resampled data directory>`.
.. _Google-Speech-Commands-Dataset:
Google Speech Commands Dataset
------------------------------
Google released two versions of the dataset with the first version containing 65k samples over 30 classes and the second containing 110k samples over 35 classes.
We refer to these datasets as `v1` and `v2` respectively.
Run the script `process_speech_commands_data.py` to process Google Speech Commands dataset in order to generate files in the supported format of `nemo_asr`,
which can be found in ``<NeMo_git_root>/scripts/dataset_processing/``.
You should set the data folder of Speech Commands using :code:`--data_root` and the version of the dataset using :code:`--data_version` as an int.
You can further rebalance the train set by randomly oversampling files inside the manifest by passing the `--rebalance` flag.
.. code-block:: bash
python process_speech_commands_data.py --data_root=<data directory> --data_version=<1 or 2> {--rebalance}
Then, you should have `train_manifest.json`, `validation_manifest.json` and `test_manifest.json`
in the directory `{data_root}/google_speech_recognition_v{1/2}`.
.. note::
You should have at least 4GB or 6GB of disk space available if you use v1 or v2 respectively.
Also, it will take some time to download and process, so go grab a coffee.
Each line is a training example.
.. code-block:: bash
{"audio_filepath": "<absolute path to dataset>/two/8aa35b0c_nohash_0.wav", "duration": 1.0, "label": "two"}
{"audio_filepath": "<absolute path to dataset>/two/ec5ab5d5_nohash_2.wav", "duration": 1.0, "label": "two"}
Speech Command & Freesound for VAD
------------------------------------
Speech Command & Freesound (SCF) dataset is used to train MarbleNet in the `paper <https://arxiv.org/pdf/2010.13886.pdf>`_. Here we show how to download and process it.
This script assumes that you already have the Freesound dataset, if not, have a look at :ref:`Freesound-dataset`.
We will use the open-source :ref:`Google-Speech-Commands-Dataset` (we will use V2 of the dataset for SCF dataset, but require very minor changes to support V1 dataset) as our speech data.
These scripts below will download the Google Speech Commands v2 dataset and convert speech and background data to a format suitable for use with nemo_asr.
.. note::
You may additionally pass :code:`--test_size` or :code:`--val_size` flag for splitting train val and test data.
You may additionally pass :code:`--seg_len` flag for indicating the segment length. Default is 0.63s.
You may additionally pass a :code:`-rebalance_method='fixed|over|under'` at the end of the script to rebalance the class samples in the manifest.
* `'fixed'`: Fixed number of sample for each class. Train 5000, val 1000, and test 1000. (Change number in script if you want)
* `'over'`: Oversampling rebalance method
* `'under'`: Undersampling rebalance method
.. code-block:: bash
mkdir './google_dataset_v2'
python process_vad_data.py --out_dir='./manifest/' --speech_data_root='./google_dataset_v2'--background_data_root=<resampled freesound data directory> --log --rebalance_method='fixed'
After download and conversion, your `manifest` folder should contain a few json manifest files:
* `(balanced_)background_testing_manifest.json`
* `(balanced_)background_training_manifest.json`
* `(balanced_)background_validation_manifest.json`
* `(balanced_)speech_testing_manifest.json`
* `(balanced_)speech_training_manifest.json`
* `(balanced_)speech_validation_manifest.json`
Each line is a training example. `audio_filepath` contains path to the wav file, `duration` is duration in seconds, `offset` is offset in seconds, and `label` is label (class):
.. code-block:: bash
{"audio_filepath": "<absolute path to dataset>/two/8aa35b0c_nohash_0.wav", "duration": 0.63, "label": "speech", "offset": 0.0}
{"audio_filepath": "<absolute path to dataset>/Emergency_vehicle/id_58368 simambulance.wav", "duration": 0.63, "label": "background", "offset": 4.0}
Preparing Custom Speech Classification Data
--------------------------------------------
Preparing Custom Speech Classification Data is almost identical to `Preparing Custom ASR Data <../datasets.html#preparing-custom-asr-data>`__.
Instead of :code:`text` entry in manifest, you need :code:`label` to determine class of this sample
Tarred Datasets
---------------
Similarly to ASR, you can tar your audio files and use ASR Dataset class ``TarredAudioToClassificationLabelDataset`` (corresponding to the ``AudioToClassificationLabelDataset``) for this case.
If you would like to use tarred dataset, have a look at `ASR Tarred Datasets <../datasets.html#tarred-datasets>`__.
tts/NeMo/docs/source/asr/speech_classification/images/marblenet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 384 KiB

tts/NeMo/docs/source/asr/speech_classification/images/matchboxnet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 479 KiB

tts/NeMo/docs/source/asr/speech_classification/intro.rst
+47
View File
@@ -0,0 +1,47 @@
Speech Classification
==================================
Speech Classification refers to a set of tasks or problems of getting a program to automatically classify input utterance or audio segment into categories,
such as Speech Command Recognition (multi-class), Voice Activity Detection (binary or multi-class), and Audio Sentiment Classification (typically multi-class), etc.
**Speech Command Recognition** is the task of classifying an input audio pattern into a discrete set of classes.
It is a subset of Automatic Speech Recognition, sometimes referred to as Key Word Spotting, in which a model is constantly analyzing speech patterns to detect certain "command" classes.
Upon detection of these commands, a specific action can be taken by the system.
It is often the objective of command recognition models to be small and efficient so that they can be deployed onto low-power sensors and remain active for long durations of time.
**Voice Activity Detection (VAD)** also known as speech activity detection or speech detection, is the task of predicting which parts of input audio contain speech versus background noise.
It is an essential first step for a variety of speech-based applications including Automatic Speech Recognition.
It serves to determine which samples to be sent to the model and when to close the microphone.
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
datasets
results
configs
Resource and Documentation Guide
--------------------------------
Hands-on speech classification tutorial notebooks can be found under ``<NeMo_git_repo>/tutorials/asr/``.
There are training and offline & online microphone inference tutorials for Speech Command Detection and Voice Activity Detection tasks.
This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular Speech Classification model or would like to find out more about the model
architectures available in the `nemo_asr` collection, check out the :doc:`Models <./models>` page.
Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
NeMo includes preprocessing scripts for several common ASR datasets, and this page contains instructions on running
those scripts.
It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
:doc:`Configuration Files <./configs>` page.
tts/NeMo/docs/source/asr/speech_classification/models.rst
+71
View File
@@ -0,0 +1,71 @@
Models
======
This page gives a brief overview of the models that NeMo's Speech Classification collection currently supports.
For Speech Classification, we support Speech Command (Keyword) Detection and Voice Activity Detection (VAD).
Each of these models can be used with the example ASR scripts (in the ``<NeMo_git_root>/examples/asr`` directory) by
specifying the model architecture in the config file used.
Examples of config files for each model can be found in the ``<NeMo_git_root>/examples/asr/conf`` directory.
For more information about the config files and how they should be structured, see the :doc:`./configs` page.
Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found on the :doc:`./results` page.
You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets.
The Checkpoints page also contains benchmark results for the available ASR models.
.. _MatchboxNet_model:
MatchboxNet (Speech Commands)
------------------------------
MatchboxNet :cite:`sc-models-matchboxnet` is an end-to-end neural network for speech command recognition based on `QuartzNet <../models.html#QuartzNet>`__.
Similarly to QuartzNet, the MatchboxNet family of models are denoted as MatchBoxNet_[BxRxC] where B is the number of blocks, and R is the number of convolutional sub-blocks within a block, and C is the number of channels. Each sub-block contains a 1-D *separable* convolution, batch normalization, ReLU, and dropout:
.. image:: images/matchboxnet_vertical.png
:align: center
:alt: MatchboxNet model
:scale: 50%
It can reach state-of-the art accuracy on the Google Speech Commands dataset while having significantly fewer parameters than similar models.
The `_v1` and `_v2` are denoted for models trained on `v1` (30-way classification) and `v2` (35-way classification) datasets;
And we use _subset_task to represent (10+2)-way subset (10 specific classes + other remaining classes + silence) classification task.
MatchboxNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecClassificationModel` class.
.. note::
For model details and deep understanding about Speech Command Detedction training, inference, finetuning and etc.,
please refer to ``<NeMo_git_root>/tutorials/asr/03_Speech_Commands.ipynb`` and ``<NeMo_git_root>/tutorials/asr/04_Online_Offline_Speech_Commands_Demo.ipynb``.
.. _MarbleNet_model:
MarbleNet (VAD)
------------------
MarbleNet :cite:`sc-models-marblenet` an end-to-end neural network for speech command recognition based on :ref:`MatchboxNet_model`,
Similarly to MatchboxNet, the MarbleNet family of models are denoted as MarbleNet_[BxRxC] where B is the number of blocks, and R is the number of convolutional sub-blocks within a block, and C is the number of channels. Each sub-block contains a 1-D *separable* convolution, batch normalization, ReLU, and dropout:
.. image:: images/marblenet_vertical.png
:align: center
:alt: MarbleNet model
:scale: 30%
It can reach state-of-the art performance on the difficult `AVA speech dataset <https://research.google.com/ava/download.html#ava_speech_download>`_ while having significantly fewer parameters than similar models even training on simple data.
MarbleNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecClassificationModel` class.
.. note::
For model details and deep understanding about VAD training, inference, postprocessing, threshold tuning and etc.,
please refer to ``<NeMo_git_root>/tutorials/asr/06_Voice_Activiy_Detection.ipynb`` and ``<NeMo_git_root>/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb``.
References
----------------
.. bibliography:: ../asr_all.bib
:style: plain
:labelprefix: SC-MODELS
:keyprefix: sc-models-
tts/NeMo/docs/source/asr/speech_classification/results.rst
+105
View File
@@ -0,0 +1,105 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo:
* Using the :code:`restore_from()` method to load a local checkpoint file (`.nemo`), or
* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
See the following sections for instructions and examples for each.
Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning.
For resuming an unfinished training experiment, please use the experiment manager to do so by setting the
``resume_if_exists`` flag to True.
Loading Local Checkpoints
-------------------------
NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
If you have a local ``.nemo`` checkpoint that you'd like to load, simply use the :code:`restore_from()` method:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
Where the model base class is the ASR model class of the original checkpoint, or the general `ASRModel` class.
Transcribing/Inference
-----------------------
The audio files should be 16KHz monochannel wav files.
**Transcribe speech command segment:**
You may perform inference and transcribe a sample of speech after loading the model by using its 'transcribe()' method:
.. code-block:: python
mbn_model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="<MODEL_NAME>")
mbn_model.transcribe([list of audio files], batch_size=BATCH_SIZE, logprobs=False)
Setting argument ``logprobs`` to True would return the log probabilities instead of transcriptions. You may find more details in `Modules <../api.html#modules>`__.
Learn how to fine tune on your own data or on subset classes in ``<NeMo_git_root>/tutorials/asr/03_Speech_Commands.ipynb``
f
**Run VAD inference:**
.. code-block:: bash
python examples/speaker_recognition/vad_infer.py --vad_model="vad_marblenet" --dataset=<FULL PATH OF MANIFEST TO BE PERFORMED INFERENCE ON> --out_dir='frame/demo' --time_length=0.63
Have a look at scripts under ``<NeMo-git-root>/scripts/voice_activity_detection`` for posterior processing and threshold tuning.
NGC Pretrained Checkpoints
--------------------------
The Speech Classification collection has checkpoints of several models trained on various datasets for a variety of tasks.
These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
The model cards on NGC contain more information about each of the checkpoints available.
The tables below list the Speech Classification models available from NGC, and the models can be accessed via the
:code:`from_pretrained()` method inside the ASR Model class.
In general, you can load any of these models with code in the following format.
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="<MODEL_NAME>")
Where the model name is the value under "Model Name" entry in the tables below.
For example, to load the MatchboxNet3x2x64_v1 model for speech command detection, run:
.. code-block:: python
model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="commandrecognition_en_matchboxnet3x2x64_v1")
You can also call :code:`from_pretrained()` from the specific model class (such as :code:`EncDecClassificationModel`
for MatchboxNet and MarbleNet) if you will need to access specific model functionality.
If you would like to programatically list the models available for a particular base class, you can use the
:code:`list_available_models()` method.
.. code-block:: python
nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
Speech Classification Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. tabularcolumns:: 30 30 40
.. csv-table::
:file: data/classification_results.csv
:header-rows: 1
:class: longtable
:widths: 1 1 1
tts/NeMo/docs/source/common/intro.rst
+39
View File
@@ -0,0 +1,39 @@
Common Collection
=================
The common collection contains things that could be used across all collections.
Tokenizers
----------
.. automodule:: nemo.collections.common.tokenizers.AutoTokenizer
:special-members: __init__
.. automodule:: nemo.collections.common.tokenizers.SentencePieceTokenizer
:special-members: __init__
.. automodule:: nemo.collections.common.tokenizers.TokenizerSpec
:special-members: __init__
Losses
------
.. automodule:: nemo.collections.common.losses.AggregatorLoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.CrossEntropyLoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.MSELoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.SmoothedCrossEntropyLoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.SpanningLoss
:special-members: __init__
Metrics
-------
.. autoclass:: nemo.collections.common.metrics.Perplexity
:show-inheritance:
:members:
:undoc-members:
tts/NeMo/docs/source/conf.py
+242
View File
@@ -0,0 +1,242 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import re
import sys
import glob
import sphinx_rtd_theme
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath("../.."))
sys.path.insert(0, os.path.abspath("../../nemo"))
sys.path.insert(0, os.path.abspath("../../nemo_text_processing"))
from package_info import __version__
autodoc_mock_imports = [
'torch',
'torch.nn',
'torch.utils',
'torch.optim',
'torch.utils.data',
'torch.utils.data.sampler',
'torchvision',
'torchvision.models',
'torchtext',
'ruamel.yaml', # ruamel.yaml has ., which is troublesome for this regex
'hydra', # hydra-core in requirements, hydra during import
'dateutil', # part of core python
'transformers.tokenization_bert', # has ., troublesome for this regex
'megatron', # megatron-lm in requirements, megatron in import
'sklearn',
'nemo_text_processing.inverse_text_normalization', # Not installed automatically
'nemo_text_processing.text_normalization', # Not installed automatically
'attr', # attrdict in requirements, attr in import
'torchmetrics', # inherited from PTL
]
_skipped_autodoc_mock_imports = ['wrapt', 'numpy']
for req_path in sorted(list(glob.glob("../../requirements/*.txt"))):
if "docs.txt" in req_path:
continue
req_file = os.path.abspath(os.path.expanduser(req_path))
with open(req_file, 'r') as f:
for line in f:
line = line.replace("\n", "")
req = re.search(r"([a-zA-Z0-9-_]*)", line)
if req:
req = req.group(1)
req = req.replace("-", "_")
if req not in autodoc_mock_imports:
if req in _skipped_autodoc_mock_imports:
print(f"Skipping req : `{req}` (lib {line})")
continue
autodoc_mock_imports.append(req)
print(f"Adding req : `{req}` to autodoc mock requirements (lib {line})")
else:
print(f"`{req}` already added to autodoc mock requirements (lib {line})")
#
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.todo",
"sphinx.ext.coverage",
"sphinx.ext.mathjax",
"sphinx.ext.ifconfig",
"sphinx.ext.viewcode",
"sphinx.ext.napoleon",
"sphinx.ext.githubpages",
"sphinxcontrib.bibtex",
"sphinx.ext.inheritance_diagram",
"sphinx.ext.intersphinx",
"sphinx.ext.autosectionlabel",
"sphinxcontrib.bibtex",
"sphinx_rtd_theme",
]
bibtex_bibfiles = [
'asr/asr_all.bib',
'nlp/nlp_all.bib',
'tools/tools_all.bib',
'nemo_text_processing/textprocessing_all.bib',
'tts_all.bib',
]
intersphinx_mapping = {
'pytorch': ('https://pytorch.org/docs/stable', None),
'pytorch-lightning': ('https://pytorch-lightning.readthedocs.io/en/latest/', None),
}
# Set default flags for all classes.
autodoc_default_options = {'members': None, 'undoc-members': None, 'show-inheritance': True}
locale_dirs = ['locale/'] # path is example but recommended.
gettext_compact = False # optional.
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = ".rst"
# The master toctree document.
master_doc = "index"
# General information about the project.
project = "NVIDIA NeMo"
copyright = "2021-, NVIDIA CORPORATION"
author = "NVIDIA CORPORATION"
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
# The short X.Y version.
# version = "0.10.0"
version = __version__
# The full version, including alpha/beta/rc tags.
# release = "0.9.0"
release = __version__
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "default"
### Previous NeMo theme
# # NVIDIA theme settings.
# html_theme = 'nvidia_theme'
# html_theme_path = ["."]
# html_theme_options = {
# 'display_version': True,
# 'project_version': version,
# 'project_name': project,
# 'logo_path': None,
# 'logo_only': True,
# }
# html_title = 'Introduction'
# html_logo = html_theme_options["logo_path"]
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = "nemodoc"
### from TLT conf.py
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
# html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
html_theme = "sphinx_rtd_theme"
html_logo = os.path.join('nv_logo.png')
html_theme_options = {
'logo_only': True,
'display_version': True,
'prev_next_buttons_location': 'bottom',
'style_external_links': False,
'style_nav_header_background': '#000000',
# Toc options
'collapse_navigation': False,
'sticky_navigation': False,
# 'navigation_depth': 10,
'includehidden': False,
'titles_only': False,
}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_favicon = 'favicon.ico'
html_static_path = ['_static']
html_last_updated_fmt = ''
def setup(app):
app.add_css_file('css/custom.css')
app.add_js_file('js/pk_scripts.js')
# html_css_files = [
# './custom.css',
# ]
# html_js_files = [
# './pk_scripts.js',
# ]
tts/NeMo/docs/source/core/core.rst
+963
View File
@@ -0,0 +1,963 @@
NeMo Models
===========
Basics
------
NeMo models contain everything needed to train and reproduce Conversational AI models:
- neural network architectures
- datasets/data loaders
- data preprocessing/postprocessing
- data augmentors
- optimizers and schedulers
- tokenizers
- language models
NeMo uses `Hydra <https://hydra.cc/>`_ for configuring both NeMo models and the PyTorch Lightning Trainer.
.. note:: Every NeMo model has an example configuration file and training script that can be found `here <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples>`_.
The end result of using NeMo, `Pytorch Lightning <https://github.com/PyTorchLightning/pytorch-lightning>`_, and Hydra is that NeMo models all have the same look and feel and are also fully compatible with the PyTorch ecosystem.
Pretrained
----------
NeMo comes with many pretrained models for each of our collections: ASR, NLP, and TTS. Every pretrained NeMo model can be downloaded
and used with the ``from_pretrained()`` method.
As an example, we can instantiate QuartzNet with the following:
.. code-block:: Python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
To see all available pretrained models for a specific NeMo model, use the ``list_available_models()`` method.
.. code-block:: Python
nemo_asr.model.EncDecCTCModel.list_available_models()
For detailed information on the available pretrained models, refer to the collections documentation:
- :ref:`Automatic Speech Recognition (ASR)`
- :ref:`Natural Language Processing (NLP)`
- :ref:`Speech Synthesis (TTS)`
Training
--------
NeMo leverages `PyTorch Lightning <https://www.pytorchlightning.ai/>`_ for model training. PyTorch Lightning lets NeMo decouple the
conversational AI code from the PyTorch training code. This means that NeMo users can focus on their domain (ASR, NLP, TTS) and
build complex AI applications without having to rewrite boiler plate code for PyTorch training.
When using PyTorch Lightning, NeMo users can automatically train with:
- multi-GPU/multi-node
- mixed precision
- model checkpointing
- logging
- early stopping
- and more
The two main aspects of the Lightning API are the `LightningModule <https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#>`_
and the `Trainer <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html>`_.
PyTorch Lightning ``LightningModule``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Every NeMo model is a ``LightningModule`` which is an ``nn.module``. This means that NeMo models are compatible with the PyTorch
ecosystem and can be plugged into existing PyTorch workflows.
Creating a NeMo model is similar to any other PyTorch workflow. We start by initializing our model architecture, then define the forward pass:
.. code-block:: python
class TextClassificationModel(NLPModel, Exportable):
...
def __init__(self, cfg: DictConfig, trainer: Trainer = None):
"""Initializes the BERTTextClassifier model."""
...
super().__init__(cfg=cfg, trainer=trainer)
# instantiate a BERT based encoder
self.bert_model = get_lm_model(
pretrained_model_name=cfg.language_model.pretrained_model_name,
config_file=cfg.language_model.config_file,
config_dict=cfg.language_model.config,
checkpoint_file=cfg.language_model.lm_checkpoint,
vocab_file=cfg.tokenizer.vocab_file,
)
# instantiate the FFN for classification
self.classifier = SequenceClassifier(
hidden_size=self.bert_model.config.hidden_size,
num_classes=cfg.dataset.num_classes,
num_layers=cfg.classifier_head.num_output_layers,
activation='relu',
log_softmax=False,
dropout=cfg.classifier_head.fc_dropout,
use_transformer_init=True,
idx_conditioned_on=0,
)
.. code-block:: python
def forward(self, input_ids, token_type_ids, attention_mask):
"""
No special modification required for Lightning, define it as you normally would
in the `nn.Module` in vanilla PyTorch.
"""
hidden_states = self.bert_model(
input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask
)
logits = self.classifier(hidden_states=hidden_states)
return logits
The ``LightningModule`` organizes PyTorch code so that across all NeMo models we have a similar look and feel.
For example, the training logic can be found in ``training_step``:
.. code-block:: python
def training_step(self, batch, batch_idx):
"""
Lightning calls this inside the training loop with the data from the training dataloader
passed in as `batch`.
"""
# forward pass
input_ids, input_type_ids, input_mask, labels = batch
logits = self.forward(input_ids=input_ids, token_type_ids=input_type_ids, attention_mask=input_mask)
train_loss = self.loss(logits=logits, labels=labels)
lr = self._optimizer.param_groups[0]['lr']
self.log('train_loss', train_loss)
self.log('lr', lr, prog_bar=True)
return {
'loss': train_loss,
'lr': lr,
}
While validation logic can be found in ``validation_step``:
.. code-block:: python
def validation_step(self, batch, batch_idx):
"""
Lightning calls this inside the validation loop with the data from the validation dataloader
passed in as `batch`.
"""
if self.testing:
prefix = 'test'
else:
prefix = 'val'
input_ids, input_type_ids, input_mask, labels = batch
logits = self.forward(input_ids=input_ids, token_type_ids=input_type_ids, attention_mask=input_mask)
val_loss = self.loss(logits=logits, labels=labels)
preds = torch.argmax(logits, axis=-1)
tp, fn, fp, _ = self.classification_report(preds, labels)
return {'val_loss': val_loss, 'tp': tp, 'fn': fn, 'fp': fp}
PyTorch Lightning then handles all of the boiler plate code needed for training. Virtually any aspect of training can be customized
via PyTorch Lightning `hooks <https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#hooks>`_,
`Plugins <https://pytorch-lightning.readthedocs.io/en/stable/extensions/plugins.html>`_,
`callbacks <https://pytorch-lightning.readthedocs.io/en/stable/extensions/callbacks.html>`_, or by overriding `methods <https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#methods>`_.
For more domain-specific information, see:
- :ref:`Automatic Speech Recognition (ASR)`
- :ref:`Natural Language Processing (NLP)`
- :ref:`Speech Synthesis (TTS)`
PyTorch Lightning Trainer
~~~~~~~~~~~~~~~~~~~~~~~~~
Since every NeMo model is a ``LightningModule``, we can automatically take advantage of the PyTorch Lightning ``Trainer``. Every NeMo
`example <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples>`_ training script uses the ``Trainer`` object to fit the model.
First, instantiate the model and trainer, then call ``.fit``:
.. code-block:: python
# We first instantiate the trainer based on the model configuration.
# See the model configuration documentation for details.
trainer = pl.Trainer(**cfg.trainer)
# Then pass the model configuration and trainer object into the NeMo model
model = TextClassificationModel(cfg.model, trainer=trainer)
# Now we can train with by calling .fit
trainer.fit(model)
# Or we can run the test loop on test data by calling
trainer.test(model=model)
All `trainer flags <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-flags>`_ can be set from from the
NeMo configuration.
Configuration
-------------
Hydra is an open-source Python framework that simplifies configuration for complex applications that must bring together many different
software libraries. Conversational AI model training is a great example of such an application. To train a conversational AI model, we
must be able to configure:
- neural network architectures
- training and optimization algorithms
- data pre/post processing
- data augmentation
- experiment logging/visualization
- model checkpointing
For an introduction to using Hydra, refer to the `Hydra Tutorials <https://hydra.cc/docs/tutorials/intro>`_.
With Hydra, we can configure everything needed for NeMo with three interfaces:
- Command Line (CLI)
- Configuration Files (YAML)
- Dataclasses (Python)
YAML
~~~~
NeMo provides YAML configuration files for all of our `example <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples>`_ training scripts.
YAML files make it easy to experiment with different model and training configurations.
Every NeMo example YAML has the same underlying configuration structure:
- trainer
- exp_manager
- model
Model configuration always contain ``train_ds``, ``validation_ds``, ``test_ds``, and ``optim``. Model architectures vary across
domains, therefore, refer to the ASR, NLP, and TTS Collections documentation for more detailed information on Model architecture configuration.
A NeMo configuration file should look similar to the following:
.. code-block:: yaml
# PyTorch Lightning Trainer configuration
# any argument of the Trainer object can be set here
trainer:
gpus: 1 # number of gpus per node
num_nodes: 1 # number of nodes
max_epochs: 10 # how many training epochs to run
val_check_interval: 1.0 # run validation after every epoch
# Experiment logging configuration
exp_manager:
exp_dir: /path/to/my/nemo/experiments
name: name_of_my_experiment
create_tensorboard_logger: True
create_wandb_logger: True
# Model configuration
# model network architecture, train/val/test datasets, data augmentation, and optimization
model:
train_ds:
manifest_filepath: /path/to/my/train/manifest.json
batch_size: 256
shuffle: True
validation_ds:
manifest_filepath: /path/to/my/validation/manifest.json
batch_size: 32
shuffle: False
test_ds:
manifest_filepath: /path/to/my/test/manifest.json
batch_size: 32
shuffle: False
optim:
name: novograd
lr: .01
betas: [0.8, 0.5]
weight_decay: 0.001
# network architecture can vary greatly depending on the domain
encoder:
...
decoder:
...
More specific details about configuration files for each collection can be found on the following pages:
:ref:`NeMo ASR Configuration Files`
CLI
~~~
With NeMo and Hydra, every aspect of model training can be modified from the command-line. This is extremely helpful for running lots
of experiments on compute clusters or for quickly testing parameters while developing.
All NeMo `examples <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples>`_ come with instructions on how to
run the training/inference script from the command-line (see `here <https://github.com/NVIDIA/NeMo/blob/4e9da75f021fe23c9f49404cd2e7da4597cb5879/examples/asr/speech_to_text.py#L24>`_
for an example).
With Hydra, arguments are set using the ``=`` operator:
.. code-block:: bash
python examples/asr/speech_to_text.py \
model.train_ds.manifest_filepath=/path/to/my/train/manifest.json \
model.validation_ds.manifest_filepath=/path/to/my/validation/manifest.json \
trainer.gpus=2 \
trainer.max_epochs=50
We can use the ``+`` operator to add arguments from the CLI:
.. code-block:: bash
python examples/asr/speech_to_text.py \
model.train_ds.manifest_filepath=/path/to/my/train/manifest.json \
model.validation_ds.manifest_filepath=/path/to/my/validation/manifest.json \
trainer.gpus=2 \
trainer.max_epochs=50 \
+trainer.fast_dev_run=true
We can use the ``~`` operator to remove configurations:
.. code-block:: bash
python examples/asr/speech_to_text.py \
model.train_ds.manifest_filepath=/path/to/my/train/manifest.json \
model.validation_ds.manifest_filepath=/path/to/my/validation/manifest.json \
~model.test_ds \
trainer.gpus=2 \
trainer.max_epochs=50 \
+trainer.fast_dev_run=true
We can specify configuration files using the ``--config-path`` and ``--config-name`` flags:
.. code-block:: bash
python examples/asr/speech_to_text.py \
--config-path=conf/quartznet \
--config-name=quartznet_15x5 \
model.train_ds.manifest_filepath=/path/to/my/train/manifest.json \
model.validation_ds.manifest_filepath=/path/to/my/validation/manifest.json \
~model.test_ds \
trainer.gpus=2 \
trainer.max_epochs=50 \
+trainer.fast_dev_run=true
Dataclasses
~~~~~~~~~~~
Dataclasses allow NeMo to ship model configurations as part of the NeMo library and also enables pure Python configuration of NeMo models.
With Hydra, dataclasses can be used to create `structured configs <https://hydra.cc/docs/tutorials/structured_config/intro>`_ for the conversational AI application.
As an example, refer to the code block below for an *Attenion is All You Need* machine translation model. The model configuration can
be instantiated and modified like any Python `Dataclass <https://docs.python.org/3/library/dataclasses.html>`_.
.. code-block:: Python
from nemo.collections.nlp.models.machine_translation.mt_enc_dec_config import AAYNBaseConfig
cfg = AAYNBaseConfig()
# modify the number of layers in the encoder
cfg.encoder.num_layers = 8
# modify the training batch size
cfg.train_ds.tokens_in_batch = 8192
.. note:: Configuration with Hydra always has the following precedence CLI > YAML > Dataclass
Optimization
------------
Optimizers and learning rate schedules are configurable across all NeMo models and have their own namespace. Here is a sample YAML
configuration for a Novograd optimizer with Cosine Annealing learning rate schedule.
.. code-block:: yaml
optim:
name: novograd
lr: 0.01
# optimizer arguments
betas: [0.8, 0.25]
weight_decay: 0.001
# scheduler setup
sched:
name: CosineAnnealing
# Optional arguments
max_steps: null # computed at runtime or explicitly set here
monitor: val_loss
reduce_on_plateau: false
# scheduler config override
warmup_steps: 1000
warmup_ratio: null
min_lr: 1e-9:
.. note:: `NeMo Examples <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples>`_ has optimizer and scheduler configurations for
every NeMo model.
Optimizers can be configured from the CLI as well:
.. code-block:: bash
python examples/asr/speech_to_text.py \
--config-path=conf/quartznet \
--config-name=quartznet_15x5 \
...
# train with the adam optimizer
model.optim=adam \
# change the learning rate
model.optim.lr=.0004 \
# modify betas
model.optim.betas=[.8, .5]
Optimizers
~~~~~~~~~~
``name`` corresponds to the lowercase name of the optimizer. To view a list of available optimizers, run:
.. code-block:: Python
from nemo.core.optim.optimizers import AVAILABLE_OPTIMIZERS
for name, opt in AVAILABLE_OPTIMIZERS.items():
print(f'name: {name}, opt: {opt}')
.. code-block:: bash
name: sgd opt: <class 'torch.optim.sgd.SGD'>
name: adam opt: <class 'torch.optim.adam.Adam'>
name: adamw opt: <class 'torch.optim.adamw.AdamW'>
name: adadelta opt: <class 'torch.optim.adadelta.Adadelta'>
name: adamax opt: <class 'torch.optim.adamax.Adamax'>
name: adagrad opt: <class 'torch.optim.adagrad.Adagrad'>
name: rmsprop opt: <class 'torch.optim.rmsprop.RMSprop'>
name: rprop opt: <class 'torch.optim.rprop.Rprop'>
name: novograd opt: <class 'nemo.core.optim.novograd.Novograd'>
Optimizer Params
~~~~~~~~~~~~~~~~
Optimizer params can vary between optimizers but the ``lr`` param is required for all optimizers. To see the available params for an
optimizer, we can look at its corresponding dataclass.
.. code-block:: python
from nemo.core.config.optimizers import NovogradParams
print(NovogradParams())
.. code-block:: bash
NovogradParams(lr='???', betas=(0.95, 0.98), eps=1e-08, weight_decay=0, grad_averaging=False, amsgrad=False, luc=False, luc_trust=0.001, luc_eps=1e-08)
``'???'`` indicates that the lr argument is required.
Register Optimizer
~~~~~~~~~~~~~~~~~~
To register a new optimizer to be used with NeMo, run:
.. autofunction:: nemo.core.optim.optimizers.register_optimizer
Learning Rate Schedulers
~~~~~~~~~~~~~~~~~~~~~~~~
Learning rate schedulers can be optionally configured under the ``optim.sched`` namespace.
``name`` corresponds to the name of the learning rate schedule. To view a list of available schedulers, run:
.. code-block:: Python
from nemo.core.optim.lr_scheduler import AVAILABLE_SCHEDULERS
for name, opt in AVAILABLE_SCHEDULERS.items():
print(f'name: {name}, schedule: {opt}')
.. code-block:: bash
name: WarmupPolicy, schedule: <class 'nemo.core.optim.lr_scheduler.WarmupPolicy'>
name: WarmupHoldPolicy, schedule: <class 'nemo.core.optim.lr_scheduler.WarmupHoldPolicy'>
name: SquareAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.SquareAnnealing'>
name: CosineAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.CosineAnnealing'>
name: NoamAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.NoamAnnealing'>
name: WarmupAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.WarmupAnnealing'>
name: InverseSquareRootAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.InverseSquareRootAnnealing'>
name: SquareRootAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.SquareRootAnnealing'>
name: PolynomialDecayAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.PolynomialDecayAnnealing'>
name: PolynomialHoldDecayAnnealing, schedule: <class 'nemo.core.optim.lr_scheduler.PolynomialHoldDecayAnnealing'>
name: StepLR, schedule: <class 'torch.optim.lr_scheduler.StepLR'>
name: ExponentialLR, schedule: <class 'torch.optim.lr_scheduler.ExponentialLR'>
name: ReduceLROnPlateau, schedule: <class 'torch.optim.lr_scheduler.ReduceLROnPlateau'>
name: CyclicLR, schedule: <class 'torch.optim.lr_scheduler.CyclicLR'>
Scheduler Params
~~~~~~~~~~~~~~~~
To see the available params for a scheduler, we can look at its corresponding dataclass:
.. code-block:: Python
from nemo.core.config.schedulers import CosineAnnealingParams
print(CosineAnnealingParams())
.. code-block:: bash
CosineAnnealingParams(last_epoch=-1, warmup_steps=None, warmup_ratio=None, min_lr=0.0)
Register scheduler
~~~~~~~~~~~~~~~~~~
To register a new scheduler to be used with NeMo, run:
.. autofunction:: nemo.core.optim.lr_scheduler.register_scheduler
Save and Restore
----------------
NeMo models all come with ``.save_to`` and ``.restore_from`` methods.
Save
~~~~
To save a NeMo model, run:
.. code-block:: Python
model.save_to('/path/to/model.nemo')
Everything needed to use the trained model is packaged and saved in the ``.nemo`` file. For example, in the NLP domain, ``.nemo`` files
include the necessary tokenizer models and/or vocabulary files, etc.
.. note:: A ``.nemo`` file is simply an archive like any other ``.tar`` file.
Restore
~~~~~~~
To restore a NeMo model, run:
.. code-block:: Python
model.restore_from('/path/to/model.nemo')
When using the PyTorch Lightning Trainer, a PyTorch Lightning checkpoint is created. These are mainly used within NeMo to auto-resume
training. Since NeMo models are ``LightningModules``, the PyTorch Lightning method ``load_from_checkpoint`` is available. Note that
``load_from_checkpoint`` won't necessarily work out-of-the-box for all models as some models require more artifacts than just the
checkpoint to be restored. For these models, the user will have to override ``load_from_checkpoint`` if they want to use it.
It's highly recommended to use ``restore_from`` to load NeMo models.
Register Artifacts
------------------
Conversational AI models can be complicated to restore as more information is needed than just the checkpoint weights in order to use the model.
NeMo models can save additional artifacts in the .nemo file by calling ``.register_artifact``.
When restoring NeMo models using ``.restore_from`` or ``.from_pretrained``, any artifacts that were registered will be available automatically.
As an example, consider an NLP model that requires a trained tokenizer model.
The tokenizer model file can be automatically added to the .nemo file with the following:
.. code-block:: python
self.encoder_tokenizer = get_nmt_tokenizer(
...
tokenizer_model=self.register_artifact(config_path='encoder_tokenizer.tokenizer_model',
src='/path/to/tokenizer.model',
verify_src_exists=True),
)
By default, ``.register_artifact`` will always return a path. If the model is being restored from a .nemo file,
then that path will be to the artifact in the .nemo file. Otherwise, ``.register_artifact`` will return the local path specified by the user.
``config_path`` is the artifact key. It usually corresponds to a model configuration but does not have to.
The model config that is packaged with the .nemo file will be updated according to the ``config_path`` key.
In the above example, the model config will have
.. code-block:: YAML
encoder_tokenizer:
...
tokenizer_model: nemo:4978b28103264263a03439aaa6560e5e_tokenizer.model
``src`` is the path to the artifact and the base-name of the path will be used when packaging the artifact in the .nemo file.
Each artifact will have a hash prepended to the basename of ``src`` in the .nemo file. This is to prevent collisions with basenames
base-names that are identical (say when there are two or more tokenizers, both called `tokenizer.model`).
The resulting .nemo file will then have the following file:
.. code-block:: bash
4978b28103264263a03439aaa6560e5e_tokenizer.model
If ``verify_src_exists`` is set to ``False``, then the artifact is optional. This means that ``.register_artifact`` will return ``None``
if the ``src`` cannot be found.
Experiment Manager
==================
NeMo's Experiment Manager leverages PyTorch Lightning for model checkpointing, TensorBoard Logging, and Weights and Biases logging. The
Experiment Manager is included by default in all NeMo example scripts.
To use the experiment manager simply call :class:`~nemo.utils.exp_manager.exp_manager` and pass in the PyTorch Lightning ``Trainer``.
.. code-block:: python
exp_manager(trainer, cfg.get("exp_manager", None))
And is configurable via YAML with Hydra.
.. code-block:: bash
exp_manager:
exp_dir: /path/to/my/experiments
name: my_experiment_name
create_tensorboard_logger: True
create_checkpoint_callback: True
Optionally, launch TensorBoard to view the training results in ``./nemo_experiments`` (by default).
.. code-block:: bash
tensorboard --bind_all --logdir nemo_experiments
..
If ``create_checkpoint_callback`` is set to ``True``, then NeMo automatically creates checkpoints during training
using PyTorch Lightning's `ModelCheckpoint <https://pytorch-lightning.readthedocs.io/en/stable/extensions/generated/pytorch_lightning.callbacks.ModelCheckpoint.html#pytorch_lightning.callbacks.ModelCheckpoint>`_.
We can configure the ``ModelCheckpoint`` via YAML or CLI.
.. code-block:: yaml
exp_manager:
...
# configure the PyTorch Lightning ModelCheckpoint using checkpoint_call_back_params
# any ModelCheckpoint argument can be set here
# save the best checkpoints based on this metric
checkpoint_callback_params.monitor=val_loss
# choose how many total checkpoints to save
checkpoint_callback_params.save_top_k=5
We can auto-resume training as well by configuring the ``exp_manager``. Being able to auto-resume is important when doing long training
runs that are premptible or may be shut down before the training procedure has completed. To auto-resume training, set the following
via YAML or CLI:
.. code-block:: yaml
exp_manager:
...
# resume training if checkpoints already exist
resume_if_exists: True
# to start training with no existing checkpoints
resume_ignore_no_checkpoint: True
# by default experiments will be versioned by datetime
# we can set our own version with
exp_manager.version: my_experiment_version
Neural Modules
==============
NeMo is built around Neural Modules, conceptual blocks of neural networks that take typed inputs and produce typed outputs. Such
modules typically represent data layers, encoders, decoders, language models, loss functions, or methods of combining activations.
NeMo makes it easy to combine and re-use these building blocks while providing a level of semantic correctness checking via its neural
type system.
.. note:: *All Neural Modules inherit from ``torch.nn.Module`` and are therefore compatible with the PyTorch ecosystem.*
There are 3 types on Neural Modules:
- Regular modules
- Dataset/IterableDataset
- Losses
Every Neural Module in NeMo must inherit from `nemo.core.classes.module.NeuralModule` class.
.. autoclass:: nemo.core.classes.module.NeuralModule
Every Neural Modules inherits the ``nemo.core.classes.common.Typing`` interface and needs to define neural types for its inputs and outputs.
This is done by defining two properties: ``input_types`` and ``output_types``. Each property should return an ordered dictionary of
"port name"->"port neural type" pairs. Here is the example from :class:`~nemo.collections.asr.modules.ConvASREncoder` class:
.. code-block:: python
@property
def input_types(self):
return OrderedDict(
{
"audio_signal": NeuralType(('B', 'D', 'T'), SpectrogramType()),
"length": NeuralType(tuple('B'), LengthsType()),
}
)
@property
def output_types(self):
return OrderedDict(
{
"outputs": NeuralType(('B', 'D', 'T'), AcousticEncodedRepresentation()),
"encoded_lengths": NeuralType(tuple('B'), LengthsType()),
}
)
@typecheck()
def forward(self, audio_signal, length=None):
...
The code snippet above means that ``nemo.collections.asr.modules.conv_asr.ConvASREncoder`` expects two arguments:
* First one, named ``audio_signal`` of shape ``[batch, dimension, time]`` with elements representing spectrogram values.
* Second one, named ``length`` of shape ``[batch]`` with elements representing lengths of corresponding signals.
It also means that ``.forward(...)`` and ``__call__(...)`` methods each produce two outputs:
* First one, of shape ``[batch, dimension, time]`` but with elements representing encoded representation (``AcousticEncodedRepresentation`` class).
* Second one, of shape ``[batch]``, corresponding to their lengths.
.. tip:: It is a good practice to define types and add ``@typecheck()`` decorator to your ``.forward()`` method after your module is ready for use by others.
.. note:: The outputs of ``.forward(...)`` method will always be of type ``torch.Tensor`` or container of tensors and will work with any other Pytorch code. The type information is attached to every output tensor.
If tensors without types is passed to your module, it will not fail, however the types will not be checked. Thus, it is recommended to define input/output types for all your modules, starting with
data layers and add ``@typecheck()`` decorator to them.
.. note:: To temporarily disable typechecking, you can enclose your code in ```with typecheck.disable_checks():``` statement.
Neural Types
============
Motivation
----------
Neural Types describe the semantics, axis order, and dimensions of a tensor. The purpose of this type system is to catch semantic and
dimensionality errors during model creation and facilitate module re-use.
.. image:: whyntypes.gif
:width: 900
:alt: Neural Types Motivation
``NeuralType`` class
--------------------
Neural Types perform semantic checks for modules and models inputs/outputs. They contain information about:
- Semantics of what is stored in the tensors. For example, logits, logprobs, audiosignal, embeddings, etc.
- Axes layout, semantic and (optionally) dimensionality. For example: ``[Batch, Time, Channel]``
Types are implemented in ``nemo.core.neural_types.NeuralType`` class. When you instantiate an instance of this class, you
are expected to include both *axes* information and *element type* information.
.. autoclass:: nemo.core.neural_types.NeuralType
Type Comparison Results
-----------------------
When comparing two neural types, the following comparison results are generated.
.. autoclass:: nemo.core.neural_types.NeuralTypeComparisonResult
Examples
--------
Long vs short notation
~~~~~~~~~~~~~~~~~~~~~~
NeMo's ``NeuralType`` class allows you to express axis semantics information in long and short form. Consider these two equivalent types. Both encoder 3 dimensional tensors and both contain elements of type ``AcousticEncodedRepresentation`` (this type is a typical output of ASR encoders).
.. code-block:: python
long_version = NeuralType(
axes=(AxisType(AxisKind.Batch, None), AxisType(AxisKind.Dimension, None), AxisType(AxisKind.Time, None)),
elements_type=AcousticEncodedRepresentation(),
)
short_version = NeuralType(('B', 'D', 'T'), AcousticEncodedRepresentation())
assert long_version.compare(short_version) == NeuralTypeComparisonResult.SAME
Transpose same
~~~~~~~~~~~~~~
Often it is useful to know if a simple transposition will solve type incompatibility. This is the case if the comparison result of two types equals ``nemo.core.neural_types.NeuralTypeComparisonResult.TRANSPOSE_SAME``.
.. code-block:: python
type1 = NeuralType(axes=('B', 'T', 'C'))
type2 = NeuralType(axes=('T', 'B', 'C'))
assert type1.compare(type2) == NeuralTypeComparisonResult.TRANSPOSE_SAME
assert type2.compare(type1) == NeuralTypeComparisonResult.TRANSPOSE_SAME
Note that in this example, we dropped ``elements_type`` argument of ``NeuralType`` constructor. If not supplied, the element type is ``VoidType``.
``VoidType`` for elements
~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes it is useful to express that elements' types don't matter but axes layout do. ``VoidType`` for elements can be used to express this.
.. note:: ``VoidType`` is compatible with every other elements' type but not the other way around. See the following code snippet below for details.
.. code-block:: python
btc_spctr = NeuralType(('B', 'T', 'C'), SpectrogramType())
btc_spct_bad = NeuralType(('B', 'T'), SpectrogramType())
# Note the VoidType for elements here
btc_void = NeuralType(('B', 'T', 'C'), VoidType())
# This is true because VoidType is compatible with every other element type (SpectrogramType in this case)
# And axes layout between btc_void and btc_spctr is the same
assert btc_void.compare(btc_spctr) == NeuralTypeComparisonResult.SAME
# These two types are incompatible because even though VoidType is used for elements on one side,
# the axes layout is different
assert btc_void.compare(btc_spct_bad) == NeuralTypeComparisonResult.INCOMPATIBLE
# Note that even though VoidType is compatible with every other type, other types are not compatible with VoidType!
# It is one-way compatibility
assert btc_spctr.compare(btc_void) == NeuralTypeComparisonResult.INCOMPATIBLE
Element type inheritance
~~~~~~~~~~~~~~~~~~~~~~~~
Neural types in NeMo support Python inheritance between element types. Consider an example where you want to develop a Neural Module which performs data augmentation for all kinds of spectrograms.
In ASR, two types of spectrograms are frequently used: mel and mfcc. To express this, we will create 3 classes to express
element's types: ``SpectrogramType``, ``MelSpectrogramType(SpectrogramType)``, ``MFCCSpectrogramType(SpectrogramType)``.
.. code-block:: python
input = NeuralType(('B', 'D', 'T'), SpectrogramType())
out1 = NeuralType(('B', 'D', 'T'), MelSpectrogramType())
out2 = NeuralType(('B', 'D', 'T'), MFCCSpectrogramType())
# MelSpectrogram and MFCCSpectrogram are not interchangeable.
assert out1.compare(out2) == NeuralTypeComparisonResult.INCOMPATIBLE
assert out2.compare(out1) == NeuralTypeComparisonResult.INCOMPATIBLE
# Type comparison detects that MFCC/MelSpectrogramType is a kind of SpectrogramType and can be accepted.
assert input.compare(out1) == NeuralTypeComparisonResult.GREATER
assert input.compare(out2) == NeuralTypeComparisonResult.GREATER
Custom element types
~~~~~~~~~~~~~~~~~~~~
It is possible to create user-defined element types to express the semantics of elements in your tensors. To do so, the user will need to inherit and implement abstract methods of the ``nemo.core.neural_types.elements.ElementType`` class
.. autoclass:: nemo.core.neural_types.elements.ElementType
Note that element types can be parametrized. Consider this example where it distinguishes between audio sampled at 8Khz and 16Khz.
.. code-block:: python
audio16K = NeuralType(axes=('B', 'T'), elements_type=AudioSignal(16000))
audio8K = NeuralType(axes=('B', 'T'), elements_type=AudioSignal(8000))
assert audio8K.compare(audio16K) == NeuralTypeComparisonResult.SAME_TYPE_INCOMPATIBLE_PARAMS
assert audio16K.compare(audio8K) == NeuralTypeComparisonResult.SAME_TYPE_INCOMPATIBLE_PARAMS
Enforcing dimensions
~~~~~~~~~~~~~~~~~~~~
In addition to specifying tensor layout and elements' semantics, neural types also allow you to enforce tensor dimensions.
The user will have to use long notations to specify dimensions. Short notations only allows you to specify axes semantics and assumes
arbitrary dimensions.
.. code-block:: python
type1 = NeuralType(
(AxisType(AxisKind.Batch, 64), AxisType(AxisKind.Time, 10), AxisType(AxisKind.Dimension, 128)),
SpectrogramType(),
)
type2 = NeuralType(('B', 'T', 'C'), SpectrogramType())
# type2 will accept elements of type1 because their axes semantics match and type2 does not care about dimensions
assert type2.compare(type1), NeuralTypeComparisonResult.SAME
# type1 will not accept elements of type2 because it need dimensions to match strictly.
assert type1.compare(type2), NeuralTypeComparisonResult.DIM_INCOMPATIBLE
Generic Axis kind
~~~~~~~~~~~~~~~~~
Sometimes (especially in the case of loss modules) it is useful to be able to specify a "generic" axis kind which will make it
compatible with any other kind of axis. This is easy to express with Neural Types by using ``nemo.core.neural_types.axes.AxisKind.Any`` for axes.
.. code-block:: python
type1 = NeuralType(('B', 'Any', 'Any'), SpectrogramType())
type2 = NeuralType(('B', 'T', 'C'), SpectrogramType())
type3 = NeuralType(('B', 'C', 'T'), SpectrogramType())
# type1 will accept elements of type2 and type3 because it only cares about element kind (SpectrogramType)
# number of axes (3) and that first one corresponds to batch
assert type1.compare(type2) == NeuralTypeComparisonResult.SAME
assert type1.compare(type3) == NeuralTypeComparisonResult.INCOMPATIBLE
Container types
~~~~~~~~~~~~~~~
The NeMo-type system understands Python containers (lists). If your module returns a nested list of typed tensors, the way to express it is by
using Python list notation and Neural Types together when defining your input/output types.
The example below shows how to express that your module returns single output ("out") which is list of lists of two dimensional tensors of shape ``[batch, dimension]`` containing logits.
.. code-block:: python
@property
def output_types(self):
return {
"out": [[NeuralType(('B', 'D'), LogitsType())]],
}
Core APIs
=========
Base class for all NeMo models
------------------------------
.. autoclass:: nemo.core.ModelPT
:show-inheritance:
:members:
:member-order: bysource
:undoc-members: cfg, num_weights
:exclude-members: set_eff_save, use_eff_save, teardown
Base Neural Module class
------------------------
.. autoclass:: nemo.core.NeuralModule
:show-inheritance:
:members:
:member-order: bysource
Neural Type classes
-------------------
.. autoclass:: nemo.core.neural_types.NeuralType
:show-inheritance:
:members:
:member-order: bysource
.. autoclass:: nemo.core.neural_types.axes.AxisType
:show-inheritance:
:members:
:member-order: bysource
.. autoclass:: nemo.core.neural_types.elements.ElementType
:show-inheritance:
:members:
:member-order: bysource
.. autoclass:: nemo.core.neural_types.comparison.NeuralTypeComparisonResult
:show-inheritance:
:members:
:member-order: bysource
Experiment manager
------------------
.. autoclass:: nemo.utils.exp_manager.exp_manager
:show-inheritance:
:members:
:member-order: bysource
tts/NeMo/docs/source/core/whyntypes.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 302 KiB

tts/NeMo/docs/source/favicon.ico
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

tts/NeMo/docs/source/index.rst
+67
View File
@@ -0,0 +1,67 @@
NVIDIA NeMo User Guide
======================
.. toctree::
:maxdepth: 2
:caption: Getting Started
:name: starthere
starthere/intro
starthere/tutorials
starthere/best-practices
.. toctree::
:maxdepth: 2
:caption: NeMo Core
:name: core
core/core
.. toctree::
:maxdepth: 2
:caption: Speech Processing
:name: Speech Processing
asr/intro
asr/speech_classification/intro
asr/speaker_recognition/intro
asr/speaker_diarization/intro
.. toctree::
:maxdepth: 2
:caption: Natural Language Processing
:name: Natural Language Processing
nlp/intro
.. toctree::
:maxdepth: 2
:caption: Text To Speech
:name: Text To Speech
:includehidden:
:titlesonly:
tts/intro
.. toctree::
:maxdepth: 2
:caption: Common
:name: Common
common/intro
.. toctree::
:maxdepth: 2
:caption: Text Processing
:name: Text Processing
nemo_text_processing/intro
.. toctree::
:maxdepth: 2
:caption: Tools
:name: Tools
tools/intro
tts/NeMo/docs/source/nemo_text_processing/api.rst
Executable
+35
View File
@@ -0,0 +1,35 @@
NeMo Text Processing API
========================
Model Classes
-------------
.. autoclass:: nemo_text_processing.text_normalization.GraphFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.text_normalization.ClassifyFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.text_normalization.VerbalizeFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.text_normalization.VerbalizeFinalFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.inverse_text_normalization.ClassifyFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.inverse_text_normalization.VerbalizeFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.inverse_text_normalization.VerbalizeFinalFst
:show-inheritance:
:members:
tts/NeMo/docs/source/nemo_text_processing/intro.rst
+17
View File
@@ -0,0 +1,17 @@
Text Processing
===============
`nemo_text_processing` is a python package that is installed with the `nemo_toolkit`.
See :doc:`NeMo Introduction <../starthere/intro>` for installation details.
Additional requirements can be found in `setup.sh <https://github.com/NVIDIA/NeMo/blob/main/nemo_text_processing/setup.sh>`_.
.. toctree::
:maxdepth: 1
text_normalization
inverse_text_normalization
api
tts/NeMo/docs/source/nemo_text_processing/inverse_text_normalization.rst
+90
View File
@@ -0,0 +1,90 @@
Inverse Text Normalization
==========================
Inverse text normalization (ITN) is a part of the Automatic Speech Recognition (ASR) post-processing pipeline.
ITN is the task of converting the raw spoken output of the ASR model into its written form to improve text readability.
For example,
`"in nineteen seventy"` -> `"in 1975"`
and `"it costs one hundred and twenty three dollars"` -> `"it costs $123"`.
NeMo ITN :cite:`textprocessing-itn-zhang2021nemo` is based on WFST-grammars :cite:`textprocessing-itn-Mohri2009`. We also provide a deployment route to C++ using `Sparrowhawk <https://github.com/google/sparrowhawk>`_ :cite:`textprocessing-itn-sparrowhawk` -- an open-source version of Google Kestrel :cite:`textprocessing-itn-ebden2015kestrel`.
See :doc:`Text Procesing Deployment <../tools/text_processing_deployment>` for details.
.. note::
For more details, see the tutorial `NeMo/tutorials/text_processing/Inverse_Text_Normalization.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/text_processing/Inverse_Text_Normalization.ipynb>`__ in `Google's Colab <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/text_processing/Inverse_Text_Normalization.ipynb>`_.
Classes
----------------------------------
The base class for every grammar is :class:`GraphFst<nemo_text_processing.text_normalization.GraphFst>`.
This tool is designed as a two-stage application: 1. `classification` of the input into semiotic tokens and 2. `verbalization` into written form.
For every stage and every semiotic token class there is a corresponding grammar, e.g. :class:`taggers.CardinalFst<nemo_text_processing.inverse_text_normalization.taggers.cardinal.CardinalFst>`
and :class:`verbalizers.CardinalFst<nemo_text_processing.inverse_text_normalization.verbalizers.cardinal.CardinalFst>`.
Together, they compose the final grammars :class:`ClassifyFst<nemo_text_processing.inverse_text_normalization.ClassifyFst>` and
:class:`VerbalizeFinalFst<nemo_text_processing.inverse_text_normalization.VerbalizeFinalFst>` that are compiled into WFST and used for inference.
.. autoclass:: nemo_text_processing.inverse_text_normalization.ClassifyFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.inverse_text_normalization.VerbalizeFinalFst
:show-inheritance:
:members:
Prediction
----------------------------------
Example prediction run:
.. code::
python run_prediction.py --input=<INPUT_TEXT_FILE> --output=<OUTPUT_PATH> [--verbose]
The input is expected to be lower-cased. Punctuation are outputted with separating spaces after semiotic tokens, e.g. `"i see, it is ten o'clock..."` -> `"I see, it is 10:00 . . ."`.
Inner-sentence white-space characters in the input are not maintained.
Data Cleaning for Evaluation
----------------------------------
.. code::
python clean_eval_data.py --input=<INPUT_TEXT_FILE>
Evaluation
----------------------------------
Example evaluation run on (cleaned) `Google's text normalization dataset <https://www.kaggle.com/richardwilliamsproat/text-normalization-for-english-russian-and-polish>`__ :cite:`textprocessing-itn-sproat2016rnn`:
.. code::
python run_evaluation.py --input=./en_with_types/output-00001-of-00100 [--cat CLASS_CATEGORY] [--filter]
References
----------
.. bibliography:: textprocessing_all.bib
:style: plain
:labelprefix: TEXTPROCESSING-ITN
:keyprefix: textprocessing-itn-
tts/NeMo/docs/source/nemo_text_processing/text_normalization.rst
+78
View File
@@ -0,0 +1,78 @@
Text Normalization
==================
NeMo Text Normalization converts text from written form into its verbalized form. It is used as a preprocessing step before Text to Speech (TTS). It could also be used for preprocessing Automatic Speech Recognition (ASR) training transcripts.
For example,
`"at 10:00"` -> `"at ten o'clock"`
and `"it weighs 10kg."` -> `"it weights ten kilograms ."`.
NeMo Text Normalization :cite:`textprocessing-norm-zhang2021nemo` is based on WFST-grammars :cite:`textprocessing-norm-Mohri2009`. We also provide a deployment route to C++ using `Sparrowhawk <https://github.com/google/sparrowhawk>`_ :cite:`textprocessing-norm-sparrowhawk` -- an open-source version of Google Kestrel :cite:`textprocessing-norm-ebden2015kestrel`.
See :doc:`Text Procesing Deployment <../tools/text_processing_deployment>` for details.
.. note::
For more details, see the tutorial `NeMo/tutorials/text_processing/Text_Normalization.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/text_processing/Text_Normalization.ipynb>`__ in `Google's Colab <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/text_processing/Text_Normalization.ipynb>`_.
Classes
----------------------------------
The base class for every grammar is :class:`GraphFst<nemo_text_processing.text_normalization.GraphFst>`.
This tool is designed as a two-stage application: 1. `classification` of the input into semiotic tokens and 2. `verbalization` into written form.
For every stage and every semiotic token class there is a corresponding grammar, e.g. :class:`taggers.CardinalFst<nemo_text_processing.text_normalization.taggers.cardinal.CardinalFst>`
and :class:`verbalizers.CardinalFst<nemo_text_processing.text_normalization.verbalizers.cardinal.CardinalFst>`.
Together, they compose the final grammars :class:`ClassifyFst<nemo_text_processing.text_normalization.ClassifyFst>` and
:class:`VerbalizeFinalFst<nemo_text_processing.text_normalization.VerbalizeFinalFst>` that are compiled into WFST and used for inference.
.. autoclass:: nemo_text_processing.text_normalization.ClassifyFst
:show-inheritance:
:members:
.. autoclass:: nemo_text_processing.text_normalization.VerbalizeFinalFst
:show-inheritance:
:members:
Prediction
----------------------------------
Example prediction run:
.. code::
python run_prediction.py <--input INPUT_TEXT_FILE> <--output OUTPUT_PATH> [--input_case INPUT_CASE]
``INPUT_CASE`` specifies whether to treat the input as lower-cased or case sensitive. By default treat the input as cased since this is more informative, especially for abbreviations. Punctuation are outputted with separating spaces after semiotic tokens, e.g. `"I see, it is 10:00..."` -> `"I see, it is ten o'clock . . ."`.
Inner-sentence white-space characters in the input are not maintained.
Evaluation
----------------------------------
Example evaluation run on `Google's text normalization dataset <https://www.kaggle.com/richardwilliamsproat/text-normalization-for-english-russian-and-polish>`__ :cite:`textprocessing-norm-sproat2016rnn`:
.. code::
python run_evaluation.py --input=./en_with_types/output-00001-of-00100 [--cat CLASS_CATEGORY] [--input_case INPUT_CASE]
References
----------
.. bibliography:: textprocessing_all.bib
:style: plain
:labelprefix: TEXTPROCESSING-NORM
:keyprefix: textprocessing-norm-
tts/NeMo/docs/source/nemo_text_processing/textprocessing_all.bib
+58
View File
@@ -0,0 +1,58 @@
@article{ebden2015kestrel,
title={The Kestrel TTS text normalization system},
author={Ebden, Peter and Sproat, Richard},
journal={Natural Language Engineering},
volume={21},
number={3},
pages={333},
year={2015},
publisher={Cambridge University Press}
}
@article{sproat2016rnn,
title={RNN approaches to text normalization: A challenge},
author={Sproat, Richard and Jaitly, Navdeep},
journal={arXiv preprint arXiv:1611.00068},
year={2016}
}
@book{taylor2009text,
title={Text-to-speech synthesis},
author={Taylor, Paul},
year={2009},
publisher={Cambridge university press}
}
@misc{zhang2021nemo,
title={NeMo Inverse Text Normalization: From Development To Production},
author={Yang Zhang and Evelina Bakhturina and Kyle Gorman and Boris Ginsburg},
year={2021},
eprint={2104.05055},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@Inbook{Mohri2009,
author="Mohri, Mehryar",
editor="Droste, Manfred
and Kuich, Werner
and Vogler, Heiko",
title="Weighted Automata Algorithms",
bookTitle="Handbook of Weighted Automata",
year="2009",
publisher="Springer Berlin Heidelberg",
address="Berlin, Heidelberg",
pages="213--254",
abstract="Weighted automata and transducers are widely used in modern applications in bioinformatics and text, speech, and image processing. This chapter describes several fundamental weighted automata and shortest-distance algorithms including composition, determinization, minimization, and synchronization, as well as single-source and all-pairs shortest distance algorithms over general semirings. It presents the pseudocode of these algorithms, gives an analysis of their running time complexity, and illustrates their use in some simple cases. Many other complex weighted automata and transducer algorithms used in practice can be obtained by combining these core algorithms.",
isbn="978-3-642-01492-5",
doi="10.1007/978-3-642-01492-5_6",
url="https://doi.org/10.1007/978-3-642-01492-5_6"
}
@inproceedings{sparrowhawk,
title = {TTS for Low Resource Languages: A Bangla Synthesizer},
author = {Alexander Gutkin and Linne Ha and Martin Jansche and Knot Pipatsrisawat and Richard Sproat},
booktitle = {10th Language Resources and Evaluation Conference},
year = {2016},
}
tts/NeMo/docs/source/nlp/api.rst
Executable
+74
View File
@@ -0,0 +1,74 @@
NeMo NLP collection API
=======================
Model Classes
-------------
.. autoclass:: nemo.collections.nlp.models.TextClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact, classifytext
.. autoclass:: nemo.collections.nlp.models.GLUEModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.nlp.models.PunctuationCapitalizationModel
:show-inheritance:
:members: add_punctuation_capitalization, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, multi_validation_epoch_end, register_artifact
.. autoclass:: nemo.collections.nlp.models.TokenClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.nlp.models.QAModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, inference, validation_epoch_end, test_epoch_end
.. autoclass:: nemo.collections.nlp.models.BERTLMModel
:show-inheritance:
:members: setup_training_data, setup_optimization
Modules
-------
.. autoclass:: nemo.collections.nlp.modules.BertModule
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.MegatronBertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.AlbertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.BertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.DistilBertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.RobertaEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.SequenceClassifier
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.SequenceRegression
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.SequenceTokenClassifier
:show-inheritance:
:members:
.. autofunction:: nemo.collections.nlp.modules.get_lm_model
.. autofunction:: nemo.collections.nlp.modules.get_pretrained_lm_models_list
.. autofunction:: nemo.collections.nlp.modules.get_megatron_lm_models_list
tts/NeMo/docs/source/nlp/bert_pretraining.rst
Executable
+133
View File
@@ -0,0 +1,133 @@
.. _bert_pretraining:
BERT
====
BERT is an autoencoding language model with a final loss composed of:
- masked language model loss
- next sentence prediction
The model architecture is published in `BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ :cite:`nlp-bert-devlin2018bert`.
The model is originally trained on English Wikipedia and BookCorpus. BERT is often used as a language model encoder for downstream tasks, for example, :ref:`token_classification`, :ref:`text_classification`, :ref:`question_answering`, etc.
Domain-specific BERT models can be advantageous for a wide range of applications. One notable application is the domain-specific BERT in a biomedical setting,
e.g. BioBERT :cite:`nlp-bert-lee2019biobert` or its improved derivative BioMegatron :cite:`nlp-bert-shin2020biomegatron`. For the latter, refer to :ref:`megatron_finetuning`.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import BERTLMModel
# to get the list of pre-trained models
BERTLMModel.list_available_models()
# Download and load the pre-trained BERT-based model
model = BERTLMModel.from_pretrained("bertbaseuncased")
Available Models
^^^^^^^^^^^^^^^^
.. list-table:: *Pretrained Models*
:widths: 5 10
:header-rows: 1
* - Model
- Pretrained Checkpoint
* - BERT-base uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:bertbaseuncased
* - BERT-large uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:bertlargeuncased
.. _dataset_bert_pretraining:
Data Input for the BERT model
-----------------------------
Data preprocessing can be either done on-the-fly during training or offline before training. The latter is optimized and recommended
for large text corpora. This was also used in the original paper to train the model on Wikipedia and BookCorpus. For on-the-fly data
processing, provide text files with sentences for training and validation, where words are separated by spaces, i.e.: ``[WORD] [SPACE] [WORD] [SPACE] [WORD]``.
To use this pipeline in training, use the dedicated configuration file `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml`.
To process data offline in advance, refer to the `BERT Quick Start Guide <https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling/BERT#quick-start-guide>`__.
To recreate the original Wikipedia and BookCorpus datasets, follow steps 1-5 in the Quick Start Guide and run the script ``./data/create_datasets_from_start.sh`` inside the Docker container.
The ``downloaded`` folder should include two sub folders ``lower_case_[0,1]_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5``
and ``lower_case_[0,1]_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5``, containing sequences of length 128 with a maximum of 20 masked tokens
and sequences of length 512 with a maximum of 80 masked tokens respectively. To use this pipeline in training, use the dedicated configuration file ``NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml``
and specify the path to the created hd5f files.
Training the BERT model
-----------------------
Example of model configuration for on-the-fly data preprocessing: `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml>`__.
Example of model configuration for offline data preprocessing: `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml>`__.
The specification can be grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the datasets: **model.train_ds**, **model.validation_ds**
- Parameters that describe the model: **model**, **model.tokenizer**, **model.language_model**
More details about parameters in the config file can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.only_mlm_loss** | bool | Only uses masked language model without next sentence prediction. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.data_file** | string | Name of the text file or hdf5 data directory. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.num_samples** | integer | Number of samples to use from the training dataset, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
More details about parameters for offline data preprocessing can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.max_predictions_per_seq** | integer | Maximum number of masked tokens in a sequence in the preprocessed data. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
More details about parameters for online data preprocessing can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.max_seq_length** | integer | The maximum total input sequence length after tokenization. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.mask_prob** | float | Probability of masking a token in the input text during data processing. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.short_seq_prob** | float | Probability of having a sequence shorter than the maximum sequence length. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
.. note::
For offline data preprocessing, **model.tokenizer** is null. For downstream task, use the same tokenizer that was used for
offline preprocessing. For online data preprocessing, **model.tokenizer** needs to be specified. See also :ref:`nlp_model` for
details.
Example of the command for training the model:
.. code::
python bert_pretraining.py \
model.train_ds.data_file=<PATH_TO_DATA> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.gpus=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>]
Fine-tuning on Downstream Tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To use a trained BERT model checkpoint on a NeMo NLP downstream task, e.g. :ref:`question_answering`, specify
:code:`model.language_model.lm_checkpoint=<PATH_TO_CHECKPOINT>`.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-BERT
:keyprefix: nlp-bert-
tts/NeMo/docs/source/nlp/glue_benchmark.rst
+10
View File
@@ -0,0 +1,10 @@
.. _glue_benchmark:
GLUE Benchmark
==============
We recommend you try the GLUE Benchmark model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_): `NeMo/tutorials/nlp/GLUE_Benchmark.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/GLUE_Benchmark.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train the model can be found here: `NeMo/examples/nlp/glue_benchmark/glue_benchmark.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/glue_benchmark/glue_benchmark.py>`__.
tts/NeMo/docs/source/nlp/information_retrieval.rst
+10
View File
@@ -0,0 +1,10 @@
.. _information_retrieval:
Information Retrieval
=====================
We recommend you try the Information Retrieval model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_): `NeMo/tutorials/nlp/Information_Retrieval_MSMARCO.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Information_Retrieval_MSMARCO.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for hardware the accelerator),
An example script on how to train the model can be found here: `NeMo/examples/nlp/information_retrieval <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp/information_retrieval>`__.
tts/NeMo/docs/source/nlp/intro.rst
Executable
+15
View File
@@ -0,0 +1,15 @@
Natural Language Processing (NLP)
=================================
.. toctree::
:maxdepth: 2
models
megatron_finetuning
api
Example scripts for the NLP collection can be found under `NeMo/example/nlp/ <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp>`__.
NLP notebook-tutorials are located under `NeMo/tutorials/nlp/ <https://github.com/NVIDIA/NeMo/tree/main/tutorials/nlp>`__. Most NeMo
tutorials can be run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.), with more details in the :ref:`tutorials` section.
tts/NeMo/docs/source/nlp/joint_intent_slot.rst
+244
View File
@@ -0,0 +1,244 @@
.. _intent_slot:
Joint Intent and Slot Classification
====================================
Joint Intent and Slot classification is a NLU task for classifying an intent and detecting all
relevant slots (Entities) for the intent in a query. For example, in the query ``What is the weather in Santa Clara tomorrow morning?``,
we would like to classify the query as a ``weather intent``, detect ``Santa Clara`` as a `location slot`,
and ``tomorrow morning`` as a ``date_time slot``. Intent and Slot names are usually task-specific and
defined as labels in the training data. This is a fundamental step that is executed in any
task-driven conversational assistant.
Our BERT-based model implementation allows you to train and detect both of these tasks together.
.. note::
We recommend you try the Joint Intent and Slot Classification model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.): `NeMo/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train the model can be found here: `NeMo/examples/nlp/intent_slot_classification <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp/intent_slot_classification>`__.
NeMo Data Format
----------------
When training the model, the dataset should be first converted to the required data format, which requires the following files:
- :code:`dict.intents.csv` - A list of all intent names in the data. One line per an intent name. The index of the intent line
(starting from ``0``) is used to identify the appropriate intent in ``train.tsv`` and ``test.tsv`` files.
.. code::
weather
alarm
meeting
...
- :code:`dict.slots.csv` - A list of all slot names in the data. One line per slot name. The index of the slot line
(starting from ``0``) is used to identify the appropriate slots in the queries in ``train_slot.tsv`` and ``test_slot.tsv`` files.
In the last line of this dictionary ``O`` slot name is used to identify all ``out of scope`` slots, which are usually the majority of the tokens
in the queries.
.. code::
date
time
city
...
O
- :code:`train.tsv/test.tsv` - A list of original queries, one per line, with the intent number
separated by a tab (e.g. "what alarms do i have set right now <TAB> 0"). Intent numbers are
set according to the intent line in the intent dictionary file (:code:`dict.intents.csv`),
starting from ``0``. The first line in these files should contain the header line ``sentence
<tab> label``.
- :code:`train_slot.tvs/test_slot.tsv` - A list that contains one line per query, when each word from the original text queries
is replaced by a token number from the slots dictionary file (``dict.slots.csv``), counted starting from ``0``. All the words
which do not contain a relevant slot are replaced by ``out-of scope`` token number, which is also a part of the slot dictionary file,
usually as the last entry there. For example a line from these files should look similar to: "54 0 0 54 54 12 12" (the numbers are
separated by a space). These files do not contain a header line.
Dataset Conversion
------------------
To convert to the format of the model data, use the ``import_datasets`` utility, which implements
the conversion for the Assistant dataset. Download the dataset `here <https://github.com/xliuhw/NLU-Evaluation-Data>`_ or you can
write your own converter for the format that you are using for data annotation.
For a dataset that follows your own annotation format, we recommend using one text file for all
samples of the same intent, with the name of the file as the name of the intent. Use one line per
query, with brackets to define slot names. This is very similar to the assistant format, and you can
adapt this converter utility or your own format with small changes:
::
did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]
Run the ``dataset_converter`` command:
.. code::
python examples/nlp/intent_slot_classification/data/import_datasets.py
--source_data_dir=`source_data_dir` \
--target_data_dir=`target_data_dir` \
--dataset_name=['assistant'|'snips'|'atis']
- :code:`source_data_dir`: the directory location of the your dataset
- :code:`target_data_dir`: the directory location where the converted dataset should be saved
- :code:`dataset_name`: one of the implemented dataset names
After conversion, ``target_data_dir`` should contain the following files:
.. code::
.
|--target_data_dir
|-- dict.intents.csv
|-- dict.slots.csv
|-- train.tsv
|-- train_slots.tsv
|-- test.tsv
|-- test_slots.tsv
Model Training
--------------
This is a pretrained BERT based model with 2 linear classifier heads on the top of it, one for classifying an intent of the query and
another for classifying slots for each token of the query. This model is trained with the combined loss function on the Intent and Slot
classification task on the given dataset. The model architecture is based on the paper `BERT for Joint Intent Classification and Slot Filling <https://arxiv.org/pdf/1902.10909.pdf>`__:cite:`nlp-jis-chen2019bert`.
For each query, the model classifies it as one the intents from the intent dictionary and for each word of the query it will classify
it as one of the slots from the slot dictionary, including out of scope slot for all the remaining words in the query which does not
fall in another slot category. Out of scope slot (``O``) is a part of slot dictionary that the model is trained on.
Example of model configuration file for training the model can be found at: `NeMo/examples/nlp/intent_slot_classification/conf/intent_slot_classification.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/intent_slot_classification/conf/intent_slot_classification_config.yaml>`__.
In the configuration file, define the parameters of the training and the model, although most of the default values will work well.
The specification can be roughly grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the model: **model**
- Parameters that describe the datasets: **model.train_ds**, **model.validation_ds**, **model.test_ds**,
More details about parameters in the spec file can be found below:
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.data_dir** | string | -- | The path of the data converted to the specified format. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.class_balancing** | string | ``null`` | Choose from ``[null, weighted_loss]``. The ``weighted_los``s enables weighted class balancing of the loss. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.intent_loss_weight** | float | ``0.6`` | The elation of intent-to-slot loss in the total loss. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.pad_label** | integer | ``-1`` | A value to pad the inputs. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.ignore_extra_tokens** | boolean | ``false`` | A flag that specifies whether to ignore extra tokens. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.ignore_start_end** | boolean | ``true`` | A flag that specifies whether to not use the first and last token for slot training. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.num_output_layers** | integer | ``2`` | The number of fully connected layers of the classifier on top of the BERT model. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.fc_dropout** | float | ``0.1`` | The dropout ratio of the fully connected layers. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.prefix** | string | ``train`` | A prefix for the training file names. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.prefix** | string | ``dev`` | A prefix for the validation file names. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **test_ds.prefix** | string | ``test`` | A prefix for the test file names. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
For additional config parameters common to all NLP models, refer to the `nlp_model doc <https://github.com/NVIDIA/NeMo/blob/main/docs/source/nlp/nlp_model.rst#model-nlp>`__.
The following is an example of the command for training the model:
.. code::
python examples/nlp/intent_slot_classification/intent_slot_classification.py
model.data_dir=<PATH_TO_DATA_DIR> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.gpus=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>]
Required Arguments for Training
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`model.data_dir`: the dataset directory
Optional Arguments
^^^^^^^^^^^^^^^^^^
Most of the default parameters in the existing configuration file are already set appropriately, however, there are some parameters
you may want to experiment with.
- ``trainer.max_epochs``: the number of training epochs (reasonable to be between 10 to 100)
- ``model.class_balancing`` - value ``weighted_loss`` may help to train the model when there is unbalanced set of classes
- ``model.intent_loss_weight`` - a number between 0 to 1 that defines a weight of the intent lost versus a slot loss during training. A
default value 0.6 gives a slight preference for the intent lose optimization.
Training Procedure
^^^^^^^^^^^^^^^^^^
At the start of evaluation, NeMo will print out a log of the experiment specification, a summary of the training dataset, and the
model architecture.
As the model starts training, you should see a progress bar per epoch. During training, after each epoch, NeMo will display accuracy
metrics on the validation dataset for every intent and slot separately, as well as the total accuracy. You can expect these numbers
to grow up to 50-100 epochs, depending on the size of the trained data. Since this is a joint iIntent and slot training, usually
intent's accuracy will grow first for the initial 10-20 epochs, and after that, slot's accuracy will start improving as well.
At the end of training, NeMo saves the best checkpoint on the validation dataset at the path specified by the experiment spec file
before finishing.
.. code::
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
[NeMo W 2021-01-28 14:52:19 exp_manager:299] There was no checkpoint folder at checkpoint_dir :results/checkpoints. Training from scratch.
[NeMo I 2021-01-28 14:52:19 exp_manager:186] Experiments will be logged at results
...
label precision recall f1 support
weather.weather (label_id: 0) 0.00 0.00 0.00 128
weather.temperature (label_id: 1) 0.00 0.00 0.00 0
weather.temperature_yes_no (label_id: 2) 0.00 0.00 0.00 0
weather.rainfall (label_id: 3) 0.00 0.00 0.00 0
weather.rainfall_yes_no (label_id: 4) 0.00 0.00 0.00 0
weather.snow (label_id: 5) 0.00 0.00 0.00 0
weather.snow_yes_no (label_id: 6) 0.00 0.00 0.00 0
weather.humidity (label_id: 7) 0.00 0.00 0.00 0
weather.humidity_yes_no (label_id: 8) 0.00 0.00 0.00 0
weather.windspeed (label_id: 9) 0.00 0.00 0.00 0
weather.sunny (label_id: 10) 0.00 0.00 0.00 0
weather.cloudy (label_id: 11) 0.00 0.00 0.00 0
weather.alert (label_id: 12) 0.00 0.00 0.00 0
context.weather (label_id: 13) 0.00 0.00 0.00 0
context.continue (label_id: 14) 0.00 0.00 0.00 0
context.navigation (label_id: 15) 0.00 0.00 0.00 0
context.rating (label_id: 16) 0.00 0.00 0.00 0
context.distance (label_id: 17) 0.00 0.00 0.00 0
-------------------
micro avg 0.00 0.00 0.00 128
macro avg 0.00 0.00 0.00 128
weighted avg 0.00 0.00 0.00 128
Model Evaluation and Inference
------------------------------
There is no separate script for the evaluation and inference of this model in NeMo, however, inside of the example file `examples/nlp/intent_slot_classification/intent_slot_classification.py`
after the training part is finished, you can see the code that evaluates the trained model on an evaluation test set and then an example of doing inference using a list of given queries.
For the deployment in the production environment, refer to `NVIDIA Jarvis <https://developer.nvidia.com/nvidia-jarvis-getting-started>`__ and `NVIDIA TLT documentation <https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/nlp/index.html>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-JIS
:keyprefix: nlp-jis-
tts/NeMo/docs/source/nlp/language_modeling.rst
+6
View File
@@ -0,0 +1,6 @@
.. _language_modeling:
Language Modeling
=================
An example script on how to train the model can be found here: `NeMo/examples/nlp/language_modeling <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp/language_modeling>`__.
tts/NeMo/docs/source/nlp/machine_translation.rst
+466
View File
@@ -0,0 +1,466 @@
.. _machine_translation:
Machine Translation Models
==========================
Machine translation is the task of translating text from one language to another. For example, from English to Spanish. Models are
based on the Transformer sequence-to-sequence architecture :cite:`nlp-machine_translation-vaswani2017attention`.
An example script on how to train the model can be found here: `NeMo/examples/nlp/machine_translation/enc_dec_nmt.py <https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/nlp/machine_translation/enc_dec_nmt.py>`__.
The default configuration file for the model can be found at: `NeMo/examples/nlp/machine_translation/conf/aayn_base.yaml <https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/nlp/machine_translation/conf/aayn_base.yaml>`__.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import MTEncDecModel
# To get the list of pre-trained models
MTEncDecModel.list_available_models()
# Download and load the a pre-trained to translate from English to Spanish
model = MTEncDecModel.from_pretrained("nmt_en_es_transformer12x2")
# Translate a sentence or list of sentences
translations = model.translate(["Hello!"], source_lang="en", target_lang="es")
Available Models
^^^^^^^^^^^^^^^^
.. list-table:: *Pretrained Models*
:widths: 5 10
:header-rows: 1
* - Model
- Pretrained Checkpoint
* - English -> German
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_de_transformer12x2
* - German -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_de_en_transformer12x2
* - English -> Spanish
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_es_transformer12x2
* - Spanish -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_es_en_transformer12x2
* - English -> French
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_fr_transformer12x2
* - French -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_fr_en_transformer12x2
* - English -> Russian
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_ru_transformer6x6
* - Russian -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_ru_en_transformer6x6
* - English -> Chinese
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_zh_transformer6x6
* - Chinese -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_zh_en_transformer6x6
Data Format
-----------
Supervised machine translation models require parallel corpora which comprises many examples of sentences in a source language and
their corresponding translation in a target language. We use parallel data formatted as seperate text files for source and target
languages where sentences in corresponding files are aligned like in the table below.
.. list-table:: *Parallel Coprus*
:widths: 10 10
:header-rows: 1
* - train.english.txt
- train.spanish.txt
* - Hello .
- Hola .
* - Thank you .
- Gracias .
* - You can now translate from English to Spanish in NeMo .
- Ahora puedes traducir del inglés al español en NeMo .
It is common practice to apply data cleaning, normalization, and tokenization to the data prior to training a translation model and
NeMo expects already cleaned, normalized, and tokenized data. The only data pre-processing NeMo does is subword tokenization with BPE
:cite:`nlp-machine_translation-sennrich2015neural`.
Data Cleaning, Normalization & Tokenization
-------------------------------------------
We recommend applying the following steps to clean, normalize, and tokenize your data. All pre-trained models released, apply these data pre-processing steps.
#. Language ID filtering - This step filters out examples from your training dataset that aren't in the correct language. For example,
many datasets contain examples where source and target sentences are in the same language. You can use a pre-trained language ID
classifier from `fastText <https://fasttext.cc/docs/en/language-identification.html>`__. You can then run our script using the
``lid.176.bin`` model downloaded from the fastText website.
.. code ::
python NeMo/scripts/neural_machine_translation/filter_by_language.py \
--input-src train.en \
--input-tgt train.es \
--output-src train_lang_filtered.en \
--output-tgt train_lang_filtered.es \
--source-lang en \
--target-lang es \
--removed-src train_noise.en \
--removed-tgt train_noise.de \
--fasttext-model lid.176.bin
#. Length filtering - We filter out sentences from the data that are below a minimum length (1) or exceed a maximum length (250). We
also filter out sentences where the ratio between source and target lengths exceeds 1.3 except for English <-> Chinese models.
`Moses <https://github.com/moses-smt/mosesdecoder>`__ is a statistical machine translation toolkit that contains many useful
pre-processing scripts.
.. code ::
perl mosesdecoder/scripts/training/clean-corpus-n.perl -ratio 1.3 train en es train.filter 1 250
#. Data cleaning - While language ID filtering can sometimes help with filtering out noisy sentences that contain too many puncutations,
it does not help in cases where the translations are potentially incorrect, disfluent, or incomplete. We use `bicleaner <https://github.com/bitextor/bicleaner>`__
a tool to identify such sentences. It trains a classifier based on many features included pre-trained language model fluency, word
alignment scores from a word-alignment model like `Giza++ <https://github.com/moses-smt/giza-pp>`__ etc. We use their available
pre-trained models wherever possible and train models ourselves using their framework for remaining languages. The following script
applies a pre-trained bicleaner model to the data and pick sentences that are clean with probability > 0.5.
.. code ::
awk '{print "-\t-"}' train.en \
| paste -d "\t" - train.filter.en train.filter.es \
| bicleaner-classify - - </path/to/bicleaner.yaml> > train.en-es.bicleaner.score
#. Data deduplication - We use `bifixer <https://github.com/bitextor/bifixer>`__ (which uses xxHash) to hash the source and target
sentences based on which we remove duplicate entries from the file. You may want to do something similar to remove training examples
that are in the test dataset.
.. code ::
cat train.en-es.bicleaner.score \
| parallel -j 25 --pipe -k -l 30000 python bifixer.py --ignore-segmentation -q - - en es \
> train.en-es.bifixer.score
awk -F awk -F "\t" '!seen[$6]++' train.en-es.bifixer.score > train.en-es.bifixer.dedup.score
#. Filter out data that bifixer assigns probability < 0.5 to.
.. code ::
awk -F "\t" '{ if ($5>0.5) {print $3}}' train.en-es.bifixer.dedup.score > train.cleaned.en
awk -F "\t" '{ if ($5>0.5) {print $4}}' train.en-es.bifixer.dedup.score > train.cleaned.es
#. Punctuation Normalization - Punctuation, especially things like quotes can be written in different ways.
It's often useful to normalize the way they appear in text. We use the moses punctuation normalizer on all languages except Chinese.
.. code ::
perl mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l es < train.cleaned.es > train.normalized.es
perl mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l en < train.cleaned.en > train.normalized.en
For example:
.. code ::
Before - Aquí se encuentran joyerías como Tiffany`s entre negocios tradicionales suizos como la confitería Sprüngli.
After - Aquí se encuentran joyerías como Tiffany's entre negocios tradicionales suizos como la confitería Sprüngli.
#. Tokenization and word segmentation for Chinese - Naturally written text often contains punctuation markers like commas, full-stops
and apostrophes that are attached to words. Tokenization by just splitting a string on spaces will result in separate token IDs for
very similar items like ``NeMo`` and ``NeMo.``. Tokenization splits punctuation from the word to create two separate tokens. In the
previous example ``NeMo.`` becomes ``NeMo .`` which when split by space, results in two tokens and adressess the earlier problem.
For example:
.. code ::
Before - Especialmente porque se enfrentará "a Mathieu (Debuchy), Yohan (Cabaye) y Adil (Rami) ", recuerda.
After - Especialmente porque se enfrentará " a Mathieu ( Debuchy ) , Yohan ( Cabaye ) y Adil ( Rami ) " , recuerda .
We use the Moses tokenizer for all languages except Chinese.
.. code ::
perl mosesdecoder/scripts/tokenizer/tokenizer.perl -l es -no-escape < train.normalized.es > train.tokenized.es
perl mosesdecoder/scripts/tokenizer/tokenizer.perl -l en -no-escape < train.normalized.en > train.tokenized.en
For languages like Chinese where there is no explicit marker like spaces that separate words, we use `Jieba <https://github.com/fxsjy/jieba>`__ to segment a string into words that are space separated.
For example:
.. code ::
Before - 同时，卫生局认为有必要接种的其他人员，包括公共部门，卫生局将主动联络有关机构取得名单后由卫生中心安排接种。
After - 同时 ， 卫生局 认为 有 必要 接种 的 其他 人员 ， 包括 公共部门 ， 卫生局 将 主动 联络 有关 机构 取得 名单 后 由 卫生 中心 安排 接种 。
Training a BPE Tokenization
---------------------------
Byte-pair encoding (BPE) :cite:`nlp-machine_translation-sennrich2015neural` is a sub-word tokenization algorithm that is commonly used
to reduce the large vocabulary size of datasets by splitting words into frequently occuring sub-words. Currently, mMachine translation
only supports the `YouTokenToMe <https://github.com/VKCOM/YouTokenToMe>`__ BPE tokenizer. One can set the tokenization configuration
as follows:
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.tokenizer_name** | str | ``yttm`` | BPE library name. Only supports ``yttm`` for now. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.tokenizer_model** | str | ``null`` | Path to an existing YTTM BPE model. If ``null``, will train one from scratch on the provided data. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.vocab_size** | int | ``null`` | Desired vocabulary size after BPE tokenization. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.bpe_dropout** | float | ``null`` | BPE dropout probability. :cite:`nlp-machine_translation-provilkov2019bpe`. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.vocab_file** | str | ``null`` | Path to pre-computed vocab file if exists. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.shared_tokenizer** | bool | ``True`` | Whether to share the tokenizer between the encoder and decoder. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
Applying BPE Tokenization, Batching, Bucketing and Padding
----------------------------------------------------------
Given BPE tokenizers, and a cleaned parallel corpus, the following steps are applied to create a `TranslationDataset <https://github.com/NVIDIA/NeMo/blob/v1.0.2/nemo/collections/nlp/data/machine_translation/machine_translation_dataset.py#L64>`__ object.
#. Text to IDs - This performs subword tokenization with the BPE model on an input string and maps it to a sequence of tokens for the
source and target text.
#. Bucketing - Sentences vary in length and when creating minibatches, we'd like sentences in them to have roughly the same length to
minimize the number of ``<pad>`` tokens and to maximize computational efficiency. This step groups sentences roughly the same length
into buckets.
#. Batching and padding - Creates minibatches with a maximum number of tokens specified by ``model.{train_ds,validation_ds,test_ds}.tokens_in_batch``
from buckets and pads, so they can be packed into a tensor.
Datasets can be configured as follows:
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.src_file_name** | str | ``null`` | Path to the source language file. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tgt_file_name** | str | ``null`` | Path to the target language file. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tokens_in_batch** | int | ``512`` | Maximum number of tokens per minibatch. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.clean** | bool | ``true`` | Whether to clean the dataset by discarding examples that are greater than ``max_seq_length``. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.max_seq_length** | int | ``512`` | Maximum sequence to be used with the ``clean`` argument above. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.shuffle** | bool | ``true`` | Whether to shuffle minibatches in the PyTorch DataLoader. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.num_samples** | int | ``-1`` | Number of samples to use. ``-1`` for the entire dataset. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.drop_last** | bool | ``false`` | Drop last minibatch if it is not of equal size to the others. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.pin_memory** | bool | ``false`` | Whether to pin memory in the PyTorch DataLoader. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.num_workers** | int | ``8`` | Number of workers for the PyTorch DataLoader. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
Tarred Datasets for Large Corpora
---------------------------------
When training with ``DistributedDataParallel``, each process has its own copy of the dataset. For large datasets, this may not always
fit in CPU memory. `Webdatasets <https://github.com/tmbdev/webdataset>`__ circumvents this problem by efficiently iterating over
tar files stored on disk. Each tar file can contain hundreds to thousands of pickle files, each containing a single minibatch.
We recommend using this method when working with datasets with > 1 million sentence pairs.
Tarred datasets can be configured as follows:
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.use_tarred_dataset** | bool | ``false`` | Whether to use tarred datasets. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tar_files** | str | ``null`` | String specifying path to all tar files. Example with 100 tarfiles ``/path/to/tarfiles._OP_1..100_CL_.tar``. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.metadata_file** | str | ``null`` | Path to JSON metadata file that contains only a single entry for the total number of batches in the dataset. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.lines_per_dataset_fragment** | int | ``1000000`` | |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.num_batches_per_tarfile** | int | ``100`` | Maximum sequence to be used with the ``clean`` argument above. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tar_shuffle_n** | int | ``100`` | Whether to cache IDs to avoid re-tokenizing data. This will be deprecated in favor of tarred datasets. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.shard_strategy** | str | ``scatter`` | Whether to cache IDs in each of the nodes in multi-node training. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.preproc_out_dir** | str | ``null`` | Path to folder that contains processed tar files or directory where new tar files are written. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
Tarred datasets can be created in two ways:
#. Using the Hydra config and `training script <https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/nlp/machine_translation/enc_dec_nmt.py>`__.
For example:
.. code ::
python examples/nlp/machine_translation/enc_dec_nmt.py \
-cn aayn_base \
do_training=false \
model.preproc_out_dir=/path/to/preproc_dir \
model.train_ds.use_tarred_dataset=true \
model.train_ds.lines_per_dataset_fragment=1000000 \
model.train_ds.num_batches_per_tarfile=200 \
model.train_ds.src_file_name=train.tokenized.en \
model.train_ds.tgt_file_name=train.tokenized.es \
model.validation_ds.src_file_name=validation.tokenized.en \
model.validation_ds.tgt_file_name=validation.tokenized.es \
model.encoder_tokenizer.vocab_size=32000 \
model.decoder_tokenizer.vocab_size=32000 \
~model.test_ds \
trainer.gpus=[0,1,2,3] \
+trainer.fast_dev_run=true \
exp_manager=null \
The above script processes the parallel tokenized text files into tarred datasets that are written to ``/path/to/preproc_dir``. Since
``do_training`` is set to ``False``, the above script only creates tarred datasets and then exits. If ``do_training`` is set ``True``,
then one of two things happen:
(a) If no tar files are present in ``model.preproc_out_dir``, the script first creates those files and then commences training.
(b) If tar files are already present in ``model.preproc_out_dir``, the script starts training from the provided tar files.
#. Using a separate script without Hydra.
Tarred datasets for parallel corpora can also be created with a script that doesn't require specifying a configs via Hydra and
just uses Python argparse.
For example:
.. code ::
python examples/nlp/machine_translation/create_tarred_parallel_dataset.py \
--shared_tokenizer \
--clean \
--bpe_dropout 0.1 \
--src_fname train.tokenized.en \
--tgt_fname train.tokenized.es \
--out_dir /path/to/preproc_dir \
--vocab_size 32000 \
--max_seq_length 512 \
--min_seq_length 1 \
--tokens_in_batch 8192 \
--lines_per_dataset_fragment 1000000 \
--num_batches_per_tarfile 200
You can then set `model.preproc_out_dir=/path/to/preproc_dir` and `model.train_ds.use_tarred_dataset=true` to train with this data.
Model Configuration and Training
--------------------------------
The overall model consists of an encoder, decoder, and classification head. Encoders and decoders have the following configuration
options:
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.max_sequence_length** | int | ``512`` | Maximum sequence length of positional encodings. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.embedding_dropout** | float | ``0.1`` | Path to JSON metadata file that contains only a single entry for the total number of batches in the dataset. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.learn_positional_encodings** | bool | ``false`` | If ``True``, this is a regular learnable embedding layer. If ``False``, fixes position encodings to sinusoidal. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.hidden_size** | int | ``512`` | Size of the transformer hidden states. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.num_layers** | int | ``6`` | Number of transformer layers. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.inner_size** | int | ``2048`` | Size of the hidden states within the feedforward layers. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.num_attention_heads** | int | ``8`` | Number of attention heads. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.ffn_dropout** | float | ``0.1`` | Dropout probability within the feedforward layers. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.attn_score_dropout** | float | ``0.1`` | Dropout probability of the attention scores before softmax normalization. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.attn_layer_dropout** | float | ``0.1`` | Dropout probability of the attention query, key, and value projection activations. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.hidden_act** | str | ``relu`` | Activation function throughout the network. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.mask_future** | bool | ``false``, ``true`` | Whether to mask future timesteps for attention. Defaults to ``True`` for decoder and ``False`` for encoder. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.pre_ln** | bool | ``false`` | Whether to apply layer-normalization before (``true``) or after (``false``) a sub-layer. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
Our pre-trained models are optimized with Adam, with a maximum learning of 0.0004, beta of (0.9, 0.98), an inverse square root learning
rate schedule from :cite:`nlp-machine_translation-vaswani2017attention`. The **model.optim** section sets the optimization parameters.
The following script creates tarred datasets based on the provided parallel corpus and train a model based on the ``base`` configuration
from :cite:`nlp-machine_translation-vaswani2017attention`.
.. code ::
python examples/nlp/machine_translation/enc_dec_nmt.py \
-cn aayn_base \
do_training=true \
trainer.gpus=8 \
~trainer.max_epochs \
+trainer.max_steps=100000 \
+trainer.val_check_interval=1000 \
+exp_manager.exp_dir=/path/to/store/results \
+exp_manager.create_checkpoint_callback=True \
+exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU \
+exp_manager.checkpoint_callback_params.mode=max \
+exp_manager.checkpoint_callback_params.save_top_k=5 \
model.preproc_out_dir=/path/to/preproc_dir \
model.train_ds.use_tarred_dataset=true \
model.train_ds.lines_per_dataset_fragment=1000000 \
model.train_ds.num_batches_per_tarfile=200 \
model.train_ds.src_file_name=train.tokenized.en \
model.train_ds.tgt_file_name=train.tokenized.es \
model.validation_ds.src_file_name=validation.tokenized.en \
model.validation_ds.tgt_file_name=validation.tokenized.es \
model.encoder_tokenizer.vocab_size=32000 \
model.decoder_tokenizer.vocab_size=32000 \
~model.test_ds \
The trainer keeps track of the sacreBLEU score :cite:`nlp-machine_translation-post2018call` on the provided validation set and saves
the checkpoints that had the top 5 (by default) sacreBLEU scores.
At the end of training, a ``.nemo`` file is written to the result directory which allows to run inference on a test set.
Multi-Validation
----------------
To run validation on multiple datasets, specify ``validation_ds.src_file_name`` and ``validation_ds.tgt_file_name`` with a list of file paths:
.. code-block:: bash
model.validation_ds.src_file_name=[/data/wmt13-en-de.src,/data/wmt14-en-de.src] \
model.validation_ds.tgt_file_name=[/data/wmt13-en-de.ref,/data/wmt14-en-de.ref] \
When using ``val_loss`` or ``val_sacreBLEU`` for the ``exp_manager.checkpoint_callback_params.monitor``
then the 0th indexed dataset will be used as the monitor.
To use other indexes, append the index:
.. code-block:: bash
exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU_dl_index_1
Multiple test datasets work exactly the same way as validation datasets, simply replace ``validation_ds`` by ``test_ds`` in the above examples.
Model Inference
---------------
To generate translations on a test set and compute sacreBLEU scores, run the inference script:
.. code ::
python examples/nlp/machine_translation/nmt_transformer_infer.py \
--model /path/to/model.nemo \
--srctext test.en \
--tgtout test.en-es.translations \
--batch_size 128 \
--source_lang en \
--target_lang es
The ``--srctext`` file must be provided before tokenization and normalization. The resulting ``--tgtout`` file is detokenized and
can be used to compute sacreBLEU scores.
.. code ::
cat test.en-es.translations | sacrebleu test.es
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: nlp-machine_translation
:keyprefix: nlp-machine_translation-
tts/NeMo/docs/source/nlp/megatron_finetuning.rst
+216
View File
@@ -0,0 +1,216 @@
.. _megatron_finetuning:
Megatron-LM for Downstream Tasks
================================
Megatron-LM :cite:`nlp-megatron-shoeybi2019megatron` is a large, powerful transformer developed by the Applied Deep Learning Research
team at NVIDIA. Unlike BERT, the position of the layer normalization and the residual connection in the model architecture (similar to
GPT-2 architucture) are swapped, which allows the models to continue to improve as they were scaled up. This model reaches higher
scores compared to BERT on a range of Natural Language Processing (NLP) tasks. More details on efficient, model-parallel and multi-node
pre-training of GPT and BERT using mixed precision can be found in the `Megatron-LM GitHub repo <https://github.com/NVIDIA/Megatron-LM>`_.
Fine-tuning
-----------
Pre-trained Megatron-LM (BERT) can be used for most of the NLP downstream tasks from `NeMo/examples/nlp <https://github.com/NVIDIA/NeMo/tree/master/examples/nlp>`_.
Specify the ``model.language_model.pretrained_model_name`` parameter:
.. code-block:: bash
model.language_model.pretrained_model_name=megatron-bert-345m-uncased
Available pre-trained Megatron-LM models:
- ``megatron-bert-345m-cased``
- ``megatron-bert-345m-uncased``
- ``biomegatron-bert-345m-uncased``
- ``biomegatron-bert-345m-cased``
For example, to fine-tune SQuAD v1.1 with Megatron-LM, run:
.. code-block:: bash
python question_answering_squad.py \
model.train_ds.file=<TRAIN_JSON_FILE> \
model.validation_ds=<VAL_JSON_FILE> \
model.language_model.pretrained_model_name=megatron-bert-345m-uncased
If you have a different checkpoint or model configuration (pre-trained with `Megatron-LM GitHub repo <https://github.com/NVIDIA/Megatron-LM>`_),
use ``model.language_model.pretrained_model_name=megatron-bert-uncased`` or ``model.language_model.pretrained_model_name=megatron-bert-cased``
and specify ``--bert_config`` and ``--bert_checkpoint`` for your model.
.. note::
Megatron-LM has its own set of training arguments (including tokenizer) that are ignored during fine-tuning in NeMo. Use downstream
task training scripts for all NeMo supported arguments.
BioMegatron
-----------
BioMegatron has the same network architecture as the Megatron-LM, but is pretrained on a different dataset - `PubMed <https://catalog.data.gov/dataset/pubmed>`_,
a large biomedical text corpus, which achieves better performance in biomedical downstream tasks than the original Megatron-LM.
Examples of using BioMegatron on biomedical downstream tasks can be found at (can be executed with `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_):
`NeMo/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb>`__ and `NeMo/tutorials/nlp/Token_Classification-BioMegatron.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification-BioMegatron.ipynb>`__.
Model Parallelism
-----------------
`Megatron-LM <https://github.com/NVIDIA/Megatron-LM>`_ is a highly optimized and efficient library for training large language models.
With Megatron model parallelism, language models can be trained with billions of weights and then used in NeMo for downstream tasks.
NeMo handles pretrained model parallel checkpoints from Megatron-LM automatically and model parallel models in NeMo have the all
the same features as other NeMo Models.
.. note::
Currently, NeMo only supports tensor model parallelism.
Training
^^^^^^^^
All of the necessary logic to train model parallel models in NeMo with PyTorch Lightning is contained in the ``NLPDDPPlugin``.
The ``NLPDDPPlugin`` subclasses the PyTorch Lightning training type plugin ``DDPPlugin``.
See `plugins <https://pytorch-lightning.readthedocs.io/en/latest/extensions/plugins.html>`_ for more information on PyTorch Lightning Plugins.
To enable model parallel training in NeMo:
.. code-block:: python
trainer = Trainer(plugins=[NLPDDPPlugin()], **cfg.trainer)
Megatron-LM checkpoints have a specific format. One checkpoint is saved for each model parallel rank:
.. code-block:: bash
iter_0080000/
├── mp_rank_00
│ └── model_optim_rng.pt
└── mp_rank_01
└── model_optim_rng.pt
To start fine-tuning from a Megatron-LM checkpoint, simply pass the path to the Megatron-LM checkpoint
via the language model config:
.. code-block:: bash
model.language_model.lm_checkpoint=/raid/megatron/bert/iter_0080000 \
We also need to input the model configuration. This can be done via json:
.. code-block:: json
{
"hidden-size": 1024,
"num-attention-heads": 16,
"num-layers": 24,
"max-seq-length": 512
}
And input via command line:
.. code-block:: bash
model.language_model.config_file=/raid/data/megatron/bert/config.json \
Or the model configuration can be input via YAML:
.. code-block:: YAML
model:
language_model:
config:
hidden_size: 1024
num_attention_heads: 16
num_layers: 24
max_position_embeddings: 512
Additionally, Megatron-LM requires a vocab file:
.. code-block:: bash
model.tokenizer.vocab_file=/path/to/vocab.txt
If using the Megatron-LM default tokenizer for training BERT the vocab file can be omitted:
.. code-block:: bash
# uncased model
model.tokenizer.tokenizer_name=megatron-bert-uncased
.. code-block:: bash
# cased model
model.tokenizer.tokenizer_name=megatron-bert-uncased
Auto-Resume
^^^^^^^^^^^
Resuming training with NeMo experiment manager and PyTorch Lightning works exactly the same as other NeMo models.
While training with PTL, model parallel checkpoint will be saved and loaded properly.
.. code-block:: bash
checkpoints/
├── mp_rank_00
│ ├── mp_autoresume-last.ckpt
│ ├── mp_autoresume---val_loss=0.35-epoch=0.ckpt
│ ├── mp_autoresume---val_loss=0.38-epoch=1.ckpt
│ └── mp_autoresume---val_loss=0.39-epoch=2.ckpt
└── mp_rank_01
├── mp_autoresume-last.ckpt
├── mp_autoresume---val_loss=0.35-epoch=0.ckpt
├── mp_autoresume---val_loss=0.38-epoch=1.ckpt
└── mp_autoresume---val_loss=0.39-epoch=2.ckpt
Save and Restore
^^^^^^^^^^^^^^^^
Model parallel .nemo files behave the same as all other .nemo files. Calling ``.save_to`` will save
a checkpoint for each model parallel rank inside the .nemo file:
.. code-block:: bash
text_class_350m
├── megatron-bert-uncased_encoder_config.json
├── megatron_checkpoint_version.json
├── model_config.yaml
├── mp_rank_00
│ └── model_weights.ckpt
├── mp_rank_01
│ └── model_weights.ckpt
├── tokenizer_vocab_dict.json
└── tokenizer.vocab_file
When restoring a model parallel .nemo file, we must pass in the ``Trainer`` as model parallel requires DDP:
.. code-block:: python
model = TokenClassificationModel.restore_from(cfg.pretrained_model, trainer=trainer)
Evaluation
^^^^^^^^^^
Since model parallel models always require more than one GPU, the ``Trainer`` is needed for evaluation:
.. code-block:: python
trainer = pl.Trainer(plugins=[NLPDDPPlugin()], **cfg.trainer)
model = TextClassificationModel.restore_from(cfg.model.nemo_path, trainer=trainer)
model.setup_test_data(test_data_config=cfg.model.test_ds)
trainer.test(model=model, ckpt_path=None)
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-MEGATRON
:keyprefix: nlp-megatron-
tts/NeMo/docs/source/nlp/models.rst
Executable
+22
View File
@@ -0,0 +1,22 @@
.. _nlp_models:
Models
======
NeMo's NLP collection supports the following models:
.. toctree::
:maxdepth: 1
punctuation_and_capitalization
token_classification
joint_intent_slot
text_classification
bert_pretraining
language_modeling
question_answering
sgd_qa
glue_benchmark
information_retrieval
nlp_model
machine_translation
tts/NeMo/docs/source/nlp/nlp_all.bib
+109
View File
@@ -0,0 +1,109 @@
@article{devlin2018bert,
title={Bert: Pre-training of deep bidirectional transformers for language understanding},
author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal={arXiv preprint arXiv:1810.04805},
year={2018}
}
@article{shoeybi2019megatron,
title={Megatron-lm: Training multi-billion parameter language models using model parallelism},
author={Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanzaro, Bryan},
journal={arXiv preprint arXiv:1909.08053},
year={2019}
}
@InProceedings{maas2011,
author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher},
title = {Learning Word Vectors for Sentiment Analysis},
booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
month = {June},
year = {2011},
address = {Portland, Oregon, USA},
publisher = {Association for Computational Linguistics},
pages = {142--150},
url = {http://www.aclweb.org/anthology/P11-1015}
}
@inproceedings{socher2013,
title = "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank",
author = "Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D. and Ng, Andrew and Potts, Christopher",
booktitle = "Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2013",
address = "Seattle, Washington, USA",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/D13-1170",
pages = "1631--1642",
}
@article{lim2018chemical,
title={Chemical--gene relation extraction using recursive neural network},
author={Lim, Sangrak and Kang, Jaewoo},
journal={Database},
volume={2018},
year={2018},
publisher={Oxford Academic}
}
@inproceedings{li2007scalable,
title={Scalable term selection for text categorization},
author={Li, Jingyang and Sun, Maosong},
booktitle={Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
pages={774--782},
year={2007}
}
@misc{lee2019biobert,
title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
author={Jinhyuk Lee and Wonjin Yoon and Sungdong Kim and Donghyeon Kim and Sunkyu Kim and Chan Ho So and Jaewoo Kang},
year={2019},
eprint={1901.08746},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@misc{shin2020biomegatron,
title={BioMegatron: Larger Biomedical Domain Language Model},
author={Hoo-Chang Shin and Yang Zhang and Evelina Bakhturina and Raul Puri and Mostofa Patwary and Mohammad Shoeybi and Raghav Mani},
year={2020},
eprint={2010.06060},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@article{vaswani2017attention,
title={Attention is all you need},
author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia},
journal={arXiv preprint arXiv:1706.03762},
year={2017}
}
@article{sennrich2015neural,
title={Neural machine translation of rare words with subword units},
author={Sennrich, Rico and Haddow, Barry and Birch, Alexandra},
journal={arXiv preprint arXiv:1508.07909},
year={2015}
}
@article{provilkov2019bpe,
title={Bpe-dropout: Simple and effective subword regularization},
author={Provilkov, Ivan and Emelianenko, Dmitrii and Voita, Elena},
journal={arXiv preprint arXiv:1910.13267},
year={2019}
}
@article{post2018call,
title={A call for clarity in reporting BLEU scores},
author={Post, Matt},
journal={arXiv preprint arXiv:1804.08771},
year={2018}
}
@misc{zhang2021sgdqa,
title={SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services},
author={Yang Zhang and Vahid Noroozi and Evelina Bakhturina and Boris Ginsburg},
year={2021},
eprint={2105.08049},
archivePrefix={arXiv},
primaryClass={cs.CL}
}

Some files were not shown because too many files have changed in this diff Show More