storytold/storyteller-ml
2
0
Fork 0
mirror of https://github.com/storytold/storyteller-ml.git synced 2026-10-17 00:23:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Introduce code from Rust monorepo. This is a mess

Browse Source
This commit is contained in:
Brandon Thomas
2022-02-28 05:43:54 -05:00
parent 921fb49f9a
commit ba326bfcfd
1058 changed files with 192192 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tts/NeMo/tutorials/asr/01_ASR_with_NeMo.ipynb
+1174
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/tutorials/asr/02_Online_ASR_Microphone_Demo.ipynb
+540
View File
@@ -0,0 +1,540 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"Please run notebook locally (if you have all the dependencies and a GPU). \n",
"Technically you can run this notebook on Google Colab but you need to set up microphone for Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"5. Set up microphone for Colab\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell.\n",
"\n",
"## Install dependencies\n",
"!pip install wget\n",
"!apt-get install sox libsndfile1 ffmpeg portaudio19-dev\n",
"!pip install unidecode\n",
"!pip install pyaudio\n",
"\n",
"# ## Install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[asr]\n",
"\n",
"## Install TorchAudio\n",
"!pip install torchaudio>=0.6.0 -f https://download.pytorch.org/whl/torch_stable.html\n",
"\n",
"## Grab the config we'll use in this example\n",
"!mkdir configs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook demonstrates automatic speech recognition (ASR) from a microphone's stream in NeMo.\n",
"\n",
"It is **not a recommended** way to do inference in production workflows. If you are interested in \n",
"production-level inference using NeMo ASR models, please sign-up to Jarvis early access program: https://developer.nvidia.com/nvidia-jarvis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The notebook requires PyAudio library to get a signal from an audio device.\n",
"For Ubuntu, please run the following commands to install it:\n",
"```\n",
"sudo apt-get install -y portaudio19-dev\n",
"pip install pyaudio\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook requires the `torchaudio` library to be installed for MatchboxNet. Please follow the instructions available at the [torchaudio Github page](https://github.com/pytorch/audio#installation) to install the appropriate version of torchaudio.\n",
"\n",
"If you would like to install the latest version, please run the following command to install it:\n",
"\n",
"```\n",
"conda install -c pytorch torchaudio\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pyaudio as pa\n",
"import os, time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import nemo\n",
"import nemo.collections.asr as nemo_asr"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# sample rate, Hz\n",
"SAMPLE_RATE = 16000"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Restore the model from NGC"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained('QuartzNet15x5Base-En')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Observing the config of the model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from omegaconf import OmegaConf\n",
"import copy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Preserve a copy of the full config\n",
"cfg = copy.deepcopy(asr_model._cfg)\n",
"print(OmegaConf.to_yaml(cfg))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Modify preprocessor parameters for inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Make config overwrite-able\n",
"OmegaConf.set_struct(cfg.preprocessor, False)\n",
"\n",
"# some changes for streaming scenario\n",
"cfg.preprocessor.dither = 0.0\n",
"cfg.preprocessor.pad_to = 0\n",
"\n",
"# spectrogram normalization constants\n",
"normalization = {}\n",
"normalization['fixed_mean'] = [\n",
" -14.95827016, -12.71798736, -11.76067913, -10.83311182,\n",
" -10.6746914, -10.15163465, -10.05378331, -9.53918999,\n",
" -9.41858904, -9.23382904, -9.46470918, -9.56037,\n",
" -9.57434245, -9.47498732, -9.7635205, -10.08113074,\n",
" -10.05454561, -9.81112681, -9.68673603, -9.83652977,\n",
" -9.90046248, -9.85404766, -9.92560366, -9.95440354,\n",
" -10.17162966, -9.90102482, -9.47471025, -9.54416855,\n",
" -10.07109475, -9.98249912, -9.74359465, -9.55632283,\n",
" -9.23399915, -9.36487649, -9.81791084, -9.56799225,\n",
" -9.70630899, -9.85148006, -9.8594418, -10.01378735,\n",
" -9.98505315, -9.62016094, -10.342285, -10.41070709,\n",
" -10.10687659, -10.14536695, -10.30828702, -10.23542833,\n",
" -10.88546868, -11.31723646, -11.46087382, -11.54877829,\n",
" -11.62400934, -11.92190509, -12.14063815, -11.65130117,\n",
" -11.58308531, -12.22214663, -12.42927197, -12.58039805,\n",
" -13.10098969, -13.14345864, -13.31835645, -14.47345634]\n",
"normalization['fixed_std'] = [\n",
" 3.81402054, 4.12647781, 4.05007065, 3.87790987,\n",
" 3.74721178, 3.68377423, 3.69344, 3.54001005,\n",
" 3.59530412, 3.63752368, 3.62826417, 3.56488469,\n",
" 3.53740577, 3.68313898, 3.67138151, 3.55707266,\n",
" 3.54919572, 3.55721289, 3.56723346, 3.46029304,\n",
" 3.44119672, 3.49030548, 3.39328435, 3.28244406,\n",
" 3.28001423, 3.26744937, 3.46692348, 3.35378948,\n",
" 2.96330901, 2.97663111, 3.04575148, 2.89717604,\n",
" 2.95659301, 2.90181116, 2.7111687, 2.93041291,\n",
" 2.86647897, 2.73473181, 2.71495654, 2.75543763,\n",
" 2.79174615, 2.96076456, 2.57376336, 2.68789782,\n",
" 2.90930817, 2.90412004, 2.76187531, 2.89905006,\n",
" 2.65896173, 2.81032176, 2.87769857, 2.84665271,\n",
" 2.80863137, 2.80707634, 2.83752184, 3.01914511,\n",
" 2.92046439, 2.78461139, 2.90034605, 2.94599508,\n",
" 2.99099718, 3.0167554, 3.04649716, 2.94116777]\n",
"\n",
"cfg.preprocessor.normalize = normalization\n",
"\n",
"# Disable config overwriting\n",
"OmegaConf.set_struct(cfg.preprocessor, True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup preprocessor with these settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"asr_model.preprocessor = asr_model.from_config_dict(cfg.preprocessor)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set model to inference mode\n",
"asr_model.eval();"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"asr_model = asr_model.to(asr_model.device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up data for Streaming Inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nemo.core.classes import IterableDataset\n",
"from nemo.core.neural_types import NeuralType, AudioSignal, LengthsType\n",
"import torch\n",
"from torch.utils.data import DataLoader"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# simple data layer to pass audio signal\n",
"class AudioDataLayer(IterableDataset):\n",
" @property\n",
" def output_types(self):\n",
" return {\n",
" 'audio_signal': NeuralType(('B', 'T'), AudioSignal(freq=self._sample_rate)),\n",
" 'a_sig_length': NeuralType(tuple('B'), LengthsType()),\n",
" }\n",
"\n",
" def __init__(self, sample_rate):\n",
" super().__init__()\n",
" self._sample_rate = sample_rate\n",
" self.output = True\n",
" \n",
" def __iter__(self):\n",
" return self\n",
" \n",
" def __next__(self):\n",
" if not self.output:\n",
" raise StopIteration\n",
" self.output = False\n",
" return torch.as_tensor(self.signal, dtype=torch.float32), \\\n",
" torch.as_tensor(self.signal_shape, dtype=torch.int64)\n",
" \n",
" def set_signal(self, signal):\n",
" self.signal = signal.astype(np.float32)/32768.\n",
" self.signal_shape = self.signal.size\n",
" self.output = True\n",
"\n",
" def __len__(self):\n",
" return 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_layer = AudioDataLayer(sample_rate=cfg.preprocessor.sample_rate)\n",
"data_loader = DataLoader(data_layer, batch_size=1, collate_fn=data_layer.collate_fn)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# inference method for audio signal (single instance)\n",
"def infer_signal(model, signal):\n",
" data_layer.set_signal(signal)\n",
" batch = next(iter(data_loader))\n",
" audio_signal, audio_signal_len = batch\n",
" audio_signal, audio_signal_len = audio_signal.to(asr_model.device), audio_signal_len.to(asr_model.device)\n",
" log_probs, encoded_len, predictions = model.forward(\n",
" input_signal=audio_signal, input_signal_length=audio_signal_len\n",
" )\n",
" return log_probs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# class for streaming frame-based ASR\n",
"# 1) use reset() method to reset FrameASR's state\n",
"# 2) call transcribe(frame) to do ASR on\n",
"# contiguous signal's frames\n",
"class FrameASR:\n",
" \n",
" def __init__(self, model_definition,\n",
" frame_len=2, frame_overlap=2.5, \n",
" offset=10):\n",
" '''\n",
" Args:\n",
" frame_len: frame's duration, seconds\n",
" frame_overlap: duration of overlaps before and after current frame, seconds\n",
" offset: number of symbols to drop for smooth streaming\n",
" '''\n",
" self.vocab = list(model_definition['labels'])\n",
" self.vocab.append('_')\n",
" \n",
" self.sr = model_definition['sample_rate']\n",
" self.frame_len = frame_len\n",
" self.n_frame_len = int(frame_len * self.sr)\n",
" self.frame_overlap = frame_overlap\n",
" self.n_frame_overlap = int(frame_overlap * self.sr)\n",
" timestep_duration = model_definition['AudioToMelSpectrogramPreprocessor']['window_stride']\n",
" for block in model_definition['JasperEncoder']['jasper']:\n",
" timestep_duration *= block['stride'][0] ** block['repeat']\n",
" self.n_timesteps_overlap = int(frame_overlap / timestep_duration) - 2\n",
" self.buffer = np.zeros(shape=2*self.n_frame_overlap + self.n_frame_len,\n",
" dtype=np.float32)\n",
" self.offset = offset\n",
" self.reset()\n",
" \n",
" def _decode(self, frame, offset=0):\n",
" assert len(frame)==self.n_frame_len\n",
" self.buffer[:-self.n_frame_len] = self.buffer[self.n_frame_len:]\n",
" self.buffer[-self.n_frame_len:] = frame\n",
" logits = infer_signal(asr_model, self.buffer).cpu().numpy()[0]\n",
" # print(logits.shape)\n",
" decoded = self._greedy_decoder(\n",
" logits[self.n_timesteps_overlap:-self.n_timesteps_overlap], \n",
" self.vocab\n",
" )\n",
" return decoded[:len(decoded)-offset]\n",
" \n",
" @torch.no_grad()\n",
" def transcribe(self, frame=None, merge=True):\n",
" if frame is None:\n",
" frame = np.zeros(shape=self.n_frame_len, dtype=np.float32)\n",
" if len(frame) < self.n_frame_len:\n",
" frame = np.pad(frame, [0, self.n_frame_len - len(frame)], 'constant')\n",
" unmerged = self._decode(frame, self.offset)\n",
" if not merge:\n",
" return unmerged\n",
" return self.greedy_merge(unmerged)\n",
" \n",
" def reset(self):\n",
" '''\n",
" Reset frame_history and decoder's state\n",
" '''\n",
" self.buffer=np.zeros(shape=self.buffer.shape, dtype=np.float32)\n",
" self.prev_char = ''\n",
"\n",
" @staticmethod\n",
" def _greedy_decoder(logits, vocab):\n",
" s = ''\n",
" for i in range(logits.shape[0]):\n",
" s += vocab[np.argmax(logits[i])]\n",
" return s\n",
"\n",
" def greedy_merge(self, s):\n",
" s_merged = ''\n",
" \n",
" for i in range(len(s)):\n",
" if s[i] != self.prev_char:\n",
" self.prev_char = s[i]\n",
" if self.prev_char != '_':\n",
" s_merged += self.prev_char\n",
" return s_merged"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Streaming Inference\n",
"\n",
"Streaming inference depends on a few factors, such as the frame length and buffer size. Experiment with a few values to see their effects in the below cells."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# duration of signal frame, seconds\n",
"FRAME_LEN = 1.0\n",
"# number of audio channels (expect mono signal)\n",
"CHANNELS = 1\n",
"\n",
"CHUNK_SIZE = int(FRAME_LEN*SAMPLE_RATE)\n",
"asr = FrameASR(model_definition = {\n",
" 'sample_rate': SAMPLE_RATE,\n",
" 'AudioToMelSpectrogramPreprocessor': cfg.preprocessor,\n",
" 'JasperEncoder': cfg.encoder,\n",
" 'labels': cfg.decoder.vocabulary\n",
" },\n",
" frame_len=FRAME_LEN, frame_overlap=2, \n",
" offset=4)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"asr.reset()\n",
"\n",
"p = pa.PyAudio()\n",
"print('Available audio input devices:')\n",
"input_devices = []\n",
"for i in range(p.get_device_count()):\n",
" dev = p.get_device_info_by_index(i)\n",
" if dev.get('maxInputChannels'):\n",
" input_devices.append(i)\n",
" print(i, dev.get('name'))\n",
"\n",
"if len(input_devices):\n",
" dev_idx = -2\n",
" while dev_idx not in input_devices:\n",
" print('Please type input device ID:')\n",
" dev_idx = int(input())\n",
"\n",
" empty_counter = 0\n",
"\n",
" def callback(in_data, frame_count, time_info, status):\n",
" global empty_counter\n",
" signal = np.frombuffer(in_data, dtype=np.int16)\n",
" text = asr.transcribe(signal)\n",
" if len(text):\n",
" print(text,end='')\n",
" empty_counter = asr.offset\n",
" elif empty_counter > 0:\n",
" empty_counter -= 1\n",
" if empty_counter == 0:\n",
" print(' ',end='')\n",
" return (in_data, pa.paContinue)\n",
"\n",
" stream = p.open(format=pa.paInt16,\n",
" channels=CHANNELS,\n",
" rate=SAMPLE_RATE,\n",
" input=True,\n",
" input_device_index=dev_idx,\n",
" stream_callback=callback,\n",
" frames_per_buffer=CHUNK_SIZE)\n",
"\n",
" print('Listening...')\n",
"\n",
" stream.start_stream()\n",
" \n",
" # Interrupt kernel and then speak for a few more words to exit the pyaudio loop !\n",
" try:\n",
" while stream.is_active():\n",
" time.sleep(0.1)\n",
" finally: \n",
" stream.stop_stream()\n",
" stream.close()\n",
" p.terminate()\n",
"\n",
" print()\n",
" print(\"PyAudio stopped\")\n",
" \n",
"else:\n",
" print('ERROR: No audio input device found.')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
tts/NeMo/tutorials/asr/03_Speech_Commands.ipynb
+1628
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/tutorials/asr/04_Online_Offline_Speech_Commands_Demo.ipynb
+747
View File
@@ -0,0 +1,747 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"\"\"\"\n",
"Please run notebook locally (if you have all the dependencies and a GPU). \n",
"Technically you can run this notebook on Google Colab but you need to set up microphone for Colab.\n",
" \n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"5. Set up microphone for Colab\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell.\n",
"\n",
"## Install dependencies\n",
"!pip install wget\n",
"!apt-get install sox libsndfile1 ffmpeg portaudio19-dev\n",
"!pip install unidecode\n",
"!pip install pyaudio\n",
"\n",
"# ## Install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[asr]\n",
"\n",
"## Install TorchAudio\n",
"!pip install torchaudio>=0.6.0 -f https://download.pytorch.org/whl/torch_stable.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"This notebook demonstrates offline and online (from a microphone's stream in NeMo) speech commands recognition \n",
"\n",
"It is **not a recommended** way to do inference in production workflows. If you are interested in \n",
"production-level inference using NeMo ASR models, please sign-up to Jarvis early access program: https://developer.nvidia.com/nvidia-jarvis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The notebook requires PyAudio library to get a signal from an audio device.\n",
"For Ubuntu, please run the following commands to install it:\n",
"```\n",
"sudo apt-get install -y portaudio19-dev\n",
"pip install pyaudio\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook requires the `torchaudio` library to be installed for MatchboxNet. Please follow the instructions available at the [torchaudio Github page](https://github.com/pytorch/audio#installation) to install the appropriate version of torchaudio.\n",
"\n",
"If you would like to install the latest version, please run the following command to install it:\n",
"\n",
"```\n",
"conda install -c pytorch torchaudio\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pyaudio as pa\n",
"import os, time\n",
"import librosa\n",
"import IPython.display as ipd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"import nemo\n",
"import nemo.collections.asr as nemo_asr"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# sample rate, Hz\n",
"SAMPLE_RATE = 16000"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Restore the model from NGC"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"mbn_model = nemo_asr.models.EncDecClassificationModel.from_pretrained(\"commandrecognition_en_matchboxnet3x1x64_v2\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since speech commands model MatchBoxNet doesn't consider non-speech scenario, \n",
"here we use a Voice Activity Detection (VAD) model to help reduce false alarm for background noise/silence. When there is speech activity detected, the speech command inference will be activated. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please note the VAD model is not perfect for various microphone input and you might need to finetune on your input and play with different parameters.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"vad_model = nemo_asr.models.EncDecClassificationModel.from_pretrained('vad_marblenet')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Observing the config of the model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from omegaconf import OmegaConf\n",
"import copy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Preserve a copy of the full config\n",
"vad_cfg = copy.deepcopy(vad_model._cfg)\n",
"mbn_cfg = copy.deepcopy(mbn_model._cfg)\n",
"print(OmegaConf.to_yaml(mbn_cfg))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What classes can this model recognize?\n",
"\n",
"Before we begin inference on the actual audio stream, let's look at what are the classes this model was trained to recognize. \n",
"\n",
"**MatchBoxNet model is not designed to recognize words out of vocabulary (OOV).**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"labels = mbn_cfg.labels\n",
"for i in range(len(labels)):\n",
" print('%-10s' % (labels[i]), end=' ')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup preprocessor with these settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set model to inference mode\n",
"mbn_model.eval();\n",
"vad_model.eval();"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up data for Streaming Inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nemo.core.classes import IterableDataset\n",
"from nemo.core.neural_types import NeuralType, AudioSignal, LengthsType\n",
"import torch\n",
"from torch.utils.data import DataLoader"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# simple data layer to pass audio signal\n",
"class AudioDataLayer(IterableDataset):\n",
" @property\n",
" def output_types(self):\n",
" return {\n",
" 'audio_signal': NeuralType(('B', 'T'), AudioSignal(freq=self._sample_rate)),\n",
" 'a_sig_length': NeuralType(tuple('B'), LengthsType()),\n",
" }\n",
"\n",
" def __init__(self, sample_rate):\n",
" super().__init__()\n",
" self._sample_rate = sample_rate\n",
" self.output = True\n",
" \n",
" def __iter__(self):\n",
" return self\n",
" \n",
" def __next__(self):\n",
" if not self.output:\n",
" raise StopIteration\n",
" self.output = False\n",
" return torch.as_tensor(self.signal, dtype=torch.float32), \\\n",
" torch.as_tensor(self.signal_shape, dtype=torch.int64)\n",
" \n",
" def set_signal(self, signal):\n",
" self.signal = signal.astype(np.float32)/32768.\n",
" self.signal_shape = self.signal.size\n",
" self.output = True\n",
"\n",
" def __len__(self):\n",
" return 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_layer = AudioDataLayer(sample_rate=mbn_cfg.train_ds.sample_rate)\n",
"data_loader = DataLoader(data_layer, batch_size=1, collate_fn=data_layer.collate_fn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## inference method for audio signal (single instance)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def infer_signal(model, signal):\n",
" data_layer.set_signal(signal)\n",
" batch = next(iter(data_loader))\n",
" audio_signal, audio_signal_len = batch\n",
" audio_signal, audio_signal_len = audio_signal.to(model.device), audio_signal_len.to(model.device)\n",
" logits = model.forward(input_signal=audio_signal, input_signal_length=audio_signal_len)\n",
" return logits"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we don't include postprocessing techniques here. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# class for streaming frame-based ASR\n",
"# 1) use reset() method to reset FrameASR's state\n",
"# 2) call transcribe(frame) to do ASR on\n",
"# contiguous signal's frames\n",
"class FrameASR:\n",
" \n",
" def __init__(self, model_definition,\n",
" frame_len=2, frame_overlap=2.5, \n",
" offset=0):\n",
" '''\n",
" Args:\n",
" frame_len (seconds): Frame's duration\n",
" frame_overlap (seconds): Duration of overlaps before and after current frame.\n",
" offset: Number of symbols to drop for smooth streaming.\n",
" '''\n",
" self.task = model_definition['task']\n",
" self.vocab = list(model_definition['labels'])\n",
" \n",
" self.sr = model_definition['sample_rate']\n",
" self.frame_len = frame_len\n",
" self.n_frame_len = int(frame_len * self.sr)\n",
" self.frame_overlap = frame_overlap\n",
" self.n_frame_overlap = int(frame_overlap * self.sr)\n",
" timestep_duration = model_definition['AudioToMFCCPreprocessor']['window_stride']\n",
" for block in model_definition['JasperEncoder']['jasper']:\n",
" timestep_duration *= block['stride'][0] ** block['repeat']\n",
" self.buffer = np.zeros(shape=2*self.n_frame_overlap + self.n_frame_len,\n",
" dtype=np.float32)\n",
" self.offset = offset\n",
" self.reset()\n",
" \n",
" @torch.no_grad()\n",
" def _decode(self, frame, offset=0):\n",
" assert len(frame)==self.n_frame_len\n",
" self.buffer[:-self.n_frame_len] = self.buffer[self.n_frame_len:]\n",
" self.buffer[-self.n_frame_len:] = frame\n",
"\n",
" if self.task == 'mbn':\n",
" logits = infer_signal(mbn_model, self.buffer).to('cpu').numpy()[0]\n",
" decoded = self._mbn_greedy_decoder(logits, self.vocab)\n",
" \n",
" elif self.task == 'vad':\n",
" logits = infer_signal(vad_model, self.buffer).to('cpu').numpy()[0]\n",
" decoded = self._vad_greedy_decoder(logits, self.vocab)\n",
" \n",
" else:\n",
" raise(\"Task should either be of mbn or vad!\")\n",
" \n",
" return decoded[:len(decoded)-offset]\n",
" \n",
" def transcribe(self, frame=None,merge=False):\n",
" if frame is None:\n",
" frame = np.zeros(shape=self.n_frame_len, dtype=np.float32)\n",
" if len(frame) < self.n_frame_len:\n",
" frame = np.pad(frame, [0, self.n_frame_len - len(frame)], 'constant')\n",
" unmerged = self._decode(frame, self.offset)\n",
" return unmerged\n",
" \n",
" \n",
" def reset(self):\n",
" '''\n",
" Reset frame_history and decoder's state\n",
" '''\n",
" self.buffer=np.zeros(shape=self.buffer.shape, dtype=np.float32)\n",
" self.mbn_s = []\n",
" self.vad_s = []\n",
" \n",
" @staticmethod\n",
" def _mbn_greedy_decoder(logits, vocab):\n",
" mbn_s = []\n",
" if logits.shape[0]:\n",
" class_idx = np.argmax(logits)\n",
" class_label = vocab[class_idx]\n",
" mbn_s.append(class_label) \n",
" return mbn_s\n",
" \n",
" \n",
" @staticmethod\n",
" def _vad_greedy_decoder(logits, vocab):\n",
" vad_s = []\n",
" if logits.shape[0]:\n",
" probs = torch.softmax(torch.as_tensor(logits), dim=-1)\n",
" probas, preds = torch.max(probs, dim=-1)\n",
" vad_s = [preds.item(), str(vocab[preds]), probs[0].item(), probs[1].item(), str(logits)]\n",
" return vad_s\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Streaming Inference"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## offline inference\n",
"Here we show an example of offline streaming inference. you can use your file or download the provided demo audio file. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Streaming inference depends on a few factors, such as the frame length (STEP) and buffer size (WINDOW SIZE). Experiment with a few values to see their effects in the below cells."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"STEP = 0.25\n",
"WINDOW_SIZE = 1.28 # input segment length for NN we used for training"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import wave\n",
"\n",
"def offline_inference(wave_file, STEP = 0.25, WINDOW_SIZE = 0.31):\n",
" \"\"\"\n",
" Arg:\n",
" wav_file: wave file to be performed inference on.\n",
" STEP: infer every STEP seconds \n",
" WINDOW_SIZE : lenght of audio to be sent to NN.\n",
" \"\"\"\n",
" \n",
" FRAME_LEN = STEP \n",
" CHANNELS = 1 # number of audio channels (expect mono signal)\n",
" RATE = SAMPLE_RATE # sample rate, 16000 Hz\n",
" \n",
" CHUNK_SIZE = int(FRAME_LEN * SAMPLE_RATE)\n",
" \n",
" mbn = FrameASR(model_definition = {\n",
" 'task': 'mbn',\n",
" 'sample_rate': SAMPLE_RATE,\n",
" 'AudioToMFCCPreprocessor': mbn_cfg.preprocessor,\n",
" 'JasperEncoder': mbn_cfg.encoder,\n",
" 'labels': mbn_cfg.labels\n",
" },\n",
" frame_len=FRAME_LEN, frame_overlap = (WINDOW_SIZE - FRAME_LEN)/2,\n",
" offset=0)\n",
"\n",
" wf = wave.open(wave_file, 'rb')\n",
" data = wf.readframes(CHUNK_SIZE)\n",
"\n",
" while len(data) > 0:\n",
"\n",
" data = wf.readframes(CHUNK_SIZE)\n",
" signal = np.frombuffer(data, dtype=np.int16)\n",
" mbn_result = mbn.transcribe(signal)\n",
" \n",
" if len(mbn_result):\n",
" print(mbn_result)\n",
" \n",
" mbn.reset()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"demo_wave = 'SpeechCommands_demo.wav'\n",
"if not os.path.exists(demo_wave):\n",
" !wget \"https://dldata-public.s3.us-east-2.amazonaws.com/SpeechCommands_demo.wav\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"wave_file = demo_wave\n",
"\n",
"CHANNELS = 1\n",
"audio, sample_rate = librosa.load(wave_file, sr=SAMPLE_RATE)\n",
"dur = librosa.get_duration(audio)\n",
"print(dur)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"ipd.Audio(audio, rate=sample_rate)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"# Ground-truth is Yes No\n",
"offline_inference(wave_file, STEP, WINDOW_SIZE)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Online inference through microphone"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Please note MatchBoxNet and VAD model are not perfect for various microphone input and you might need to finetune on your input and play with different parameter. \\\n",
"**We also recommend to use a headphone.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vad_threshold = 0.8 \n",
"\n",
"STEP = 0.1 \n",
"WINDOW_SIZE = 0.15\n",
"mbn_WINDOW_SIZE = 1\n",
"\n",
"CHANNELS = 1 \n",
"RATE = SAMPLE_RATE\n",
"FRAME_LEN = STEP # use step of vad inference as frame len\n",
"\n",
"CHUNK_SIZE = int(STEP * RATE)\n",
"vad = FrameASR(model_definition = {\n",
" 'task': 'vad',\n",
" 'sample_rate': SAMPLE_RATE,\n",
" 'AudioToMFCCPreprocessor': vad_cfg.preprocessor,\n",
" 'JasperEncoder': vad_cfg.encoder,\n",
" 'labels': vad_cfg.labels\n",
" },\n",
" frame_len=FRAME_LEN, frame_overlap=(WINDOW_SIZE - FRAME_LEN) / 2, \n",
" offset=0)\n",
"\n",
"mbn = FrameASR(model_definition = {\n",
" 'task': 'mbn',\n",
" 'sample_rate': SAMPLE_RATE,\n",
" 'AudioToMFCCPreprocessor': mbn_cfg.preprocessor,\n",
" 'JasperEncoder': mbn_cfg.encoder,\n",
" 'labels': mbn_cfg.labels\n",
" },\n",
" frame_len=FRAME_LEN, frame_overlap = (mbn_WINDOW_SIZE-FRAME_LEN)/2,\n",
" offset=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vad.reset()\n",
"mbn.reset()\n",
"\n",
"# Setup input device\n",
"p = pa.PyAudio()\n",
"print('Available audio input devices:')\n",
"input_devices = []\n",
"for i in range(p.get_device_count()):\n",
" dev = p.get_device_info_by_index(i)\n",
" if dev.get('maxInputChannels'):\n",
" input_devices.append(i)\n",
" print(i, dev.get('name'))\n",
"\n",
"if len(input_devices):\n",
" dev_idx = -2\n",
" while dev_idx not in input_devices:\n",
" print('Please type input device ID:')\n",
" dev_idx = int(input())\n",
"\n",
" \n",
" def callback(in_data, frame_count, time_info, status):\n",
" \"\"\"\n",
" callback function for streaming audio and performing inference\n",
" \"\"\"\n",
" signal = np.frombuffer(in_data, dtype=np.int16)\n",
" vad_result = vad.transcribe(signal) \n",
" mbn_result = mbn.transcribe(signal) \n",
" \n",
" if len(vad_result):\n",
" # if speech prob is higher than threshold, we decide it contains speech utterance \n",
" # and activate MatchBoxNet \n",
" if vad_result[3] >= vad_threshold: \n",
" print(mbn_result) # print mbn result when speech present\n",
" else:\n",
" print(\"no-speech\")\n",
" return (in_data, pa.paContinue)\n",
"\n",
" # streaming\n",
" stream = p.open(format=pa.paInt16,\n",
" channels=CHANNELS,\n",
" rate=SAMPLE_RATE,\n",
" input=True,\n",
" input_device_index=dev_idx,\n",
" stream_callback=callback,\n",
" frames_per_buffer=CHUNK_SIZE)\n",
"\n",
" \n",
" print('Listening...')\n",
" stream.start_stream()\n",
" \n",
" # Interrupt kernel and then speak for a few more words to exit the pyaudio loop !\n",
" try:\n",
" while stream.is_active():\n",
" time.sleep(0.1)\n",
" finally: \n",
" stream.stop_stream()\n",
" stream.close()\n",
" p.terminate()\n",
" print()\n",
" print(\"PyAudio stopped\")\n",
" \n",
"else:\n",
" print('ERROR: No audio input device found.')"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## ONNX Deployment\n",
"You can also export the model to ONNX file and deploy it to TensorRT or MS ONNX Runtime inference engines. If you don't have one installed yet, please run:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!mkdir -p ort\n",
"%cd ort\n",
"!git clone --depth 1 --branch v1.5.1 https://github.com/microsoft/onnxruntime.git .\n",
"!./build.sh --skip_tests --config Release --build_shared_lib --parallel --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu --build_wheel\n",
"!pip install ./build/Linux/Release/dist/onnxruntime*.whl\n",
"%cd .."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then just replace `infer_signal` implementation with this code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import onnxruntime\n",
"mbn_model.export('mbn.onnx')\n",
"ort_session = onnxruntime.InferenceSession('mbn.onnx')\n",
"\n",
"def to_numpy(tensor):\n",
" return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()\n",
"\n",
"def infer_signal(signal):\n",
" data_layer.set_signal(signal)\n",
" batch = next(iter(data_loader))\n",
" audio_signal, audio_signal_len = batch\n",
" audio_signal, audio_signal_len = audio_signal.to(mbn_model.device), audio_signal_len.to(mbn_model.device)\n",
" processed_signal, processed_signal_len = mbn_model.preprocessor(\n",
" input_signal=audio_signal, length=audio_signal_len,\n",
" )\n",
" ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(processed_signal), }\n",
" ologits = ort_session.run(None, ort_inputs)\n",
" alogits = np.asarray(ologits)\n",
" logits = torch.from_numpy(alogits[0])\n",
" return logits"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
tts/NeMo/tutorials/asr/05_Online_Noise_Augmentation.ipynb
+1320
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/tutorials/asr/06_Voice_Activiy_Detection.ipynb
+1169
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb
+830
View File
@@ -0,0 +1,830 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"Please run notebook locally (if you have all the dependencies and a GPU). \n",
"Technically you can run this notebook on Google Colab but you need to set up microphone for Colab.\n",
" \n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"5. Set up microphone for Colab\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell.\n",
"\n",
"## Install dependencies\n",
"!pip install wget\n",
"!apt-get install sox libsndfile1 ffmpeg portaudio19-dev\n",
"!pip install unidecode\n",
"!pip install pyaudio\n",
"\n",
"# ## Install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[asr]\n",
"\n",
"## Install TorchAudio\n",
"!pip install torchaudio>=0.6.0 -f https://download.pytorch.org/whl/torch_stable.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Voice Activity Detection (VAD)\n",
"\n",
"\n",
"This notebook demonstrates how to perform\n",
"1. [offline streaming inference on audio files (offline VAD)](#Offline-streaming-inference);\n",
"2. [finetuning](#Finetune) and use [posterior](#Posterior);\n",
"2. [threshold tuning](#Tuning-threshold);\n",
"4. [online streaming inference](#Online-streaming-inference);\n",
"3. [online streaming inference from a microphone's stream](#Online-streaming-inference-through-microphone).\n",
"\n",
"\n",
"It is **not a recommended** way to do inference in production workflows. If you are interested in \n",
"production-level inference using NeMo ASR models, please sign-up to Jarvis early access program: https://developer.nvidia.com/nvidia-jarvis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The notebook requires PyAudio library to get a signal from an audio device.\n",
"For Ubuntu, please run the following commands to install it:\n",
"```\n",
"sudo apt-get install -y portaudio19-dev\n",
"pip install pyaudio\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook requires the `torchaudio` library to be installed for MarbleNet. Please follow the instructions available at the [torchaudio Github page](https://github.com/pytorch/audio#installation) to install the appropriate version of torchaudio.\n",
"\n",
"If you would like to install the latest version, please run the following command to install it:\n",
"\n",
"```\n",
"conda install -c pytorch torchaudio\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pyaudio as pa\n",
"import os, time\n",
"import librosa\n",
"import IPython.display as ipd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"import nemo\n",
"import nemo.collections.asr as nemo_asr"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# sample rate, Hz\n",
"SAMPLE_RATE = 16000"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Restore the model from NGC"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"vad_model = nemo_asr.models.EncDecClassificationModel.from_pretrained('vad_marblenet')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Observing the config of the model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from omegaconf import OmegaConf\n",
"import copy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Preserve a copy of the full config\n",
"cfg = copy.deepcopy(vad_model._cfg)\n",
"print(OmegaConf.to_yaml(cfg))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup preprocessor with these settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vad_model.preprocessor = vad_model.from_config_dict(cfg.preprocessor)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set model to inference mode\n",
"vad_model.eval();"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vad_model = vad_model.to(vad_model.device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We demonstrate two methods for streaming inference:\n",
"1. [offline streaming inference (script)](#Offline-streaming-inference)\n",
"2. [online streaming inference (step-by-step)](#Online-streaming-inference)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Offline streaming inference\n",
"\n",
"VAD relies on shorter fixed-length segments for prediction. Duration inference, we generate frame-level prediction by two approaches:\n",
"\n",
"1. shift the window of length `time_length` (e.g. 0.63s) by `shift_length` (e.g. 10ms) to generate the frame and use the prediction of the window to represent the label for the frame; Use \n",
"```python\n",
" <NeMo_git_root>/examples/asr/vad_infer.py\n",
"```\n",
"\n",
" This script will automatically split long audio file to avoid CUDA memory issue and performing **streaming** inside `AudioLabelDataset`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Posterior\n",
"<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/v1.0.2/tutorials/asr/images/vad_post_overlap_diagram.png\" width=\"500\">\n",
"\n",
"2. generate predictions with overlapping input segments. Then a smoothing filter is applied to decide the label for a frame spanned by multiple segments. Get frame level prediction from above step and use \n",
"```python\n",
"<NeMo_git_root>/scripts/voice_activity_detection/vad_overlap_posterior.py\n",
"```\n",
"\n",
"Have a look at [MarbleNet paper](https://arxiv.org/pdf/2010.13886.pdf) for choices about segment length, smoothing filter, etc. And play with those parameters with your data.\n",
"\n",
"You can also find posterior about converting frame level prediction to speech/no-speech segment in start and end times format in `vad_overlap_posterior.py`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Finetune\n",
"You might need to finetune on your data for better performance. For finetuning/transfer learning, please refer to [**Transfer learning** part of ASR tutorial](https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tuning threshold"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"Of course you can do threshold tuning on frame level prediction. We also provide a script \n",
"```python\n",
"<NeMo_git_root>/scripts/voice_activity_detection/vad_tune_threshold.py\n",
"```\n",
"\n",
"to help you find best threshold if you have ground truth label file in RTTM format. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Online streaming inference"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up data for Streaming Inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nemo.core.classes import IterableDataset\n",
"from nemo.core.neural_types import NeuralType, AudioSignal, LengthsType\n",
"import torch\n",
"from torch.utils.data import DataLoader"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# simple data layer to pass audio signal\n",
"class AudioDataLayer(IterableDataset):\n",
" @property\n",
" def output_types(self):\n",
" return {\n",
" 'audio_signal': NeuralType(('B', 'T'), AudioSignal(freq=self._sample_rate)),\n",
" 'a_sig_length': NeuralType(tuple('B'), LengthsType()),\n",
" }\n",
"\n",
" def __init__(self, sample_rate):\n",
" super().__init__()\n",
" self._sample_rate = sample_rate\n",
" self.output = True\n",
" \n",
" def __iter__(self):\n",
" return self\n",
" \n",
" def __next__(self):\n",
" if not self.output:\n",
" raise StopIteration\n",
" self.output = False\n",
" return torch.as_tensor(self.signal, dtype=torch.float32), \\\n",
" torch.as_tensor(self.signal_shape, dtype=torch.int64)\n",
" \n",
" def set_signal(self, signal):\n",
" self.signal = signal.astype(np.float32)/32768.\n",
" self.signal_shape = self.signal.size\n",
" self.output = True\n",
"\n",
" def __len__(self):\n",
" return 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_layer = AudioDataLayer(sample_rate=cfg.train_ds.sample_rate)\n",
"data_loader = DataLoader(data_layer, batch_size=1, collate_fn=data_layer.collate_fn)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# inference method for audio signal (single instance)\n",
"def infer_signal(model, signal):\n",
" data_layer.set_signal(signal)\n",
" batch = next(iter(data_loader))\n",
" audio_signal, audio_signal_len = batch\n",
" audio_signal, audio_signal_len = audio_signal.to(vad_model.device), audio_signal_len.to(vad_model.device)\n",
" logits = model.forward(input_signal=audio_signal, input_signal_length=audio_signal_len)\n",
" return logits"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# class for streaming frame-based VAD\n",
"# 1) use reset() method to reset FrameVAD's state\n",
"# 2) call transcribe(frame) to do VAD on\n",
"# contiguous signal's frames\n",
"class FrameVAD:\n",
" \n",
" def __init__(self, model_definition,\n",
" threshold=0.5,\n",
" frame_len=2, frame_overlap=2.5, \n",
" offset=10):\n",
" '''\n",
" Args:\n",
" threshold: If prob of speech is larger than threshold, classify the segment to be speech.\n",
" frame_len: frame's duration, seconds\n",
" frame_overlap: duration of overlaps before and after current frame, seconds\n",
" offset: number of symbols to drop for smooth streaming\n",
" '''\n",
" self.vocab = list(model_definition['labels'])\n",
" self.vocab.append('_')\n",
" \n",
" self.sr = model_definition['sample_rate']\n",
" self.threshold = threshold\n",
" self.frame_len = frame_len\n",
" self.n_frame_len = int(frame_len * self.sr)\n",
" self.frame_overlap = frame_overlap\n",
" self.n_frame_overlap = int(frame_overlap * self.sr)\n",
" timestep_duration = model_definition['AudioToMFCCPreprocessor']['window_stride']\n",
" for block in model_definition['JasperEncoder']['jasper']:\n",
" timestep_duration *= block['stride'][0] ** block['repeat']\n",
" self.buffer = np.zeros(shape=2*self.n_frame_overlap + self.n_frame_len,\n",
" dtype=np.float32)\n",
" self.offset = offset\n",
" self.reset()\n",
" \n",
" def _decode(self, frame, offset=0):\n",
" assert len(frame)==self.n_frame_len\n",
" self.buffer[:-self.n_frame_len] = self.buffer[self.n_frame_len:]\n",
" self.buffer[-self.n_frame_len:] = frame\n",
" logits = infer_signal(vad_model, self.buffer).cpu().numpy()[0]\n",
" decoded = self._greedy_decoder(\n",
" self.threshold,\n",
" logits,\n",
" self.vocab\n",
" )\n",
" return decoded \n",
" \n",
" \n",
" @torch.no_grad()\n",
" def transcribe(self, frame=None):\n",
" if frame is None:\n",
" frame = np.zeros(shape=self.n_frame_len, dtype=np.float32)\n",
" if len(frame) < self.n_frame_len:\n",
" frame = np.pad(frame, [0, self.n_frame_len - len(frame)], 'constant')\n",
" unmerged = self._decode(frame, self.offset)\n",
" return unmerged\n",
" \n",
" def reset(self):\n",
" '''\n",
" Reset frame_history and decoder's state\n",
" '''\n",
" self.buffer=np.zeros(shape=self.buffer.shape, dtype=np.float32)\n",
" self.prev_char = ''\n",
"\n",
" @staticmethod\n",
" def _greedy_decoder(threshold, logits, vocab):\n",
" s = []\n",
" if logits.shape[0]:\n",
" probs = torch.softmax(torch.as_tensor(logits), dim=-1)\n",
" probas, _ = torch.max(probs, dim=-1)\n",
" probas_s = probs[1].item()\n",
" preds = 1 if probas_s >= threshold else 0\n",
" s = [preds, str(vocab[preds]), probs[0].item(), probs[1].item(), str(logits)]\n",
" return s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"Streaming inference depends on a few factors, such as the frame length (STEP) and buffer size (WINDOW SIZE). Experiment with a few values to see their effects in the below cells."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"STEP_LIST = [0.01,0.01]\n",
"WINDOW_SIZE_LIST = [0.31,0.15]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import wave\n",
"\n",
"def offline_inference(wave_file, STEP = 0.025, WINDOW_SIZE = 0.5, threshold=0.5):\n",
" \n",
" FRAME_LEN = STEP # infer every STEP seconds \n",
" CHANNELS = 1 # number of audio channels (expect mono signal)\n",
" RATE = 16000 # sample rate, Hz\n",
" \n",
" \n",
" CHUNK_SIZE = int(FRAME_LEN*RATE)\n",
" \n",
" vad = FrameVAD(model_definition = {\n",
" 'sample_rate': SAMPLE_RATE,\n",
" 'AudioToMFCCPreprocessor': cfg.preprocessor,\n",
" 'JasperEncoder': cfg.encoder,\n",
" 'labels': cfg.labels\n",
" },\n",
" threshold=threshold,\n",
" frame_len=FRAME_LEN, frame_overlap = (WINDOW_SIZE-FRAME_LEN)/2,\n",
" offset=0)\n",
"\n",
" wf = wave.open(wave_file, 'rb')\n",
" p = pa.PyAudio()\n",
"\n",
" empty_counter = 0\n",
"\n",
" preds = []\n",
" proba_b = []\n",
" proba_s = []\n",
" \n",
" stream = p.open(format=p.get_format_from_width(wf.getsampwidth()),\n",
" channels=CHANNELS,\n",
" rate=RATE,\n",
" output = True)\n",
"\n",
" data = wf.readframes(CHUNK_SIZE)\n",
"\n",
" while len(data) > 0:\n",
"\n",
" data = wf.readframes(CHUNK_SIZE)\n",
" signal = np.frombuffer(data, dtype=np.int16)\n",
" result = vad.transcribe(signal)\n",
"\n",
" preds.append(result[0])\n",
" proba_b.append(result[2])\n",
" proba_s.append(result[3])\n",
" \n",
" if len(result):\n",
" print(result,end='\\n')\n",
" empty_counter = 3\n",
" elif empty_counter > 0:\n",
" empty_counter -= 1\n",
" if empty_counter == 0:\n",
" print(' ',end='')\n",
" \n",
" p.terminate()\n",
" vad.reset()\n",
" \n",
" return preds, proba_b, proba_s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Here we show an example of online streaming inference\n",
"You can use your file or download the provided demo audio file. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"demo_wave = 'VAD_demo.wav'\n",
"if not os.path.exists(demo_wave):\n",
" !wget \"https://dldata-public.s3.us-east-2.amazonaws.com/VAD_demo.wav\" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"wave_file = demo_wave\n",
"\n",
"CHANNELS = 1\n",
"RATE = 16000\n",
"audio, sample_rate = librosa.load(wave_file, sr=RATE)\n",
"dur = librosa.get_duration(audio)\n",
"print(dur)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"ipd.Audio(audio, rate=sample_rate)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"threshold=0.4\n",
"\n",
"results = []\n",
"for STEP, WINDOW_SIZE in zip(STEP_LIST, WINDOW_SIZE_LIST, ):\n",
" print(f'====== STEP is {STEP}s, WINDOW_SIZE is {WINDOW_SIZE}s ====== ')\n",
" preds, proba_b, proba_s = offline_inference(wave_file, STEP, WINDOW_SIZE, threshold)\n",
" results.append([STEP, WINDOW_SIZE, preds, proba_b, proba_s])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above prediction is based on `threshold=0.4`.\n",
"\n",
"You can play with other [threshold](#Tuning-threshold) and see how they would impact performance. \n",
"\n",
"**Note** if you want better performance, [finetune](#Finetune) on your data and use posteriors such as [overlapped prediction](#Posterior). \n",
"\n",
"Let's plot the prediction and melspectrogram"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import librosa.display\n",
"plt.figure(figsize=[20,10])\n",
"\n",
"num = len(results)\n",
"for i in range(num):\n",
" len_pred = len(results[i][2]) \n",
" FRAME_LEN = results[i][0]\n",
" ax1 = plt.subplot(num+1,1,i+1)\n",
"\n",
" ax1.plot(np.arange(audio.size) / sample_rate, audio, 'b')\n",
" ax1.set_xlim([-0.01, int(dur)+1]) \n",
" ax1.tick_params(axis='y', labelcolor= 'b')\n",
" ax1.set_ylabel('Signal')\n",
" ax1.set_ylim([-1, 1])\n",
"\n",
" proba_s = results[i][4]\n",
" pred = [1 if p > threshold else 0 for p in proba_s]\n",
" ax2 = ax1.twinx()\n",
" ax2.plot(np.arange(len_pred)/(1/results[i][0]), np.array(pred) , 'r', label='pred')\n",
" ax2.plot(np.arange(len_pred)/(1/results[i][0]), np.array(proba_s) , 'g--', label='speech prob')\n",
" ax2.tick_params(axis='y', labelcolor='r')\n",
" legend = ax2.legend(loc='lower right', shadow=True)\n",
" ax1.set_ylabel('prediction')\n",
"\n",
" ax2.set_title(f'step {results[i][0]}s, buffer size {results[i][1]}s')\n",
" ax2.set_ylabel('Preds and Probas')\n",
" \n",
" \n",
"ax = plt.subplot(num+1,1,i+2)\n",
"S = librosa.feature.melspectrogram(y=audio, sr=sample_rate, n_mels=64, fmax=8000)\n",
"S_dB = librosa.power_to_db(S, ref=np.max)\n",
"librosa.display.specshow(S_dB, x_axis='time', y_axis='mel', sr=sample_rate, fmax=8000)\n",
"ax.set_title('Mel-frequency spectrogram')\n",
"ax.grid()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Online streaming inference through microphone"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please note the VAD model is not perfect for various microphone input and you might need to finetune on your input and play with different parameters.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"STEP = 0.01 \n",
"WINDOW_SIZE = 0.31\n",
"CHANNELS = 1 \n",
"RATE = 16000\n",
"FRAME_LEN = STEP\n",
"THRESHOLD = 0.5\n",
"\n",
"CHUNK_SIZE = int(STEP * RATE)\n",
"vad = FrameVAD(model_definition = {\n",
" 'sample_rate': SAMPLE_RATE,\n",
" 'AudioToMFCCPreprocessor': cfg.preprocessor,\n",
" 'JasperEncoder': cfg.encoder,\n",
" 'labels': cfg.labels\n",
" },\n",
" threshold=THRESHOLD,\n",
" frame_len=FRAME_LEN, frame_overlap=(WINDOW_SIZE - FRAME_LEN) / 2, \n",
" offset=0)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vad.reset()\n",
"\n",
"p = pa.PyAudio()\n",
"print('Available audio input devices:')\n",
"input_devices = []\n",
"for i in range(p.get_device_count()):\n",
" dev = p.get_device_info_by_index(i)\n",
" if dev.get('maxInputChannels'):\n",
" input_devices.append(i)\n",
" print(i, dev.get('name'))\n",
"\n",
"if len(input_devices):\n",
" dev_idx = -2\n",
" while dev_idx not in input_devices:\n",
" print('Please type input device ID:')\n",
" dev_idx = int(input())\n",
"\n",
" empty_counter = 0\n",
"\n",
" def callback(in_data, frame_count, time_info, status):\n",
" global empty_counter\n",
" signal = np.frombuffer(in_data, dtype=np.int16)\n",
" text = vad.transcribe(signal)\n",
" if len(text):\n",
" print(text,end='\\n')\n",
" empty_counter = vad.offset\n",
" elif empty_counter > 0:\n",
" empty_counter -= 1\n",
" if empty_counter == 0:\n",
" print(' ',end='\\n')\n",
" return (in_data, pa.paContinue)\n",
"\n",
" stream = p.open(format=pa.paInt16,\n",
" channels=CHANNELS,\n",
" rate=SAMPLE_RATE,\n",
" input=True,\n",
" input_device_index=dev_idx,\n",
" stream_callback=callback,\n",
" frames_per_buffer=CHUNK_SIZE)\n",
"\n",
" print('Listening...')\n",
"\n",
" stream.start_stream()\n",
" \n",
" # Interrupt kernel and then speak for a few more words to exit the pyaudio loop !\n",
" try:\n",
" while stream.is_active():\n",
" time.sleep(0.1)\n",
" finally: \n",
" stream.stop_stream()\n",
" stream.close()\n",
" p.terminate()\n",
"\n",
" print()\n",
" print(\"PyAudio stopped\")\n",
" \n",
"else:\n",
" print('ERROR: No audio input device found.')"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## ONNX Deployment\n",
"You can also export the model to ONNX file and deploy it to TensorRT or MS ONNX Runtime inference engines. If you don't have one installed yet, please run:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!mkdir -p ort\n",
"%cd ort\n",
"!git clone --depth 1 --branch v1.5.1 https://github.com/microsoft/onnxruntime.git .\n",
"!./build.sh --skip_tests --config Release --build_shared_lib --parallel --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu --build_wheel\n",
"!pip install ./build/Linux/Release/dist/onnxruntime*.whl\n",
"%cd .."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then just replace `infer_signal` implementation with this code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import onnxruntime\n",
"vad_model.export('vad.onnx')\n",
"ort_session = onnxruntime.InferenceSession('vad.onnx')\n",
"\n",
"def to_numpy(tensor):\n",
" return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()\n",
"\n",
"def infer_signal(signal):\n",
" data_layer.set_signal(signal)\n",
" batch = next(iter(data_loader))\n",
" audio_signal, audio_signal_len = batch\n",
" audio_signal, audio_signal_len = audio_signal.to(vad_model.device), audio_signal_len.to(vad_model.device)\n",
" processed_signal, processed_signal_len = vad_model.preprocessor(\n",
" input_signal=audio_signal, length=audio_signal_len,\n",
" )\n",
" ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(processed_signal), }\n",
" ologits = ort_session.run(None, ort_inputs)\n",
" alogits = np.asarray(ologits)\n",
" logits = torch.from_numpy(alogits[0])\n",
" return logits"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
tts/NeMo/tutorials/asr/08_ASR_with_Subword_Tokenization.ipynb
+1445
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/tutorials/asr/10_ASR_CTC_Language_Finetuning.ipynb
+2163
View File
File diff suppressed because it is too large Load Diff
tts/NeMo/tutorials/asr/Offline_ASR.ipynb
+506
View File
@@ -0,0 +1,506 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Offline_ASR.ipynb",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "_wIWPxBVc3_O"
},
"source": [
"# NeMo offline ASR\n",
"\n",
"This notebook demonstrates how to \n",
"\n",
"* transcribe an audio file (offline ASR) with greedy decoder\n",
"* extract timestamps information from the model to split audio into separate words\n",
"* use beam search decoder with N-gram language model re-scoring\n",
"\n",
"You may find more info on how to train and use language models for ASR models here:\n",
"https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gzcsqceVdtj3"
},
"source": [
"## Installation\n",
"NeMo can be installed via simple pip command. \n",
"\n",
"Optional CTC beam search decoder might require restart of Colab runtime after installation."
]
},
{
"cell_type": "code",
"metadata": {
"id": "I9eIxAyKHREB"
},
"source": [
"BRANCH = 'v1.0.2'\n",
"try:\n",
" # Import NeMo Speech Recognition collection\n",
" import nemo.collections.asr as nemo_asr\n",
"except ModuleNotFoundError:\n",
" !python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]\n",
"\n",
"# check if we have optional Plotly for visualization\n",
"try:\n",
" from plotly import graph_objects as go\n",
"except ModuleNotFoundError:\n",
" !pip install plotly\n",
" from plotly import graph_objects as go\n",
"\n",
"# check if we have optional ipywidgets for tqdm/notebook\n",
"try:\n",
" import ipywidgets\n",
"except ModuleNotFoundError:\n",
" !pip install ipywidgets\n",
"\n",
"# check if CTC beam decoders are installed\n",
"try:\n",
" import ctc_decoders\n",
"except ModuleNotFoundError:\n",
" # install beam search decoder\n",
" !apt-get install -y swig\n",
" !git clone https://github.com/NVIDIA/NeMo -b \"$BRANCH\"\n",
" !cd NeMo && bash scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh\n",
" print('Restarting Colab runtime to successfully import built module.')\n",
" print('Please re-run the notebook.')\n",
" import os\n",
" os.kill(os.getpid(), 9)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "-X2OyAxreGfl"
},
"source": [
"import numpy as np\n",
"# Import audio processing library\n",
"import librosa\n",
"# We'll use this to listen to audio\n",
"from IPython.display import Audio, display"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "zodyzdyTVXas"
},
"source": [
"## Instantiate pre-trained NeMo model\n",
"``from_pretrained(...)`` API downloads and initializes model directly from the cloud. \n",
"\n",
"Alternatively, ``restore_from(...)`` allows loading a model from a disk.\n",
"\n",
"To display available pre-trained models from the cloud, please use ``list_available_models()`` method."
]
},
{
"cell_type": "code",
"metadata": {
"id": "f_J9cuU1H6Bn"
},
"source": [
"nemo_asr.models.EncDecCTCModel.list_available_models()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "x2LMVI9qqtEV"
},
"source": [
"Let's load a base English QuartzNet15x5 model."
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZhWmR7lbvwSm"
},
"source": [
"asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name='QuartzNet15x5Base-En', strict=False)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "HESTZmIzzCEj"
},
"source": [
"## Get test audio clip"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QPWn89l-zLXo"
},
"source": [
"Let's download and analyze a test audio signal."
]
},
{
"cell_type": "code",
"metadata": {
"id": "02gDfK7czSVV"
},
"source": [
"# Download audio sample which we'll try\n",
"# This is a sample from LibriSpeech dev clean subset - the model hasn't seen it before\n",
"AUDIO_FILENAME = '1919-142785-0028.wav'\n",
"!wget https://dldata-public.s3.us-east-2.amazonaws.com/1919-142785-0028.wav\n",
"\n",
"# load audio signal with librosa\n",
"signal, sample_rate = librosa.load(AUDIO_FILENAME, sr=None)\n",
"\n",
"# display audio player for the signal\n",
"display(Audio(data=signal, rate=sample_rate))\n",
"\n",
"# plot the signal in time domain\n",
"fig_signal = go.Figure(\n",
" go.Scatter(x=np.arange(signal.shape[0])/sample_rate,\n",
" y=signal, line={'color': 'green'},\n",
" name='Waveform',\n",
" hovertemplate='Time: %{x:.2f} s<br>Amplitude: %{y:.2f}<br><extra></extra>'),\n",
" layout={\n",
" 'height': 300,\n",
" 'xaxis': {'title': 'Time, s'},\n",
" 'yaxis': {'title': 'Amplitude'},\n",
" 'title': 'Audio Signal',\n",
" 'margin': dict(l=0, r=0, t=40, b=0, pad=0),\n",
" }\n",
")\n",
"fig_signal.show()\n",
"\n",
"# calculate amplitude spectrum\n",
"time_stride=0.01\n",
"hop_length = int(sample_rate*time_stride)\n",
"n_fft = 512\n",
"# linear scale spectrogram\n",
"s = librosa.stft(y=signal,\n",
" n_fft=n_fft,\n",
" hop_length=hop_length)\n",
"s_db = librosa.power_to_db(np.abs(s)**2, ref=np.max, top_db=100)\n",
"\n",
"# plot the signal in frequency domain\n",
"fig_spectrum = go.Figure(\n",
" go.Heatmap(z=s_db,\n",
" colorscale=[\n",
" [0, 'rgb(30,62,62)'],\n",
" [0.5, 'rgb(30,128,128)'],\n",
" [1, 'rgb(30,255,30)'],\n",
" ],\n",
" colorbar=dict(\n",
" ticksuffix=' dB'\n",
" ),\n",
" dx=time_stride, dy=sample_rate/n_fft/1000,\n",
" name='Spectrogram',\n",
" hovertemplate='Time: %{x:.2f} s<br>Frequency: %{y:.2f} kHz<br>Magnitude: %{z:.2f} dB<extra></extra>'),\n",
" layout={\n",
" 'height': 300,\n",
" 'xaxis': {'title': 'Time, s'},\n",
" 'yaxis': {'title': 'Frequency, kHz'},\n",
" 'title': 'Spectrogram',\n",
" 'margin': dict(l=0, r=0, t=40, b=0, pad=0),\n",
" }\n",
")\n",
"fig_spectrum.show()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "jQSj-IhEhrtI"
},
"source": [
"## Offline inference\n",
"If we have an entire audio clip available, then we can do offline inference with a pre-trained model to transcribe it.\n",
"\n",
"The easiest way to do it is to call ASR model's ``transcribe(...)`` method that allows transcribing multiple files in a batch."
]
},
{
"cell_type": "code",
"metadata": {
"id": "s0ERrXIzKpwu"
},
"source": [
"# Convert our audio sample to text\n",
"files = [AUDIO_FILENAME]\n",
"transcript = asr_model.transcribe(paths2audio_files=files)[0]\n",
"print(f'Transcript: \"{transcript}\"')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "_UOoj-WfQoL_"
},
"source": [
"## Extract timestamps and split words\n",
"``transcribe()`` generates a text applying a CTC greedy decoder to raw probabilities distribution over alphabet's characters from ASR model. We can get those raw probabilities with ``logprobs=True`` argument."
]
},
{
"cell_type": "code",
"metadata": {
"id": "-0Sk0C9-LmAR"
},
"source": [
"# softmax implementation in NumPy\n",
"def softmax(logits):\n",
" e = np.exp(logits - np.max(logits))\n",
" return e / e.sum(axis=-1).reshape([logits.shape[0], 1])\n",
"\n",
"# let's do inference once again but without decoder\n",
"logits = asr_model.transcribe(files, logprobs=True)[0]\n",
"probs = softmax(logits)\n",
"\n",
"# 20ms is duration of a timestep at output of the model\n",
"time_stride = 0.02\n",
"\n",
"# get model's alphabet\n",
"labels = list(asr_model.decoder.vocabulary) + ['blank']\n",
"labels[0] = 'space'\n",
"\n",
"# plot probability distribution over characters for each timestep\n",
"fig_probs = go.Figure(\n",
" go.Heatmap(z=probs.transpose(),\n",
" colorscale=[\n",
" [0, 'rgb(30,62,62)'],\n",
" [1, 'rgb(30,255,30)'],\n",
" ],\n",
" y=labels,\n",
" dx=time_stride,\n",
" name='Probs',\n",
" hovertemplate='Time: %{x:.2f} s<br>Character: %{y}<br>Probability: %{z:.2f}<extra></extra>'),\n",
" layout={\n",
" 'height': 300,\n",
" 'xaxis': {'title': 'Time, s'},\n",
" 'yaxis': {'title': 'Characters'},\n",
" 'title': 'Character Probabilities',\n",
" 'margin': dict(l=0, r=0, t=40, b=0, pad=0),\n",
" }\n",
")\n",
"fig_probs.show()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "YiNMZBodIaSP"
},
"source": [
"It is easy to identify timesteps for space character."
]
},
{
"cell_type": "code",
"metadata": {
"id": "32aaW3HEJ89l"
},
"source": [
"# get timestamps for space symbols\n",
"spaces = []\n",
"\n",
"state = ''\n",
"idx_state = 0\n",
"\n",
"if np.argmax(probs[0]) == 0:\n",
" state = 'space'\n",
"\n",
"for idx in range(1, probs.shape[0]):\n",
" current_char_idx = np.argmax(probs[idx])\n",
" if state == 'space' and current_char_idx != 0 and current_char_idx != 28:\n",
" spaces.append([idx_state, idx-1])\n",
" state = ''\n",
" if state == '':\n",
" if current_char_idx == 0:\n",
" state = 'space'\n",
" idx_state = idx\n",
"\n",
"if state == 'space':\n",
" spaces.append([idx_state, len(pred)-1])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "rqg4oxpsL8cW"
},
"source": [
"Then we can split original audio signal into separate words. It is worth to mention that all timestamps have a delay (or an offset) depending on the model. We need to take it into account for alignment."
]
},
{
"cell_type": "code",
"metadata": {
"id": "a-LSg9dSL_O1"
},
"source": [
"# calibration offset for timestamps: 180 ms\n",
"offset = -0.18\n",
"\n",
"# split the transcript into words\n",
"words = transcript.split()\n",
"\n",
"# cut words\n",
"pos_prev = 0\n",
"for j, spot in enumerate(spaces):\n",
" display(words[j])\n",
" pos_end = offset + (spot[0]+spot[1])/2*time_stride\n",
" display(Audio(signal[int(pos_prev*sample_rate):int(pos_end*sample_rate)],\n",
" rate=sample_rate))\n",
" pos_prev = pos_end\n",
"\n",
"display(words[j+1])\n",
"display(Audio(signal[int(pos_prev*sample_rate):],\n",
" rate=sample_rate))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q8Jvwe4Ahncx"
},
"source": [
"## Offline inference with beam search decoder and N-gram language model re-scoring\n",
"\n",
"It is possible to use an external [KenLM](https://kheafield.com/code/kenlm/)-based N-gram language model to rescore multiple transcription candidates. \n",
"\n",
"Let's download and preprocess LibriSpeech 3-gram language model."
]
},
{
"cell_type": "code",
"metadata": {
"id": "EIh8wTVs5uH7"
},
"source": [
"import gzip\n",
"import os, shutil, wget\n",
"\n",
"lm_gzip_path = '3-gram.pruned.1e-7.arpa.gz'\n",
"if not os.path.exists(lm_gzip_path):\n",
" print('Downloading pruned 3-gram model.')\n",
" lm_url = 'http://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz'\n",
" lm_gzip_path = wget.download(lm_url)\n",
" print('Downloaded the 3-gram language model.')\n",
"else:\n",
" print('Pruned .arpa.gz already exists.')\n",
"\n",
"uppercase_lm_path = '3-gram.pruned.1e-7.arpa'\n",
"if not os.path.exists(uppercase_lm_path):\n",
" with gzip.open(lm_gzip_path, 'rb') as f_zipped:\n",
" with open(uppercase_lm_path, 'wb') as f_unzipped:\n",
" shutil.copyfileobj(f_zipped, f_unzipped)\n",
" print('Unzipped the 3-gram language model.')\n",
"else:\n",
" print('Unzipped .arpa already exists.')\n",
"\n",
"lm_path = 'lowercase_3-gram.pruned.1e-7.arpa'\n",
"if not os.path.exists(lm_path):\n",
" with open(uppercase_lm_path, 'r') as f_upper:\n",
" with open(lm_path, 'w') as f_lower:\n",
" for line in f_upper:\n",
" f_lower.write(line.lower())\n",
"print('Converted language model file to lowercase.')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fLDbUkzzUAqW"
},
"source": [
"Let's instantiate ``BeamSearchDecoderWithLM`` module."
]
},
{
"cell_type": "code",
"metadata": {
"id": "_qgKa9L954bJ"
},
"source": [
"beam_search_lm = nemo_asr.modules.BeamSearchDecoderWithLM(\n",
" vocab=list(asr_model.decoder.vocabulary),\n",
" beam_width=16,\n",
" alpha=2, beta=1.5,\n",
" lm_path=lm_path,\n",
" num_cpus=max(os.cpu_count(), 1),\n",
" input_tensor=False)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "NSH8EvL7USac"
},
"source": [
"Now we can check all transcription candidates along with their scores."
]
},
{
"cell_type": "code",
"metadata": {
"id": "nV1CAy0Dit-g"
},
"source": [
"beam_search_lm.forward(log_probs = np.expand_dims(probs, axis=0), log_probs_length=None)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ubpcxp6z3ZF-"
},
"source": [
""
],
"execution_count": null,
"outputs": []
}
]
}
tts/NeMo/tutorials/asr/images/alignment_example.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

tts/NeMo/tutorials/asr/images/vad_post_overlap_diagram.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB