mirror of
https://github.com/storytold/storyteller-ml.git
synced 2026-10-17 00:23:14 +00:00
Introduce code from Rust monorepo. This is a mess
This commit is contained in:
Brandon Thomas
@@ -0,0 +1,312 @@
|
||||
.. _best-practices:
|
||||
|
||||
Best Practices
|
||||
==============
|
||||
|
||||
The NVIDIA NeMo Toolkit is available on GitHub as `open source <https://github.com/NVIDIA/NeMo>`_ as well as
|
||||
a `Docker container on NGC <https://ngc.nvidia.com/catalog/containers/nvidia:nemo>`_. It's assumed the user has
|
||||
already installed NeMo by following the :ref:`quick-start-guide` instructions.
|
||||
|
||||
The conversational AI pipeline consists of three major stages:
|
||||
|
||||
- Automatic Speech Recognition (ASR)
|
||||
- Natural Language Processing (NLP) or Natural Language Understanding (NLU)
|
||||
- Text-to-Speech (TTS) or voice synthesis
|
||||
|
||||
As you talk to a computer, the ASR phase converts the audio signal into text, the NLP stage interprets the question
|
||||
and generates a smart response, and finally the TTS phase converts the text into speech signals to generate audio for
|
||||
the user. The toolkit enables development and training of deep learning models involved in conversational AI and easily
|
||||
chain them together.
|
||||
|
||||
Why NeMo?
|
||||
---------
|
||||
|
||||
Deep learning model development for conversational AI is complex. It involves defining, building, and training several
|
||||
models in specific domains; experimenting several times to get high accuracy, fine tuning on multiple tasks and domain
|
||||
specific data, ensuring training performance and making sure the models are ready for deployment to inference applications.
|
||||
Neural modules are logical blocks of AI applications which take some typed inputs and produce certain typed outputs. By
|
||||
separating a model into its essential components in a building block manner, NeMo helps researchers develop state-of-the-art
|
||||
accuracy models for domain specific data faster and easier.
|
||||
|
||||
Collections of modules for core tasks as well as specific to speech recognition, natural language, speech synthesis help
|
||||
develop modular, flexible, and reusable pipelines.
|
||||
|
||||
A neural module’s inputs/outputs have a neural type, that describes the semantics, the axis order and meaning, and the dimensions
|
||||
of the input/output tensors. This typing allows neural modules to be safely chained together to build models for applications.
|
||||
|
||||
NeMo can be used to train new models or perform transfer learning on existing pre-trained models. Pre-trained weights per module
|
||||
(such as encoder, decoder) help accelerate model training for domain specific data.
|
||||
|
||||
ASR, NLP and TTS pre-trained models are trained on multiple datasets (including some languages such as Mandarin) and optimized
|
||||
for high accuracy. They can be used for transfer learning as well.
|
||||
|
||||
NeMo supports developing models that work with Mandarin Chinese data. Tutorials help users train or fine tune models for
|
||||
conversational AI with the Mandarin Chinese language. The export method provided in NeMo makes it easy to transform a trained
|
||||
model into inference ready format for deployment.
|
||||
|
||||
A key area of development in the toolkit is interoperability with other tools used by speech researchers. Data layer for Kaldi
|
||||
compatibility is one such example.
|
||||
|
||||
NeMo, PyTorch Lightning, And Hydra
|
||||
----------------------------------
|
||||
|
||||
Conversational AI architectures are typically very large and require a lot of data and compute for training. NeMo uses
|
||||
`Pytorch Lightning <https://github.com/PyTorchLightning/pytorch-lightning>`_ for easy and performant multi-GPU/multi-node
|
||||
mixed precision training.
|
||||
|
||||
Pytorch Lightning is a high-performance PyTorch wrapper that organizes PyTorch code, scales model training, and reduces
|
||||
boilerplate. PyTorch Lightning has two main components, the ``LightningModule`` and the Trainer. The ``LightningModule`` is
|
||||
used to organize PyTorch code so that deep learning experiments can be easily understood and reproduced. The Pytorch Lightning
|
||||
Trainer is then able to take the ``LightningModule`` and automate everything needed for deep learning training.
|
||||
|
||||
NeMo models are LightningModules that come equipped with all supporting infrastructure for training and reproducibility. This
|
||||
includes the deep learning model architecture, data preprocessing, optimizer, check-pointing and experiment logging. NeMo
|
||||
models, like LightningModules, are also PyTorch modules and are fully compatible with the broader PyTorch ecosystem. Any NeMo
|
||||
model can be taken and plugged into any PyTorch workflow.
|
||||
|
||||
Configuring conversational AI applications is difficult due to the need to bring together many different Python libraries into
|
||||
one end-to-end system. NeMo uses Hydra for configuring both NeMo models and the PyTorch Lightning Trainer. `Hydra <https://github.com/facebookresearch/hydra>`_
|
||||
is a flexible solution that makes it easy to configure all of these libraries from a configuration file or from the command-line.
|
||||
|
||||
Every NeMo model has an example configuration file and a corresponding script that contains all configurations needed for training
|
||||
to state-of-the-art accuracy. NeMo models have the same look and feel so that it is easy to do conversational AI research across
|
||||
multiple domains.
|
||||
|
||||
Using Optimized Pretrained Models With NeMo
|
||||
-------------------------------------------
|
||||
|
||||
`NVIDIA GPU Cloud (NGC) <https://ngc.nvidia.com/catalog>`_ is a software repository that has containers and models optimized
|
||||
for deep learning. NGC hosts many conversational AI models developed with NeMo that have been trained to state-of-the-art accuracy
|
||||
on large datasets. NeMo models on NGC can be automatically downloaded and used for transfer learning tasks. Pretrained models
|
||||
are the quickest way to get started with conversational AI on your own data. NeMo has many `example scripts <https://github.com/NVIDIA/NeMo/tree/main/examples>`_
|
||||
and `Jupyter Notebook tutorials <https://github.com/NVIDIA/NeMo#tutorials>`_ showing step-by-step how to fine-tune pretrained NeMo
|
||||
models on your own domain-specific datasets.
|
||||
|
||||
For BERT based models, the model weights provided are ready for
|
||||
downstream NLU tasks. For speech models, it can be helpful to start with a pretrained model and then continue pretraining on your
|
||||
own domain-specific data. Jasper and QuartzNet base model pretrained weights have been known to be very efficient when used as
|
||||
base models. For an easy to follow guide on transfer learning and building domain specific ASR models, you can follow this `blog <https://developer.nvidia.com/blog/how-to-build-domain-specific-automatic-speech-recognition-models-on-gpus/>`_.
|
||||
All pre-trained NeMo models can be found on the `NGC NeMo Collection <https://ngc.nvidia.com/catalog/collections?orderBy=scoreDESC&pageNumber=0&query=NeMo&quickFilter=&filters=>`_. Everything needed to quickly get started
|
||||
with NeMo ASR, NLP, and TTS models is there.
|
||||
|
||||
Pre-trained models are packaged as a ``.nemo`` file and contain the PyTorch checkpoint along with everything needed to use the model.
|
||||
NeMo models are trained to state-of-the-art accuracy and trained on multiple datasets so that they are robust to small differences
|
||||
in data. NeMo contains a large variety of models such as speaker identification and Megatron BERT and the best models in speech and
|
||||
language are constantly being added as they become available. NeMo is the premier toolkit for conversational AI model building and
|
||||
training.
|
||||
|
||||
For a list of supported models, refer to the :ref:`tutorials` section.
|
||||
|
||||
ASR Guidance
|
||||
------------
|
||||
|
||||
This section is to help guide your decision making by answering our most asked ASR questions.
|
||||
|
||||
**Q: Is there a way to add domain specific vocabulary in NeMo? If so, how do I do that?**
|
||||
A: QuartzNet and Jasper models are character-based. So pretrained models we provide for these two output lowercase English
|
||||
letters and ‘. Users can re-retrain them on vocabulary with upper case letters and punctuation symbols.
|
||||
|
||||
**Q: When training, there are “Reference” lines and “Decoded” lines that are printed out. It seems like the reference line should
|
||||
be the “truth” line and the decoded line should be what the ASR is transcribing. Why do I see that even the reference lines do not
|
||||
appear to be correct?**
|
||||
A: Because our pre-trained models can only output lowercase letters and apostrophe, everything else is dropped. So the model will
|
||||
transcribe 10 as ten. The best way forward is to prepare the training data first by transforming everything to lowercase and convert
|
||||
the numbers from digit representation to word representation using a simple library such as `inflect <https://pypi.org/project/inflect/>`_. Then, add the uppercase letters
|
||||
and punctuation back using the NLP punctuation model. Here is an example of how this is incorporated: `NeMo voice swap demo <https://github.com/NVIDIA/NeMo/blob/main/tutorials/VoiceSwapSample.ipynb>`_.
|
||||
|
||||
**Q: What languages are supported in NeMo currently?**
|
||||
A: Along with English, we provide pre-trained models for Zh, Es, Fr, De, Ru, It, Ca and Pl languages.
|
||||
For more information, see `NeMo Speech Models <https://ngc.nvidia.com/catalog/collections/nvidia:nemo_asr>`_.
|
||||
|
||||
Data Augmentation
|
||||
-----------------
|
||||
|
||||
Data augmentation in ASR is invaluable. It comes at the cost of increased training time if samples are augmented during training
|
||||
time. To save training time, it is recommended to pre-process the dataset offline for a one time preprocessing cost and then train
|
||||
the dataset on this augmented training set.
|
||||
|
||||
For example, processing a single sample involves:
|
||||
|
||||
- Speed perturbation
|
||||
- Time stretch perturbation (sample level)
|
||||
- Noise perturbation
|
||||
- Impulse perturbation
|
||||
- Time stretch augmentation (batch level, neural module)
|
||||
|
||||
A simple tutorial guides users on how to use these utilities provided in `GitHub: NeMo <https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/05_Online_Noise_Augmentation.ipynb>`_.
|
||||
|
||||
Speech Data Explorer
|
||||
--------------------
|
||||
|
||||
Speech data explorer is a `Dash-based tool <https://plotly.com/dash/>`_ for interactive exploration of ASR/TTS datasets.
|
||||
|
||||
Speech data explorer collects:
|
||||
|
||||
- dataset statistics (alphabet, vocabulary, and duration-based histograms)
|
||||
- navigation across datasets (sorting and filtering)
|
||||
- inspections of individual utterances (waveform, spectrogram, and audio player)
|
||||
- errors analysis (word error rate, character error rate, word match rate, mean word accuracy, and diff)
|
||||
|
||||
In order to use the tool, it needs to be installed separately. Perform the steps `here <https://github.com/NVIDIA/NeMo/tree/main/tools/speech_data_explorer>`_ to install speech data explorer.
|
||||
|
||||
Using Kaldi Formatted Data
|
||||
--------------------------
|
||||
|
||||
The `Kaldi Speech Recognition Toolkit <https://kaldi-asr.org/>`_ project began in 2009 at `Johns Hopkins University <https://www.jhu.edu/>`. It is a toolkit written in C++. If
|
||||
researchers have used Kaldi and have datasets that are formatted to be used with the toolkit; they can use NeMo to develop models
|
||||
based on that data.
|
||||
|
||||
To load Kaldi-formatted data, you can simply use ``KaldiFeatureDataLayer`` instead of ``AudioToTextDataLayer``. The ``KaldiFeatureDataLayer``
|
||||
takes in the argument ``kaldi_dir`` instead of a ``manifest_filepath``. The ``manifest_filepath`` argument should be set to the directory
|
||||
that contains the files ``feats.scp`` and ``text``.
|
||||
|
||||
Using Speech Command Recognition Task For ASR Models
|
||||
----------------------------------------------------
|
||||
|
||||
Speech Command Recognition is the task of classifying an input audio pattern into a set of discrete classes. It is a subset of ASR,
|
||||
sometimes referred to as Key Word Spotting, in which a model is constantly analyzing speech patterns to detect certain ``action`` classes.
|
||||
|
||||
Upon detection of these commands, a specific action can be taken. An example Jupyter notebook provided in NeMo shows how to train a
|
||||
QuartzNet model with a modified decoder head trained on a speech commands dataset.
|
||||
|
||||
.. note:: It is preferred that you use absolute paths to ``data_dir`` when preprocessing the dataset.
|
||||
|
||||
NLP Fine-Tuning BERT
|
||||
--------------------
|
||||
|
||||
BERT, or Bidirectional Encoder Representations from Transformers, is a neural approach to pre-train language representations which
|
||||
obtains near state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks, including the GLUE benchmark and
|
||||
SQuAD Question & Answering dataset.
|
||||
|
||||
BERT model checkpoints (`BERT-large-uncased <https://ngc.nvidia.com/catalog/models/nvidia:bertlargeuncasedfornemo>`_ and `BERT-base-uncased <https://ngc.nvidia.com/catalog/models/nvidia:bertbaseuncasedfornemo>`_) are provided can be used for either fine tuning BERT on your custom
|
||||
dataset, or fine tuning downstream tasks, including GLUE benchmark tasks, Question & Answering tasks, Joint Intent & Slot detection,
|
||||
Punctuation and Capitalization, Named Entity Recognition, and Speech Recognition post processing model to correct mistakes.
|
||||
|
||||
.. note:: Almost all NLP examples also support RoBERTa and ALBERT models for downstream fine-tuning tasks (see the list of all
|
||||
supported models by calling ``nemo.collections.nlp.modules.common.lm_utils.get_pretrained_lm_models_list()``. The user needs to specify
|
||||
the name of the model desired while running the example scripts.
|
||||
|
||||
BioMegatron Medical BERT
|
||||
------------------------
|
||||
|
||||
BioMegatron is a large language model (Megatron-LM) trained on larger domain text corpus (PubMed abstract + full-text-commercial).
|
||||
It achieves state-of-the-art results for certain tasks such as Relationship Extraction, Named Entity Recognition and Question &
|
||||
Answering. Follow these tutorials to learn how to train and fine tune BioMegatron; pretrained models are provided on NGC:
|
||||
|
||||
- `Relation Extraction BioMegatron <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb>`_
|
||||
- `Token Classification BioMegatron <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification-BioMegatron.ipynb>`_
|
||||
|
||||
Efficient Training With NeMo
|
||||
----------------------------
|
||||
|
||||
Using Mixed Precision
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Mixed precision accelerates training speed while protecting against noticeable loss. Tensor Cores is a specific hardware unit that
|
||||
comes starting with the Volta and Turing architectures to accelerate large matrix to matrix multiply-add operations by operating them
|
||||
on half precision inputs and returning the result in full precision.
|
||||
|
||||
Neural networks which usually use massive matrix multiplications can be significantly sped up with mixed precision and Tensor Cores.
|
||||
However, some neural network layers are numerically more sensitive than others. Apex AMP is an NVIDIA library that maximizes the
|
||||
benefit of mixed precision and Tensor Cores usage for a given network.
|
||||
|
||||
Multi-GPU Training
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This section is to help guide your decision making by answering our most asked multi-GPU training questions.
|
||||
|
||||
**Q: Why is multi-GPU training preferred over other types of training?**
|
||||
A: Multi-GPU training can reduce the total training time by distributing the workload onto multiple compute instances. This is
|
||||
particularly important for large neural networks which would otherwise take weeks to train until convergence. Since NeMo supports
|
||||
multi-GPU training, no code change is needed to move from single to multi-GPU training, only a slight change in your launch command
|
||||
is required.
|
||||
|
||||
**Q: What are the advantages of mixed precision training?**
|
||||
A: Mixed precision accelerates training speed while protecting against noticeable loss in precision. Tensor Cores is a specific
|
||||
hardware unit that comes starting with the Volta and Turing architectures to accelerate large matrix multiply-add operations by
|
||||
operating on half precision inputs and returning the result in full precision in order to prevent loss in precision. Neural
|
||||
networks which usually use massive matrix multiplications can be significantly sped up with mixed precision and Tensor Cores.
|
||||
However, some neural network layers are numerically more sensitive than others. Apex AMP is a NVIDIA library that maximizes the
|
||||
benefit of mixed precision and Tensor Core usage for a given network.
|
||||
|
||||
**Q: What is the difference between multi-GPU and multi-node training?**
|
||||
A: Multi-node is an abstraction of multi-GPU training, which requires a distributed compute cluster, where each node can have multiple
|
||||
GPUs. Multi-node training is needed to scale training beyond a single node to large amounts of GPUs.
|
||||
|
||||
From the framework perspective, nothing changes from moving to multi-node training. However, a master address and port needs to be set
|
||||
up for inter-node communication. Multi-GPU training will then be launched on each node with passed information. You might also consider
|
||||
the underlying inter-node network topology and type to achieve full performance, such as HPC-style hardware such as NVLink, InfiniBand
|
||||
networking, or Ethernet.
|
||||
|
||||
|
||||
Recommendations For Optimization And FAQs
|
||||
-----------------------------------------
|
||||
|
||||
This section is to help guide your decision making by answering our most asked NeMo questions.
|
||||
|
||||
**Q: Are there areas where performance can be increased?**
|
||||
A: You should try using mixed precision for improved performance. Note that typically when using mixed precision, memory consumption
|
||||
is decreased and larger batch sizes could be used to further improve the performance.
|
||||
|
||||
When fine-tuning ASR models on your data, it is almost always possible to take advantage of NeMo's pre-trained modules. Even if you
|
||||
have a different target vocabulary, or even a different language; you can still try starting with pre-trained weights from Jasper or
|
||||
QuartzNet ``encoder`` and only adjust the ``decoder`` for your needs.
|
||||
|
||||
**Q: What is the recommended sampling rate for ASR?**
|
||||
A: The released models are based on 16 KHz audio, therefore, ensure you use models with 16 KHz audio. Reduced performance should be
|
||||
expected for any audio that is up-sampled from a sampling frequency less than 16 KHz data.
|
||||
|
||||
**Q: How do we use this toolkit for audio with different types of compression and frequency than the training domain for ASR?**
|
||||
A: You have to match the compression and frequency.
|
||||
|
||||
**Q: How do you replace the 6-gram out of the ASR model with a custom language model? What is the language format supported in NeMo?**
|
||||
A: NeMo’s Beam Search decoder with Levenberg-Marquardt (LM) neural module supports the KenLM language model.
|
||||
|
||||
- You should retrain the KenLM language model on your own dataset. Refer to `KenLM’s documentation <https://github.com/kpu/kenlm#kenlm>`_.
|
||||
- If you want to use a different language model, other than KenLM, you will need to implement a corresponding decoder module.
|
||||
- Transformer-XL example is present in OS2S. It would need to be updated to work with NeMo. `Here is the code <https://github.com/NVIDIA/OpenSeq2Seq/tree/master/external_lm_rescore>`_.
|
||||
|
||||
**Q: How do I use text-to-speech (TTS)?**
|
||||
A:
|
||||
- Obtain speech data ideally at 22050 Hz or alternatively at a higher sample rate and then down sample to 22050 Hz.
|
||||
- If less than 22050 Hz and above 16000 Hz:
|
||||
- Retrain WaveGlow on your own dataset.
|
||||
- Tweak the spectrogram generation parameters, namely the ``window_size`` and the ``window_stride`` for their fourier
|
||||
transforms.
|
||||
- For below 16000 Hz, look into obtaining new data.
|
||||
- In terms of bitrate/quantization, the general advice is the higher the better. We have not experimented enough to state how much
|
||||
this impacts quality.
|
||||
- For the amount of data, again the more the better, and the more diverse in terms of phonemes the better. Aim for around 20 hours
|
||||
of speech after filtering for silences and non-speech audio.
|
||||
- Most open speech datasets are in ~10 second format so training Tacotron 2 on audio on the order of 10s - 20s per sample is known
|
||||
to work. Additionally, the longer the speech samples, the more difficult it will be to train Tacotron 2.
|
||||
- Audio files should be clean. There should be little background noise or music. Data recorded from a studio mic is likely to be easier
|
||||
to train compared to data captured using a phone.
|
||||
- To ensure pronunciation of words are accurate; the technical challenge is related to the dataset, text to phonetic spelling is
|
||||
required, use phonetic alphabet (notation) that has the name correctly pronounced.
|
||||
- Here are some example parameters you can use to train Tacotron 2:
|
||||
- Tacotron 2 requires a single speaker dataset
|
||||
- Use AMP level O0
|
||||
- Trim long silences in the beginning and end
|
||||
- ``optimizer="adam"``
|
||||
- ``beta1 = 0.9``
|
||||
- ``beta2 = 0.999``
|
||||
- ``lr=0.001 (constant)``
|
||||
- ``amp_opt_level="O0"``
|
||||
- ``weight_decay=1e-6``
|
||||
- ``batch_size=48 (per GPU)``
|
||||
- ``trim_silence=True``
|
||||
|
||||
Resources
|
||||
---------
|
||||
|
||||
Ensure you are familiar with the following resources for NeMo.
|
||||
|
||||
- Developer blogs
|
||||
- `How to Build Domain Specific Automatic Speech Recognition Models on GPUs <https://developer.nvidia.com/blog/how-to-build-domain-specific-automatic-speech-recognition-models-on-gpus/>`_
|
||||
- `Develop Smaller Speech Recognition Models with NVIDIA’s NeMo Framework <https://developer.nvidia.com/blog/develop-smaller-speech-recognition-models-with-nvidias-nemo-framework/>`_
|
||||
- `Neural Modules for Fast Development of Speech and Language Models <https://developer.nvidia.com/blog/neural-modules-for-speech-language-models/>`_
|
||||
|
||||
- Domain specific, transfer learning, Docker container with Jupyter Notebooks
|
||||
- `Domain Specific NeMo ASR Application <https://ngc.nvidia.com/catalog/containers/nvidia:nemo_asr_app_img>`_
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
Introduction
|
||||
============
|
||||
|
||||
.. # define a hard line break for html
|
||||
.. |br| raw:: html
|
||||
|
||||
<br />
|
||||
|
||||
.. _dummy_header:
|
||||
|
||||
`NVIDIA NeMo <https://github.com/NVIDIA/NeMo>`_ is a toolkit for building new state-of-the-art
|
||||
conversational AI models. NeMo has separate collections for Automatic Speech Recognition (ASR),
|
||||
Natural Language Processing (NLP), and Text-to-Speech (TTS) models. Each collection consists of
|
||||
prebuilt modules that include everything needed to train on your data.
|
||||
Every module can easily be customized, extended, and composed to create new conversational AI
|
||||
model architectures.
|
||||
|
||||
Conversational AI architectures are typically large and require a lot of data and compute
|
||||
for training. NeMo uses `PyTorch Lightning <https://www.pytorchlightning.ai/>`_ for easy and performant multi-GPU/multi-node
|
||||
mixed-precision training.
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<div style="position: relative; padding-bottom: 3%; height: 0; overflow: hidden; max-width: 100%; height: auto;">
|
||||
<iframe width="560" height="315" src="https://www.youtube.com/embed/wBgpMf_KQVw" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
</div>
|
||||
|
||||
For more information and questions, visit the `NVIDIA NeMo Discussion Board <https://github.com/NVIDIA/NeMo/discussions>`_.
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
Before you begin using NeMo, it's assumed you meet the following prerequisites.
|
||||
|
||||
#. You have Python version 3.6, 3.7 or 3.8.
|
||||
|
||||
#. You have Pytorch version 1.8.1.
|
||||
|
||||
#. You have access to an NVIDIA GPU for training.
|
||||
|
||||
.. _quick_start_guide:
|
||||
|
||||
Quick Start Guide
|
||||
-----------------
|
||||
|
||||
This NeMo Quick Start Guide is a starting point for users who want to try out NeMo; specifically, this guide enables users to quickly get started with the NeMo fundamentals by walking you through an example audio translator and voice swap.
|
||||
|
||||
If you're new to NeMo, the best way to get started is to take a look at the following tutorials:
|
||||
|
||||
* `Text Classification (Sentiment Analysis) <https://github.com/NVIDIA/NeMo/blob/main/tutorials/Text_Classification_Sentiment_Analysis>`__ - demonstrates the Text Classification model using the NeMo NLP collection.
|
||||
* `NeMo Primer <https://github.com/NVIDIA/NeMo/blob/main/tutorials/00_NeMo_Primer.ipynb>`__ - introduces NeMo, PyTorch Lightning, and OmegaConf, and shows how to use, modify, save, and restore NeMo models.
|
||||
* `NeMo Models <https://github.com/NVIDIA/NeMo/blob/main/tutorials/01_NeMo_Models.ipynb>`__ - explains the fundamental concepts of the NeMo model.
|
||||
* `NeMo voice swap demo <https://github.com/NVIDIA/NeMo/blob/main/tutorials/NeMo_voice_swap_app.ipynb>`__ - demonstrates how to swap a voice in the audio fragment with a computer generated one using NeMo.
|
||||
|
||||
Below we is the code snippet of Audio Translator application.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Import NeMo and it's ASR, NLP and TTS collections
|
||||
import nemo
|
||||
# Import Speech Recognition collection
|
||||
import nemo.collections.asr as nemo_asr
|
||||
# Import Natural Language Processing colleciton
|
||||
import nemo.collections.nlp as nemo_nlp
|
||||
# Import Speech Synthesis collection
|
||||
import nemo.collections.tts as nemo_tts
|
||||
|
||||
# Next, we instantiate all the necessary models directly from NVIDIA NGC
|
||||
# Speech Recognition model - QuartzNet trained on Russian part of MCV 6.0
|
||||
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="stt_ru_quartznet15x5").cuda()
|
||||
# Neural Machine Translation model
|
||||
nmt_model = nemo_nlp.models.MTEncDecModel.from_pretrained(model_name='nmt_ru_en_transformer6x6').cuda()
|
||||
# Spectrogram generator which takes text as an input and produces spectrogram
|
||||
spectrogram_generator = nemo_tts.models.Tacotron2Model.from_pretrained(model_name="tts_en_tacotron2").cuda()
|
||||
# Vocoder model which takes spectrogram and produces actual audio
|
||||
vocoder = nemo_tts.models.WaveGlowModel.from_pretrained(model_name="tts_waveglow_88m").cuda()
|
||||
# Transcribe an audio file
|
||||
# IMPORTANT: The audio must be mono with 16Khz sampling rate
|
||||
# Get example from: https://nemo-public.s3.us-east-2.amazonaws.com/mcv-samples-ru/common_voice_ru_19034087.wav
|
||||
russian_text = quartznet.transcribe(['Path_to_audio_file'])
|
||||
print(russian_text)
|
||||
# You should see russian text here. Let's translate it to English
|
||||
english_text = nmt_model.translate(russian_text)
|
||||
print(english_text)
|
||||
# After this you should see English translation
|
||||
# Let's convert it into audio
|
||||
# A helper function which combines Tacotron2 and WaveGlow to go directly from
|
||||
# text to audio
|
||||
def text_to_audio(text):
|
||||
parsed = spectrogram_generator.parse(text)
|
||||
spectrogram = spectrogram_generator.generate_spectrogram(tokens=parsed)
|
||||
audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)
|
||||
return audio.to('cpu').numpy()
|
||||
audio = text_to_audio(english_text[0])
|
||||
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
Pip
|
||||
~~~
|
||||
Use this installation mode if you want the latest released version.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
apt-get update && apt-get install -y libsndfile1 ffmpeg
|
||||
pip install Cython
|
||||
pip install nemo_toolkit[all]
|
||||
|
||||
Pip from source
|
||||
~~~~~~~~~~~~~~~
|
||||
Use this installation mode if you want the version from a particular GitHub branch (for example, ``main``).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
apt-get update && apt-get install -y libsndfile1 ffmpeg
|
||||
pip install Cython
|
||||
python -m pip install git+https://github.com/NVIDIA/NeMo.git@{BRANCH}#egg=nemo_toolkit[all]
|
||||
# For v1.0.2, replace {BRANCH} with v1.0.2 like so:
|
||||
# python -m pip install git+https://github.com/NVIDIA/NeMo.git@v1.0.2#egg=nemo_toolkit[all]
|
||||
|
||||
From source
|
||||
~~~~~~~~~~~
|
||||
Use this installation mode if you are contributing to NeMo.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
apt-get update && apt-get install -y libsndfile1 ffmpeg
|
||||
git clone https://github.com/NVIDIA/NeMo
|
||||
cd NeMo
|
||||
./reinstall.sh
|
||||
|
||||
Docker containers
|
||||
~~~~~~~~~~~~~~~~~
|
||||
If you chose to work with the ``main`` branch, we recommend using `NVIDIA's PyTorch container version 21.05-py3 <https://ngc.nvidia.com/containers/nvidia:pytorch/tags>`_, then install from GitHub.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker run --gpus all -it --rm -v <nemo_github_folder>:/NeMo --shm-size=8g \
|
||||
-p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit \
|
||||
stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:21.05-py3
|
||||
|
||||
|
||||
FAQ
|
||||
---
|
||||
Have a look at our `discussions board <https://github.com/NVIDIA/NeMo/discussions>`_ and feel free to post a question or start a discussion.
|
||||
|
||||
|
||||
Contributing
|
||||
------------
|
||||
|
||||
We welcome community contributions! Refer to the `CONTRIBUTING.md <https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md>`_ file for the process.
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
NeMo is under `Apache 2.0 license <https://github.com/NVIDIA/NeMo/blob/main/LICENSE>`_.
|
||||
@@ -0,0 +1,117 @@
|
||||
.. _tutorials:
|
||||
|
||||
Tutorials
|
||||
=========
|
||||
|
||||
The best way to get started with NeMo is to start wtih one of our tutorials.
|
||||
|
||||
Most NeMo tutorials can be run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.
|
||||
|
||||
To run a tutorial:
|
||||
|
||||
#. Click the **Colab** link (see table below).
|
||||
#. Connect to an instance with a GPU. For example, click **Runtime** > **Change runtime type** and select **GPU** for the hardware accelerator.
|
||||
|
||||
.. list-table:: **Tutorials**
|
||||
:widths: 15 25 25
|
||||
:header-rows: 1
|
||||
|
||||
* - Domain
|
||||
- Title
|
||||
- GitHub URL
|
||||
* - General
|
||||
- Getting Started: Exploring Nemo Fundamentals
|
||||
- `NeMo Fundamentals <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/00_NeMo_Primer.ipynb>`_
|
||||
* - General
|
||||
- Getting Started: Sample Conversational AI application
|
||||
- `Audio translator example <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/AudioTranslationSample.ipynb>`_
|
||||
* - General
|
||||
- Getting Started: Voice swap application
|
||||
- `Voice swap example <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/VoiceSwapSample.ipynb>`_
|
||||
* - General
|
||||
- Exploring NeMo Model Construction
|
||||
- `NeMo Models <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/01_NeMo_Models.ipynb>`_
|
||||
* - ASR
|
||||
- ASR with NeMo
|
||||
- `ASR with NeMo <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb>`_
|
||||
* - ASR
|
||||
- ASR with Subword Tokenization
|
||||
- `ASR with Subword Tokenization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/08_ASR_with_Subword_Tokenization.ipynb>`_
|
||||
* - ASR
|
||||
- Offline ASR Inference with Beam Search and External Language Model Rescoring
|
||||
- `Offline ASR <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/Offline_ASR.ipynb>`_
|
||||
* - ASR
|
||||
- Online ASR inference with Microphone
|
||||
- `Online ASR Microphone <https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/02_Online_ASR_Microphone_Demo.ipynb>`_
|
||||
* - ASR
|
||||
- Fine-tuning CTC Models on New Languages
|
||||
- `ASR CTC Language Fine-Tuning <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/10_ASR_CTC_Language_Finetuning.ipynb>`_
|
||||
* - ASR
|
||||
- Speech Commands
|
||||
- `Speech Commands <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/03_Speech_Commands.ipynb>`_
|
||||
* - ASR
|
||||
- Online and Offline Speech Commands Inference
|
||||
- `Online Offline Microphone Speech Commands <https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/04_Online_Offline_Speech_Commands_Demo.ipynb>`_
|
||||
* - ASR
|
||||
- Voice Activiy Detection (VAD)
|
||||
- `Voice Activiy Detection <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/06_Voice_Activiy_Detection.ipynb>`_
|
||||
* - ASR
|
||||
- Online and Offline VAD Inference
|
||||
- `Online Offline Microphone VAD <https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb>`_
|
||||
* - ASR
|
||||
- Speaker Recognition and Verification
|
||||
- `Speaker Recognition and Verification <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/speaker_recognition/Speaker_Recognition_Verification.ipynb>`_
|
||||
* - ASR
|
||||
- Speaker Diarization Inference
|
||||
- `Speaker Diarization Inference <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb>`_
|
||||
* - ASR
|
||||
- ASR with Speaker Diarization
|
||||
- `ASR with Speaker Diarization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/speaker_recognition/ASR_with_SpeakerDiarization.ipynb>`_
|
||||
* - ASR
|
||||
- Online Noise Augmentation
|
||||
- `Online Noise Augmentation <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/05_Online_Noise_Augmentation.ipynb>`_
|
||||
* - NLP
|
||||
- Using Pretrained Language Models for Downstream Tasks
|
||||
- `Pretrained Language Models for Downstream Tasks <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb>`_
|
||||
* - NLP
|
||||
- Exploring NeMo NLP Tokenizers
|
||||
- `NLP Tokenizers <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/02_NLP_Tokenizers.ipynb>`_
|
||||
* - NLP
|
||||
- Text Classification (Sentiment Analysis) with BERT
|
||||
- `Text Classification (Sentiment Analysis) <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb>`_
|
||||
* - NLP
|
||||
- Question Answering with SQuAD
|
||||
- `Question Answering Squad <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Question_Answering_Squad.ipynb>`_
|
||||
* - NLP
|
||||
- Token Classification (Named Entity Recognition)
|
||||
- `Token Classification: Named Entity Recognition <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb>`_
|
||||
* - NLP
|
||||
- Joint Intent Classification and Slot Filling
|
||||
- `Joint Intent and Slot Classification <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb>`_
|
||||
* - NLP
|
||||
- GLUE Benchmark
|
||||
- `GLUE Benchmark <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/GLUE_Benchmark.ipynb>`_
|
||||
* - NLP
|
||||
- Punctuation and Capitalization
|
||||
- `Punctuation and Capitalization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Punctuation_and_Capitalization.ipynb>`_
|
||||
* - NLP
|
||||
- Named Entity Recognition - BioMegatron
|
||||
- `Named Entity Recognition - BioMegatron <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Token_Classification-BioMegatron.ipynb>`_
|
||||
* - NLP
|
||||
- Relation Extraction - BioMegatron
|
||||
- `Relation Extraction - BioMegatron <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb>`_
|
||||
* - TTS
|
||||
- Speech Synthesis
|
||||
- `TTS Inference <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/tts/1_TTS_inference.ipynb>`_
|
||||
* - TTS
|
||||
- Speech Synthesis
|
||||
- `Tacotron2 Training <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/tts/2_TTS_Tacotron2_Training.ipynb>`_
|
||||
* - Tools
|
||||
- CTC Segmentation
|
||||
- `CTC Segmentation <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/tools/CTC_Segmentation_Tutorial.ipynb>`_
|
||||
* - Text Processing
|
||||
- Text Normalization for TTS
|
||||
- `Text Normalization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/text_processing/Text_Normalization.ipynb>`_
|
||||
* - Text Processing
|
||||
- Inverse Text Normalization for ASR
|
||||
- `Inverse Text Normalization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/text_processing/Inverse_Text_Normalization.ipynb>`_
|
||||
Reference in New Issue
Block a user