Introduce code from Rust monorepo. This is a mess

tts/NeMo/docs/source/asr/speech_classification/configs.rst

@@ -0,0 +1,104 @@
+NeMo Speech Classification Configuration Files
+================================================
+
+This page covers NeMo configuration file setup that is specific to models in the Speech Classification collection.
+For general information about how to set up and run experiments that is common to all NeMo models (e.g.
+experiment manager and PyTorch Lightning trainer parameters), see the :doc:`../../core/core`  page.
+
+The model section of NeMo Speech Classification configuration files will generally require information about the dataset(s) being
+used, the preprocessor for audio files, parameters for any augmentation being performed, as well as the
+model architecture specification.
+The sections on this page cover each of these in more detail.
+
+Example configuration files for all of the NeMo ASR scripts can be found in the
+``<NeMo_git_root>/examples/asr/conf>``.
+
+
+Dataset Configuration
+---------------------
+
+Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
+``test_ds`` sections of your configuration file, respectively.
+Depending on the task, you may have arguments specifying the sample rate of your audio files, labels, whether or not to shuffle the dataset, and so on.
+You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command line
+at runtime.
+
+Any initialization parameters that are accepted for the Dataset class used in your experiment
+can be set in the config file.
+See the `Datasets <../api.html#Datasets>`__ section of the API for a list of Datasets and their respective parameters.
+
+An example Speech Classification train and validation configuration could look like: 
+
+.. code-block:: yaml
+
+  model:
+    sample_rate: 16000
+    repeat: 2 # number of convolutional sub-blocks within a block, R in <MODEL>_[BxRxC]
+    dropout: 0.0
+    kernel_size_factor: 1.0
+    labels: ['bed', 'bird', 'cat', 'dog', 'down', 'eight', 'five', 'four', 'go', 'happy', 'house', 'left', 'marvin',
+    'nine', 'no', 'off', 'on', 'one', 'right', 'seven', 'sheila', 'six', 'stop', 'three', 'tree', 'two', 'up',
+    'wow', 'yes', 'zero']
+
+    train_ds:
+      manifest_filepath: ???
+      sample_rate: ${model.sample_rate}
+      labels: ${model.labels} # Uses the labels above
+      batch_size: 128
+      shuffle: True
+
+    validation_ds:
+      manifest_filepath: ???
+      sample_rate: ${model.sample_rate}
+      labels: ${model.labels} # Uses the labels above
+      batch_size: 128
+      shuffle: False # No need to shuffle the validation data
+
+If you would like to use tarred dataset, have a look at `Datasets Configuration <../configs.html#dataset-configuration>`__.
+
+
+Preprocessor Configuration
+--------------------------
+Preprocessor helps to compute MFCC or mel spectrogram features that are given as inputs to model. 
+For details on how to write this section, refer to `Preprocessor Configuration <../configs.html#preprocessor-configuration>`__
+
+Check config yaml files in ``<NeMo_git_root>/examples/asr/conf`` to find the processors been used by speech classification models.
+
+
+Augmentation Configurations
+---------------------------
+
+There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
+configuration file using the ``augmentor`` and ``spec_augment`` section.
+For details on how to write this section, refer to `Augmentation Configuration <../configs.html#augmentation-configurations>`__
+
+Check config yaml files in ``<NeMo_git_root>/tutorials/asr/conf`` to find the processors been used by speech classification models. 
+
+
+Model Architecture Configurations
+---------------------------------
+
+Each configuration file should describe the model architecture being used for the experiment.
+Models in the NeMo ASR collection need a ``encoder`` section and a ``decoder`` section, with the ``_target_`` field
+specifying the module to use for each.
+
+The following sections go into more detail about the specific configurations of each model architecture.
+
+The `MatchboxNet <./models.html#matchboxnet-speech-commands>`__ and `MarbleNet <./models.html#marblenet-vad>`__ models are very similar, and they are based on `QuartzNet <../models.html#quartznet>`__  and as such the components in their
+configs are very similar as well.
+
+Decoder Configurations
+------------------------
+
+After features have been computed from ConvASREncoder, we pass the features to decoder to compute embeddings and then to compute log_probs 
+for training models.
+
+.. code-block:: yaml
+
+  model:
+    ...
+    decoder:
+      _target_: nemo.collections.asr.modules.ConvASRDecoderClassification
+      feat_in: *enc_final_filters
+      return_logits: true # return logits if true, else return softmax output
+      pooling_type: 'avg' # AdaptiveAvgPool1d 'avg' or AdaptiveMaxPool1d 'max'

tts/NeMo/docs/source/asr/speech_classification/data/classification_results.csv

@@ -0,0 +1,9 @@
+Model Name,Model Base Class,Model Card
+vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
+vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
+commandrecognition_en_matchboxnet3x1x64_v1,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v1"
+commandrecognition_en_matchboxnet3x2x64_v1,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v1"
+commandrecognition_en_matchboxnet3x1x64_v2,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2"
+commandrecognition_en_matchboxnet3x2x64_v2,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2"
+commandrecognition_en_matchboxnet3x1x64_v2_subset_task,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2_subset_task"
+commandrecognition_en_matchboxnet3x2x64_v2_subset_task,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2_subset_task"

tts/NeMo/docs/source/asr/speech_classification/datasets.rst

@@ -0,0 +1,131 @@
+Datasets
+========
+
+NeMo has scripts to convert several common ASR datasets into the format expected by the `nemo_asr` collection.
+You can get started with those datasets by following the instructions to run those scripts in the section appropriate to each dataset below.
+
+If you have your own data and want to preprocess it to use with NeMo ASR models, check out the `Preparing Custom Speech Classification Data`_ section at the bottom of the page.
+
+.. _Freesound-dataset:
+
+Freesound
+-----------
+
+`Freesound <http://www.freesound.org/>`_ is a website that aims to create a huge open collaborative database of audio snippets, samples, recordings, bleeps. 
+Most audio samples are released under Creative Commons licenses that allow their reuse. 
+Researchers and developers can access Freesound content using the Freesound API to retrieve meaningful sound information such as metadata, analysis files, and the sounds themselves. 
+
+**Instructions**
+
+Go to ``<NeMo_git_root>/scripts/freesound_download_resample`` and follow the below steps to download and convert freedsound data into a format expected by the `nemo_asr` collection.
+
+1. We will need some required libraries including freesound, requests, requests_oauthlib, joblib, librosa and sox. If they are not installed, please run `pip install -r freesound_requirements.txt`
+2. Create an API key for freesound.org at https://freesound.org/help/developers/
+3. Create a python file called `freesound_private_apikey.py` and add lined `api_key = <your Freesound api key> and client_id = <your Freesound client id>`
+4. Authorize by run `python freesound_download.py --authorize` and visit the website and paste response code
+5. Feel free to change any arguments in `download_resample_freesound.sh` such as max_samples and max_filesize
+6. Run `bash download_resample_freesound.sh <numbers of files you want> <download data directory> <resampled data directory>` . For example: 
+
+.. code-block:: bash
+
+    bash download_resample_freesound.sh 4000 ./freesound ./freesound_resampled_background
+
+Note that downloading this dataset may take hours. Change categories in download_resample_freesound.sh to include other (speech) categories audio files.
+Then, you should have 16khz mono wav files in `<resampled data directory>`. 
+
+
+.. _Google-Speech-Commands-Dataset:
+
+Google Speech Commands Dataset
+------------------------------
+
+Google released two versions of the dataset with the first version containing 65k samples over 30 classes and the second containing 110k samples over 35 classes.
+We refer to these datasets as `v1` and `v2` respectively.
+
+Run the script `process_speech_commands_data.py` to process Google Speech Commands dataset in order to generate files in the supported format of  `nemo_asr`,
+which can be found in ``<NeMo_git_root>/scripts/dataset_processing/``. 
+You should set the data folder of Speech Commands using :code:`--data_root` and the version of the dataset using :code:`--data_version` as an int.
+
+You can further rebalance the train set by randomly oversampling files inside the manifest by passing the `--rebalance` flag.
+
+.. code-block:: bash
+
+    python process_speech_commands_data.py --data_root=<data directory> --data_version=<1 or 2> {--rebalance}
+
+
+Then, you should have `train_manifest.json`, `validation_manifest.json` and `test_manifest.json`
+in the directory `{data_root}/google_speech_recognition_v{1/2}`.
+
+.. note::
+  You should have at least 4GB or 6GB of disk space available if you use v1 or v2 respectively. 
+  Also, it will take some time to download and process, so go grab a coffee.
+
+Each line is a training example.
+
+.. code-block:: bash
+
+  {"audio_filepath": "<absolute path to dataset>/two/8aa35b0c_nohash_0.wav", "duration": 1.0, "label": "two"}
+  {"audio_filepath": "<absolute path to dataset>/two/ec5ab5d5_nohash_2.wav", "duration": 1.0, "label": "two"}
+
+
+
+Speech Command & Freesound for VAD
+------------------------------------
+Speech Command & Freesound (SCF) dataset is used to train MarbleNet in the `paper <https://arxiv.org/pdf/2010.13886.pdf>`_. Here we show how to download and process it. 
+This script assumes that you already have the Freesound dataset, if not, have a look at :ref:`Freesound-dataset`. 
+We will use the open-source :ref:`Google-Speech-Commands-Dataset` (we will use V2 of the dataset for SCF dataset, but require very minor changes to support V1 dataset) as our speech data. 
+
+These scripts below will download the Google Speech Commands v2 dataset and convert speech and background data to a format suitable for use with nemo_asr.
+
+.. note::
+  You may additionally pass :code:`--test_size` or :code:`--val_size` flag for splitting train val and test data. 
+
+  You may additionally pass :code:`--seg_len` flag for indicating the segment length. Default is 0.63s.
+
+  You may additionally pass a :code:`-rebalance_method='fixed|over|under'` at the end of the script to rebalance the class samples in the manifest. 
+    
+
+
+* `'fixed'`: Fixed number of sample for each class. Train 5000, val 1000, and test 1000. (Change number in script if you want)
+* `'over'`: Oversampling rebalance method
+* `'under'`: Undersampling rebalance method
+
+
+.. code-block:: bash
+
+    mkdir './google_dataset_v2'
+    python process_vad_data.py --out_dir='./manifest/' --speech_data_root='./google_dataset_v2'--background_data_root=<resampled freesound data directory> --log --rebalance_method='fixed' 
+
+
+After download and conversion, your `manifest` folder should contain a few json manifest files:
+
+* `(balanced_)background_testing_manifest.json`
+* `(balanced_)background_training_manifest.json`
+* `(balanced_)background_validation_manifest.json`
+* `(balanced_)speech_testing_manifest.json`
+* `(balanced_)speech_training_manifest.json`
+* `(balanced_)speech_validation_manifest.json`
+
+Each line is a training example. `audio_filepath` contains path to the wav file, `duration` is duration in seconds, `offset` is offset in seconds, and `label` is label (class):
+
+.. code-block:: bash
+
+    {"audio_filepath": "<absolute path to dataset>/two/8aa35b0c_nohash_0.wav", "duration": 0.63, "label": "speech", "offset": 0.0}
+    {"audio_filepath": "<absolute path to dataset>/Emergency_vehicle/id_58368 simambulance.wav", "duration": 0.63, "label": "background", "offset": 4.0}
+
+
+
+Preparing Custom Speech Classification Data
+--------------------------------------------
+
+Preparing Custom Speech Classification Data is almost identical to `Preparing Custom ASR Data <../datasets.html#preparing-custom-asr-data>`__.
+
+Instead of :code:`text` entry in manifest, you need :code:`label` to determine class of this sample
+
+
+Tarred Datasets
+---------------
+
+Similarly to ASR, you can tar your audio files and use ASR Dataset class ``TarredAudioToClassificationLabelDataset`` (corresponding to the ``AudioToClassificationLabelDataset``) for this case.
+
+If you would like to use tarred dataset, have a look at `ASR Tarred Datasets <../datasets.html#tarred-datasets>`__.

tts/NeMo/docs/source/asr/speech_classification/intro.rst

@@ -0,0 +1,47 @@
+Speech Classification
+==================================
+Speech Classification refers to a set of tasks or problems of getting a program to automatically classify input utterance or audio segment into categories, 
+such as Speech Command Recognition (multi-class), Voice Activity Detection (binary or multi-class), and Audio Sentiment Classification (typically multi-class), etc.
+
+**Speech Command Recognition** is the task of classifying an input audio pattern into a discrete set of classes. 
+It is a subset of Automatic Speech Recognition, sometimes referred to as Key Word Spotting, in which a model is constantly analyzing speech patterns to detect certain "command" classes. 
+Upon detection of these commands, a specific action can be taken by the system. 
+It is often the objective of command recognition models to be small and efficient so that they can be deployed onto low-power sensors and remain active for long durations of time.
+
+
+**Voice Activity Detection (VAD)** also known as speech activity detection or speech detection, is the task of predicting which parts of input audio contain speech versus background noise.
+It is an essential first step for a variety of speech-based applications including Automatic Speech Recognition. 
+It serves to determine which samples to be sent to the model and when to close the microphone.
+
+
+The full documentation tree is as follows:
+
+.. toctree::
+   :maxdepth: 8
+
+   models
+   datasets
+   results
+   configs
+
+
+Resource and Documentation Guide
+--------------------------------
+
+Hands-on speech classification tutorial notebooks can be found under ``<NeMo_git_repo>/tutorials/asr/``.
+There are training and offline & online microphone inference tutorials for Speech Command Detection and Voice Activity Detection tasks.
+This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
+
+If you are looking for information about a particular Speech Classification model or would like to find out more about the model
+architectures available in the `nemo_asr` collection, check out the :doc:`Models <./models>` page.
+
+Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
+NeMo includes preprocessing scripts for several common ASR datasets, and this page contains instructions on running
+those scripts.
+It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
+
+Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
+of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
+
+Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
+:doc:`Configuration Files <./configs>` page.

tts/NeMo/docs/source/asr/speech_classification/models.rst

@@ -0,0 +1,71 @@
+Models
+======
+
+This page gives a brief overview of the models that NeMo's Speech Classification collection currently supports.
+For Speech Classification, we support Speech Command (Keyword) Detection and Voice Activity Detection (VAD).
+
+Each of these models can be used with the example ASR scripts (in the ``<NeMo_git_root>/examples/asr`` directory) by
+specifying the model architecture in the config file used.
+Examples of config files for each model can be found in the ``<NeMo_git_root>/examples/asr/conf`` directory.
+
+For more information about the config files and how they should be structured, see the :doc:`./configs` page.
+
+Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found on the :doc:`./results` page.
+You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets.
+The Checkpoints page also contains benchmark results for the available ASR models.
+
+.. _MatchboxNet_model:
+
+MatchboxNet (Speech Commands) 
+------------------------------
+
+MatchboxNet :cite:`sc-models-matchboxnet` is an end-to-end neural network for speech command recognition based on `QuartzNet <../models.html#QuartzNet>`__.
+
+Similarly to QuartzNet, the MatchboxNet family of models are denoted as MatchBoxNet_[BxRxC] where B is the number of blocks, and R is the number of convolutional sub-blocks within a block, and C is the number of channels. Each sub-block contains a 1-D *separable* convolution, batch normalization, ReLU, and dropout:
+
+    .. image:: images/matchboxnet_vertical.png
+        :align: center
+        :alt: MatchboxNet model
+        :scale: 50%
+
+It can reach state-of-the art accuracy on the Google Speech Commands dataset while having significantly fewer parameters than similar models. 
+The `_v1` and `_v2` are denoted for models trained on `v1` (30-way classification) and `v2` (35-way classification) datasets; 
+And we use _subset_task to represent (10+2)-way subset (10 specific classes + other remaining classes + silence) classification task.
+
+MatchboxNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecClassificationModel` class.
+
+.. note::
+  For model details and deep understanding about Speech Command Detedction training, inference, finetuning and etc., 
+  please refer to  ``<NeMo_git_root>/tutorials/asr/03_Speech_Commands.ipynb`` and ``<NeMo_git_root>/tutorials/asr/04_Online_Offline_Speech_Commands_Demo.ipynb``.
+
+
+
+.. _MarbleNet_model:
+
+MarbleNet (VAD) 
+------------------
+
+MarbleNet :cite:`sc-models-marblenet` an end-to-end neural network for speech command recognition based on :ref:`MatchboxNet_model`, 
+
+Similarly to MatchboxNet, the MarbleNet family of models are denoted as MarbleNet_[BxRxC] where B is the number of blocks, and R is the number of convolutional sub-blocks within a block, and C is the number of channels. Each sub-block contains a 1-D *separable* convolution, batch normalization, ReLU, and dropout:
+
+    .. image:: images/marblenet_vertical.png
+        :align: center
+        :alt: MarbleNet model
+        :scale: 30%
+
+It can reach state-of-the art performance on the difficult `AVA speech dataset <https://research.google.com/ava/download.html#ava_speech_download>`_ while having significantly fewer parameters than similar models even training on simple data.
+MarbleNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecClassificationModel` class.
+
+.. note::
+  For model details and deep understanding about VAD training, inference, postprocessing, threshold tuning and etc., 
+  please refer to  ``<NeMo_git_root>/tutorials/asr/06_Voice_Activiy_Detection.ipynb`` and ``<NeMo_git_root>/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb``.
+
+
+References
+----------------
+
+.. bibliography:: ../asr_all.bib
+    :style: plain
+    :labelprefix: SC-MODELS
+    :keyprefix: sc-models-

tts/NeMo/docs/source/asr/speech_classification/results.rst

@@ -0,0 +1,105 @@
+Checkpoints
+===========
+
+There are two main ways to load pretrained checkpoints in NeMo:
+
+* Using the :code:`restore_from()` method to load a local checkpoint file (`.nemo`), or
+* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
+
+See the following sections for instructions and examples for each.
+
+Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning.
+For resuming an unfinished training experiment, please use the experiment manager to do so by setting the
+``resume_if_exists`` flag to True.
+
+Loading Local Checkpoints
+-------------------------
+
+NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
+You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
+
+If you have a local ``.nemo`` checkpoint that you'd like to load, simply use the :code:`restore_from()` method:
+
+.. code-block:: python
+
+  import nemo.collections.asr as nemo_asr
+  model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
+
+Where the model base class is the ASR model class of the original checkpoint, or the general `ASRModel` class.
+
+
+Transcribing/Inference
+-----------------------
+   
+The audio files should be 16KHz monochannel wav files.
+
+**Transcribe speech command segment:**
+  
+You may perform inference and transcribe a sample of speech after loading the model by using its 'transcribe()' method:
+
+.. code-block:: python 
+
+  mbn_model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="<MODEL_NAME>")
+  mbn_model.transcribe([list of audio files],  batch_size=BATCH_SIZE, logprobs=False) 
+
+Setting argument ``logprobs`` to True would return the log probabilities instead of transcriptions. You may find more details in `Modules <../api.html#modules>`__.
+
+Learn how to fine tune on your own data or on subset classes in ``<NeMo_git_root>/tutorials/asr/03_Speech_Commands.ipynb``
+f
+
+**Run VAD inference:**
+
+.. code-block:: bash 
+
+  python examples/speaker_recognition/vad_infer.py  --vad_model="vad_marblenet" --dataset=<FULL PATH OF MANIFEST TO BE PERFORMED INFERENCE ON> --out_dir='frame/demo' --time_length=0.63
+
+Have a look at scripts under ``<NeMo-git-root>/scripts/voice_activity_detection`` for posterior processing and threshold tuning.
+
+
+NGC Pretrained Checkpoints
+--------------------------
+
+The Speech Classification collection has checkpoints of several models trained on various datasets for a variety of tasks.
+These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
+The model cards on NGC contain more information about each of the checkpoints available.
+
+The tables below list the Speech Classification models available from NGC, and the models can be accessed via the
+:code:`from_pretrained()` method inside the ASR Model class.
+
+In general, you can load any of these models with code in the following format.
+
+.. code-block:: python
+
+  import nemo.collections.asr as nemo_asr
+  model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="<MODEL_NAME>")
+
+Where the model name is the value under "Model Name" entry in the tables below.
+
+For example, to load the MatchboxNet3x2x64_v1 model for speech command detection, run:
+
+.. code-block:: python
+
+  model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="commandrecognition_en_matchboxnet3x2x64_v1")
+
+You can also call :code:`from_pretrained()` from the specific model class (such as :code:`EncDecClassificationModel`
+for MatchboxNet and MarbleNet) if you will need to access specific model functionality.
+
+If you would like to programatically list the models available for a particular base class, you can use the
+:code:`list_available_models()` method.
+
+.. code-block:: python
+
+  nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
+
+
+Speech Classification Models
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. tabularcolumns:: 30 30 40
+
+.. csv-table::
+   :file: data/classification_results.csv
+   :header-rows: 1
+   :class: longtable
+   :widths: 1 1 1
+