Introduce code from Rust monorepo. This is a mess

tts/NeMo/docs/source/asr/speaker_diarization/api.rst

@@ -0,0 +1,17 @@
+NeMo Speaker Diarization API
+=============================
+
+
+Model Classes
+-------------
+.. autoclass:: nemo.collections.asr.models.ClusteringDiarizer
+    :show-inheritance:
+    :members: init_speaker_model, set_vad_model, init_vad_model, setup_vad_test_data, setup_spkr_test_data, run_vad, extract_embeddings, path2audio_files_to_manifest, diarize, make_nemo_file_from_folder, save_to, unpack_nemo_file, restore_from
+
+
+Mixins
+------
+
+.. autoclass:: nemo.collections.asr.parts.mixins.mixins.DiarizationMixin
+    :show-inheritance:
+    :members:

tts/NeMo/docs/source/asr/speaker_diarization/configs.rst

@@ -0,0 +1,65 @@
+NeMo Speaker Diarization Configuration Files
+============================================
+
+Since speaker diarization model here is not a fully-trainble End-to-End model but an inference pipeline, we use **diarizer** instead of **model** which is used in other tasks.
+
+The diarizer section will generally require information about the dataset(s) being
+used, models used in this pipline, as well as inference related parameters such as postprocessing of each models.
+The sections on this page cover each of these in more detail.
+
+
+Example configuration files for speaker diarization can be found in the
+``<NeMo_git_root>/examples/speaker_recognition/conf/speaker_diarization.yaml>``
+
+.. note::
+  For model details and deep understanding about configs, finetuning, tuning threshold, and evaluation, 
+  please refer to ``<NeMo_git_root>/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb``;
+  for other applications such as possible integration with ASR, have a look at ``<NeMo_git_root>/tutorials/speaker_recognition/ASR_with_SpeakerDiarization.ipynb``.
+
+
+Dataset Configuration
+-----------------------
+
+In contrast to other ASR related tasks or models in NeMo, speaker diarization supported in NeMo is a modular inference pipeline.
+Datasets here denotes the data you would like to perform speaker diarization on. 
+
+An example Speaker Diarization dataset configuration could look like:
+
+.. code-block:: yaml
+
+  diarizer:
+    oracle_num_speakers: ??? # if number of speakers is known input it or pass null if not known. Accepts int or path to file containing uniq-id and num of speakers of that session
+    max_num_speakers: 8 # max number of speakers for each recording. If oracle num speakers is passed this value is ignored
+    out_dir: ??? 
+    paths2audio_files: ??? # file containing paths to audio files for diarzation
+    path2groundtruth_rttm_files: null # (Optional) file containing paths to rttm files (this can be passed if we need to calculate DER rate based on ground truth rttm files).
+    ...
+    
+.. note::
+  We expect audio and the corresponding RTTM to have the same base name and the name should be unique.
+
+
+Diarizer Architecture Configurations
+-------------------------------------
+
+.. code-block:: yaml
+
+  diarizer:
+  ...
+    vad:
+      model_path: null #.nemo local model path or pretrained model name or none
+      window_length_in_sec: 0.15
+      shift_length_in_sec: 0.01
+      threshold: 0.5 # tune threshold on dev set. Check <NeMo_git_root>/scripts/voice_activity_detection/vad_tune_threshold.py
+      vad_decision_smoothing: True
+      smoothing_params:
+        method: "median" 
+        overlap: 0.875
+
+    speaker_embeddings:
+      oracle_vad_manifest: null # leave it null if to perform diarization with above VAD model else path to manifest file genrerated as shown in Datasets section
+      model_path: ??? #.nemo local model path or pretrained model name
+      window_length_in_sec: 1.5 # for better performace try with 3 sec and 1.5 sec
+      shift_length_in_sec: 0.75 # for better performace try with 1.5 sec and 0.75 sec
+
+

tts/NeMo/docs/source/asr/speaker_diarization/data/diarization_results.csv

@@ -0,0 +1,4 @@
+Model Name,Model Base Class,Model Card
+vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
+vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
+speakerdiarization_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet"

tts/NeMo/docs/source/asr/speaker_diarization/datasets.rst

@@ -0,0 +1,72 @@
+Datasets
+========
+Check out page :doc:`Speech Classification Datasets <../speech_classification/datasets>` and :doc:`Speaker Recogniton Datasets <../speaker_recognition/datasets>` 
+for preparing datasets for training and validating VAD and speaker embedding models respectively.
+
+For Speaker Diarization inference, ``diarizer`` expects following arguments :
+
+  - paths2audio_files
+  - oracle_num_speakers
+  - path2groundtruth_rttm_files [Optional]
+  - oracle_vad_manifest (if vad/rttm files are known)[Optional but suggested for good performace. This avoids FPR and TPR]
+
+paths2audio_files:
+  - a file containing absolute paths to audio files or list of paths to audio files. 
+  For generating a file that contains paths to audio files, you can simply use ``find`` bash command as shown below:
+.. code-block:: bash
+
+  find $PWD/{data_dir} -iname '*.wav' > path_to_audiofiles.txt
+
+.. note::
+  We expect audio and the corresponding RTTM files to have the same base name and the name should be unique (uniq-id).
+
+oracle_num_speakers:
+  - if number of speakers is known input it or pass null if not known. Accepts int or path to file containing uniq-id with num of speakers of that session 
+
+  Sample line (uniq-id `iaaa` session has 2 unique speakers )::
+    
+    iaaa 2
+
+path2groundtruth_rttm_files [Optional]:
+  - To evaluate the diarizer system with known rttm files, One needs to provide a txt file (path2groundtruth_rttm_files) for groundtruth label files.
+  use above mentioned ``find`` command to get all reference rttm files (use '*.rttm' as search pattern)
+
+Each groundtruth label file should be in NIST Rich Transcription Time Marked (RTTM) format. Take one line from a RTTM file for example:
+
+.. code-block:: bash
+
+  SPEAKER TS3012d.Mix-Headset 1 331.573 0.671 <NA> <NA> MTD046ID <NA> <NA>
+
+
+oracle_vad_manifest [Optional but suggested for good performace. This reduces FPR and TPR]:
+  - To perform just oracle diarization, that is taking speech activity time stamps from groundtruth RTTMs or another VAD from wild instead of NeMo VAD output, ``diarizer`` expects an oracle manifest json file that contains paths to audio files with offset for start time and duration of segment.
+
+To prepare an oracle manifest file, use the script from ``scripts`` folder as shown below:
+
+.. code-block:: bash
+
+  python $NeMo/scripts/ --paths2rttm_files=<path2groundtruth_rttm_files.txt> --paths2audio_files=<paths2audio_files.txt> --manifest_file=<output_oracle_manifest_file.json>
+
+Here ``paths2audio_files`` and ``path2groundtruth_rttm_files`` are files containing paths to audio files as shown above.
+
+AMI Meeting Corpus
+------------------
+
+The following are the suggested parameters for getting Speaker Error Rate (SER) of 4.1% on AMI Lapel test set corpus:
+  - diarizer.oracle_num_speakers = 4 (since there are exactly 4 speakers per each Lapel test set session)
+  - diarizer.speaker_embeddings.model_path = ``speakerverification_speakernet`` (This model is trained on voxceleb dataset. ``Use this model for simialr non-telephonic speech datasets``)
+  - diarizer.speaker_embeddings.window_length_in_sec = 3 
+  - diarizer.speaker_embeddings.shift_length_in_sec = 1.5 
+
+Input paths2audio_files, paths2rttm_files and oracle_vad_manifest by following steps as shown above
+
+CallHome LDC97S42 (CH109)
+-------------------------
+
+The following are the suggested parameters for getting Speaker Error Rate (SER) of 5.4% on CH109 set:
+  - diarizer.oracle_num_speakers = 2 (since there are exactly 2 speakers per each ch109 session)
+  - diarizer.speaker_embeddings.model_path = ``speakerdiarization_speakernet`` (This model is trained on voxceleb and telephonic speech Fisher and SWBD. ``Use this model for similar telephonic speech datasets``)
+  - diarizer.speaker_embeddings.window_length_in_sec = 1.5
+  - diarizer.speaker_embeddings.shift_length_in_sec = 0.75
+
+Input paths2audio_files, paths2rttm_files and oracle_vad_manifest by following steps as shown above

tts/NeMo/docs/source/asr/speaker_diarization/intro.rst

@@ -0,0 +1,52 @@
+Speaker Diarization
+==================================
+
+Speaker Diarization (SD) is the task of segmenting audio recordings by speaker labels, that is Who Speaks When?
+
+.. image:: images/sd_pipeline.png
+        :align: center
+        :alt: Speaker Diarization pipeline [todo]
+        :scale: 50%
+
+
+A diarization system consists of a **Voice Activity Detection (VAD)** model to get the time stamps of audio where speech is being spoken while ignoring the background noise and 
+a **Speaker Embeddings** model to get speaker embeddings on speech segments obtained from VAD time stamps. 
+These speaker embeddings would then be clustered into clusters based on number of speakers present in the audio recording.
+
+In NeMo we support both **oracle VAD** and **non-oracle VAD** diarization. 
+
+
+The full documentation tree is as follows:
+
+.. toctree::
+   :maxdepth: 8
+
+   models
+   datasets
+   results
+   configs
+   api
+
+Resource and Documentation Guide
+--------------------------------
+
+Hands-on speaker diarization tutorial notebooks can be found under ``<NeMo_git_root>/tutorials/speaker_recognition/``.
+
+There are tutorials for peformming inference using :ref:`MarbleNet_model` and :ref:`SpeakerNet_model`, 
+and how one can get ASR transcriptions combined with Speaker labels along with voice activity time stamps with NeMo asr collections.
+
+Most of the tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
+
+If you are looking for information about a particular model used for speaker diarization inference, or would like to find out more about the model
+architectures available in the `nemo_asr` collection, check out the :doc:`Models <./models>` page.
+
+Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
+NeMo includes preprocessing scripts for several common ASR datasets, and this page contains instructions on running
+those scripts.
+It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
+
+Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
+of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
+
+Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
+:doc:`Configuration Files <./configs>` page.

tts/NeMo/docs/source/asr/speaker_diarization/models.rst

@@ -0,0 +1,5 @@
+Models
+======
+
+
+Currenlty NeMo's Speaker Diarization pipeline uses `MarbleNet <../speech_classification/models.html#marblenet-vad>`__ model for Voice Activity Detection (VAD) and `SpeakerNet <../speaker_recognition/models.html#speakernet>`__ model for Speaker Embedding Extraction.

tts/NeMo/docs/source/asr/speaker_diarization/results.rst

@@ -0,0 +1,76 @@
+Checkpoints
+===========
+
+There are two main ways to load pretrained checkpoints in NeMo as introduced in `loading ASR checkpoints <../results.html#checkpoints>`__.
+In speaker diarization, the diarizer loads checkpoints that are passed through the config file. For example: 
+
+Loading Local Checkpoints
+---------------------------
+
+Load VAD model
+
+.. code-block:: bash
+
+  pretrained_vad_model='/path/to/vad_marblenet.nemo' # local .nemo or pretrained vad model name
+  ...
+  # pass with hydra config
+  config.diarizer.vad.model_path=pretrained_vad_model
+
+
+Load Speaker Embedding model
+
+.. code-block:: bash
+
+  pretrained_speaker_model='/path/to/speakerdiarization_speakernet.nemo' # local .nemo or pretrained speakernet model name
+  ...
+  # pass with hydra config
+  config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
+
+
+NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
+You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
+
+
+Inference
+----------
+
+.. note::
+  For details and deep understanding, please refer to ``<NeMo_git_root>/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb``.
+
+Check out :doc:`Datasets <./datasets>` for preparing audio files and optional label files.
+
+Run and evaluate speaker diarizer with below command:
+
+.. code-block:: bash
+
+  # Have a look at the instruction inside the script and pass the arguments you might need. 
+  python <NeMo_git_root>/examples/speaker_recognition/speaker_diarize.py 
+
+
+NGC Pretrained Checkpoints
+----------------------------
+
+The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
+These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
+The model cards on NGC contain more information about each of the checkpoints available.
+
+In general, you can load models with model name in the following format, 
+
+.. code-block:: python
+
+  pretrained_vad_model='vad_telephony_marblenet' 
+  pretrained_speaker_model='speakerdiarization_speakernet' 
+  ...
+  config.diarizer.vad.model_path=retrained_vad_model \
+  config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
+
+where the model name is the value under "Model Name" entry in the tables below.
+
+Models for Speaker Diarization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :file: data/diarization_results.csv
+   :align: left
+   :widths: 30, 30, 40
+   :header-rows: 1