storytold/storyteller-ml
2
0
Fork 0
mirror of https://github.com/storytold/storyteller-ml.git synced 2026-10-17 00:23:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Introduce code from Rust monorepo. This is a mess

Browse Source
This commit is contained in:
Brandon Thomas
2022-02-28 05:43:54 -05:00
parent 921fb49f9a
commit ba326bfcfd
1058 changed files with 192192 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tts/NeMo/docs/source/asr/api.rst
+203
View File
@@ -0,0 +1,203 @@
NeMo ASR collection API
=======================
Model Classes
-------------
.. autoclass:: nemo.collections.asr.models.EncDecCTCModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecCTCModelBPE
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTBPEModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecSpeakerLabelModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
Modules
-------
.. autoclass:: nemo.collections.asr.modules.ConvASREncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConvASRDecoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConvASRDecoderClassification
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.SpeakerDecoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConformerEncoder
:show-inheritance:
:members:
Parts
-----
.. autoclass:: nemo.collections.asr.parts.submodules.jasper.JasperBlock
:show-inheritance:
:members:
Mixins
------
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin
:show-inheritance:
:members:
Datasets
--------
Character Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToCharDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToCharWithDursF0Dataset
:show-inheritance:
:members:
Subword Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToBPEDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset
:show-inheritance:
:members:
Audio Preprocessors
-------------------
.. autoclass:: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.AudioToMFCCPreprocessor
:show-inheritance:
:members:
Audio Augmentors
----------------
.. autoclass:: nemo.collections.asr.modules.SpectrogramAugmentation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.CropOrPadSpectrogramAugmentation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.SpeedPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TimeStretchPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.GainPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ImpulsePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ShiftPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.NoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.WhiteNoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.RirAndNoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TranscodePerturbation
:show-inheritance:
:members:
Miscellaneous Classes
---------------------
RNNT Decoding
~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.metrics.rnnt_wer.RNNTDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.metrics.rnnt_wer_bpe.RNNTBPEDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamRNNTInfer
:show-inheritance:
:members:
Hypotheses
~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis
:show-inheritance:
:no-members:
.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses
:show-inheritance:
:no-members:
tts/NeMo/docs/source/asr/asr_all.bib
+980
View File
@@ -0,0 +1,980 @@
@article{matchboxnet,
title={{MatchboxNet}: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition},
author={Majumdar, Somshubra and Ginsburg, Boris},
journal={Proc. Interspeech 2020},
year={2020}
}
@article{marblenet,
title={MarbleNet: Deep 1D Time-Channel Separable Convolutional Neural Network for Voice Activity Detection},
author={Jia, Fei and Majumdar, Somshubra and Ginsburg, Boris},
journal={arXiv preprint arXiv:2010.13886},
year={2020}
}
@inproceedings{panayotov2015librispeech,
title={Librispeech: an ASR corpus based on public domain audio books},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on},
pages={5206--5210},
year={2015},
organization={IEEE}
}
@article{luong17,
author = {Minh{-}Thang Luong and Eugene Brevdo and Rui Zhao},
title = {Neural Machine Translation (seq2seq) Tutorial},
journal = {https://github.com/tensorflow/nmt},
year = {2017},
}
@INPROCEEDINGS{LaurentSeqWiseBN,
author={C. {Laurent} and G. {Pereyra} and P. {Brakel} and Y. {Zhang} and Y. {Bengio}},
booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={Batch normalized recurrent neural networks},
year={2016},
volume={},
number={},
pages={2657-2661},
keywords={feedforward neural nets;learning (artificial intelligence);recurrent neural nets;speech recognition;batch normalized recurrent neural networks;RNN;sequential data;long-term dependency learning;convergence rate improvement;intermediate representation normalization;feedforward neural networks;speech recognition task;language modeling;training criterion;Training;Recurrent neural networks;Convergence;Speech recognition;Computer architecture;Speech;batch normalization;RNN;LSTM;optimization},
doi={10.1109/ICASSP.2016.7472159},
ISSN={2379-190X},
month={March},}
@article{graves2005,
author = {Alex Graves and Jurgen Schmidhuber},
title = {Framewise phoneme classification with bidirectional LSTM and other neural network architectures},
journal = {Neural Networks, vol. 18},
pages={602–-610},
year = {2005},
}
@inproceedings{graves2006,
title={Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks},
author={Graves, Alex and Fern{\'a}ndez, Santiago and Gomez, Faustino and Schmidhuber, J{\"u}rgen},
booktitle={Proceedings of the 23rd international conference on Machine learning},
pages={369--376},
year={2006},
organization={ACM}
}
@article{li2019jasper,
title={Jasper: An End-to-End Convolutional Neural Acoustic Model},
author={Li, Jason and Lavrukhin, Vitaly and Ginsburg, Boris and Leary, Ryan and Kuchaiev, Oleksii and Cohen, Jonathan M and Nguyen, Huyen and Gadde, Ravi Teja},
journal={arXiv preprint arXiv:1904.03288},
year={2019}
}
@misc{ardila2019common,
title={Common Voice: A Massively-Multilingual Speech Corpus},
author={Rosana Ardila and Megan Branson and Kelly Davis and Michael Henretty and Michael Kohler and Josh Meyer and Reuben Morais and Lindsay Saunders and Francis M. Tyers and Gregor Weber},
year={2019},
eprint={1912.06670},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@article{graves2012,
title={Sequence Transduction with Recurrent Neural Networks},
author={Graves, Alex},
journal={arXiv preprint arXiv:1211.3711},
year={2012}
}
@article{graves2013,
title={Generating sequences with recurrent neural networks},
author={Graves, Alex},
journal={arXiv preprint arXiv:1308.0850},
year={2013}
}
@article{sergeev2018horovod,
title={Horovod: fast and easy distributed deep learning in TensorFlow},
author={Sergeev, Alexander and Del Balso, Mike},
journal={arXiv preprint arXiv:1802.05799},
year={2018}
}
@misc{NVVolta,
title = {NVIDIA TESLA V100 GPU ARCHITECTURE},
howpublished = {\url{http://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf}},
note = {Accessed: 2018-10-09}
}
@article{NVTuring,
title = {NVIDIA TURING GPU ARCHITECTURE},
howpublished = {\url{https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf}},
author = {NVIDIA},
year = {2018},
note = {Accessed: 2018-10-09}
}
@misc{Rygaard2015,
title = {Using Synthesized Speech to Improve Speech Recognition for Low-Resource Languages},
author = {Luise Valentin Rygaard},
howpublished = {\url{https://parasol.tamu.edu/dreu2015/Rygaard/report.pdf}},
year = {2015},
}
@misc{OpenSeq2Seq,
title = {OpenSeq2Seq: extensible toolkit for distributed and mixed precision training of sequence-to-sequence models},
author = {Kuchaiev, Oleksii and Ginsburg, Boris and Gitman, Igor and Lavrukhin,Vitaly and Case, Carl and Micikevicius, Paulius},
howpublished = {\url{https://arxiv.org/abs/1805.10387}},
year = {2018},
}
@misc{MPGuide,
title = {Training with Mixed Precision},
howpublished = {\url{http://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/}},
note = {Accessed: 2018-04-06},
}
@misc{Mozilla,
title = {Mozilla: A Journey to less than 10\% Word Error Rate},
howpublished = {\url{https://hacks.mozilla.org/2017/11/a-journey-to-10-word-error-rate/}},
note = {Accessed: 2018-04-06},
}
@article{Waibel1989,
title={A time-delay neural network architecture for isolated word recognition},
author={Waibel, Alexander, and Hanazawa, Toshiyki and Hinton,Geoffrey and Shirano, Kiyohiro and Lang, Kevin },
journal={IEEE Trans. on Acoustics, Speech and Signal Processing},
year={1989}
}
@article{Lang1990,
title={A time-delay neural network architecture for isolated word recognition},
author={Lang, Kevin and Waibel, Alexander, and Hinton,Geoffrey },
journal={Neural Networks},
year={1990}
}
@book{Bengio1996,
Author = {Bengio, Y.},
Publisher = {International Thomson Computer Press},
Title = {Neural Networks for Speech and Sequence Recognition},
Year = {1996}
}
@article{Bengio1992,
title={Global optimization of a neural network-hidden Markov model hybrid},
author={Bengio, Y., and De Mori, R., and Flammia, G., and Kompe, R. },
journal={IEEE Transactions on Neural Networks, 3(2), 252–259},
year={1992}
}
@article{Bourlard1994,
title={Connectionist speech recognition: a hybrid approach},
author={Bourlard, H. A. and Morgan, N.},
journal={volume 247 Springer },
year={1994}
}
@article{srivastava14a,
author = {Nitish Srivastava, and Geoffrey Hinton, and Alex Krizhevsky, and Ilya Sutskever, and Ruslan Salakhutdinov},
title = {Dropout: A Simple Way to Prevent Neural Networks from Overfitting},
journal = {Journal of Machine Learning Research},
year = {2014},
volume = {15},
pages = {1929-1958},
url = {http://jmlr.org/papers/v15/srivastava14a.html}
}
@article{Hinton2012,
title={Deep Neural Networks for Acoustic Modeling in Speech Recognition},
author={ Hinton,Geoffrey and Deng, Li and Yu, Dong and Dahl,George and Mohamed,Abdel-rahman and Jaitly, Navdeep and Senior, Andrew and Vanhoucke, Vincent and Nguyen, Patrick and Kingsbury, Brian and Sainath, Tara},
journal={IEEE Signal Processing Magazine},
year={2012}
}
@article{Graves2014,
title={Towards End-to-End Speech Recognition with Recurrent Neural Networks},
author={Graves, Alex and Jaitly, Navdeep},
journal={International Conference on Machine Learning},
year={2014}
}
@article{Chorowski2014,
title={End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results},
author={ Chorowski, Jan, and Bahdanau, Dzmitry , and Cho, Kyunghyun , and Bengio, Yoshua },
journal={Neural Information Processing Systems: Workshop Deep Learning and Representation Learning Workshop },
year={2014}
}
@article{Sak2014,
title={Long short-term memory recurrent neural network architectures for large scale acoustic modeling},
author={Sak, Hasim and Senior, Andrew and Beaufays, Francoise },
journal={Interspeech 2014},
year={2014}
}
@article{Ko2015,
title={Audio Augmentation for Speech Recognition},
author={Tom, Ko and Vijayaditya, Peddinti and Daniel, Povey
and Sanjeev, Khudanpur },
journal={Interspeech 2015},
year={2015}
}
@article{Tjandra2017,
title={Listening while Speaking: Speech Chain by Deep Learning},
author={Andros, Tjandra and Sakriani, Sakti and Satoshi, Nakamura },
journal={ASRU 2017},
year={2017}
}
@article{Tjandra2018,
title={Machine Speech Chain with One-shot Speaker Adaptation},
author={Andros, Tjandra and Sakriani, Sakti and Satoshi, Nakamura },
journal={Interspeech 2018},
year={2018}
}
@article{bahdanau2014neural,
title={Neural machine translation by jointly learning to align and translate},
author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
journal={arXiv preprint arXiv:1409.0473},
year={2014}
}
@article{cho2014learning,
title={Learning phrase representations using RNN encoder-decoder for statistical machine translation},
author={Cho, Kyunghyun and Van Merri{\"e}nboer, Bart and Gulcehre, Caglar and Bahdanau, Dzmitry and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua},
journal={arXiv preprint arXiv:1406.1078},
year={2014}
}
@article{rush2015neural,
title={A neural attention model for abstractive sentence summarization},
author={Rush, Alexander M and Chopra, Sumit and Weston, Jason},
journal={arXiv preprint arXiv:1509.00685},
year={2015}
}
@article{micikevicius2017mixed,
title={Mixed precision training},
author={Micikevicius, Paulius and Narang, Sharan and Alben, Jonah and Diamos, Gregory and Elsen, Erich and Garcia, David and Ginsburg, Boris and Houston, Michael and Kuchaev, Oleksii and Venkatesh, Ganesh and others},
journal={arXiv preprint arXiv:1710.03740},
year={2017}
}
@ARTICLE{Britz:2017,
author = {{Britz}, Denny and {Goldie}, Anna and {Luong}, Thang and {Le}, Quoc},
title = "{Massive Exploration of Neural Machine Translation Architectures}",
journal = {ArXiv e-prints arXiv:1703.03906},
archivePrefix = "arXiv",
eprinttype = {arxiv},
eprint = {1703.03906},
primaryClass = "cs.CL",
keywords = {Computer Science - Computation and Language},
year = 2017,
month = mar,
}
@inproceedings{vaswani2017attention,
title={Attention is all you need},
author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia},
booktitle={Advances in Neural Information Processing Systems},
pages={6000--6010},
year={2017}
}
@inproceedings{abadi2016tensorflow,
title={TensorFlow: A System for Large-Scale Machine Learning.},
author={Abadi, Mart{\'\i}n and Barham, Paul and Chen, Jianmin and Chen, Zhifeng and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Irving, Geoffrey and Isard, Michael and others},
booktitle={OSDI},
volume={16},
pages={265--283},
year={2016}
}
@article{tensor2tensor,
author = {Ashish Vaswani and Samy Bengio and Eugene Brevdo and Francois Chollet and Aidan N. Gomez and Stephan Gouws and Llion Jones and \L{}ukasz Kaiser and Nal Kalchbrenner and Niki Parmar and Ryan Sepassi and
Noam Shazeer and Jakob Uszkoreit},
title = {Tensor2Tensor for Neural Machine Translation},
journal = {CoRR},
volume = {abs/1803.07416},
year = {2018},
url = {http://arxiv.org/abs/1803.07416},
}
@article{gehring2017convs2s,
author = {Gehring, Jonas, and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N},
title = "{Convolutional Sequence to Sequence Learning}",
journal = {ArXiv e-prints arXiv:1705.03122},
archivePrefix = "arXiv",
eprinttype = {arxiv},
eprint = {1705.03122},
primaryClass = "cs.CL",
keywords = {Computer Science - Computation and Language},
year = 2017,
month = May,
}
@inproceedings{chan2015,
title={Listen, attend and spell},
author={Chan, William and Jaitly, Navdeep and Le, Quoc V and Vinyals, Oriol},
booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on},
pages={5206--5210},
year={2016},
organization={IEEE}
}
@inproceedings{xu2015show,
title={Show, attend and tell: Neural image caption generation with visual attention},
author={Xu, Kelvin and Ba, Jimmy and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhudinov, Ruslan and Zemel, Rich and Bengio, Yoshua},
booktitle={International Conference on Machine Learning},
pages={2048--2057},
year={2015}
}
@incollection{Sutskever2014,
title = {Sequence to Sequence Learning with Neural Networks},
author = {Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V},
booktitle = {Advances in Neural Information Processing Systems 27},
editor = {Z. Ghahramani and M. Welling and C. Cortes and N. D. Lawrence and K. Q. Weinberger},
pages = {3104--3112},
year = {2014},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf}
}
@article{DeepSpeech2014,
title = {Deep Speech: Scaling up end-to-end speech recognition},
author = {Awni Y. Hannun and Carl Case and Jared Casper and Bryan Catanzaro and Greg Diamos and Erich Elsen and Ryan Prenger and Sanjeev Satheesh and Shubho Sengupta and Adam Coates and Andrew Y. Ng},
journal = {CoRR},
volume = {abs/1412.5567},
year = {2014},
url = {http://arxiv.org/abs/1412.5567},
archivePrefix = {arXiv},
eprint = {1412.5567},
timestamp = {Mon, 13 Aug 2018 16:48:07 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/HannunCCCDEPSSCN14},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{DeepSpeech2,
author = {Amodei, Dario and Ananthanarayanan, Sundaram and Anubhai, Rishita and Bai, Jingliang and Battenberg, Eric and Case, Carl and Casper, Jared and Catanzaro, Bryan and Cheng, Qiang and Chen, Guoliang and Chen, Jie and Chen, Jingdong and Chen, Zhijie and Chrzanowski, Mike and Coates, Adam and Diamos, Greg and Ding, Ke and Du, Niandong and Elsen, Erich and Engel, Jesse and Fang, Weiwei and Fan, Linxi and Fougner, Christopher and Gao, Liang and Gong, Caixia and Hannun, Awni and Han, Tony and Johannes, Lappi Vaino and Jiang, Bing and Ju, Cai and Jun, Billy and LeGresley, Patrick and Lin, Libby and Liu, Junjie and Liu, Yang and Li, Weigao and Li, Xiangang and Ma, Dongpeng and Narang, Sharan and Ng, Andrew and Ozair, Sherjil and Peng, Yiping and Prenger, Ryan and Qian, Sheng and Quan, Zongfeng and Raiman, Jonathan and Rao, Vinay and Satheesh, Sanjeev and Seetapun, David and Sengupta, Shubho and Srinet, Kavya and Sriram, Anuroop and Tang, Haiyuan and Tang, Liliang and Wang, Chong and Wang, Jidong and Wang, Kaifu and Wang, Yi and Wang, Zhijian and Wang, Zhiqian and Wu, Shuang and Wei, Likai and Xiao, Bo and Xie, Wen and Xie, Yan and Yogatama, Dani and Yuan, Bin and Zhan, Jun and Zhu, Zhenyao},
title = {Deep Speech 2: End-to-end Speech Recognition in English and Mandarin},
booktitle = {Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48},
series = {ICML'16},
year = {2016},
location = {New York, NY, USA},
pages = {173--182},
numpages = {10},
url = {http://dl.acm.org/citation.cfm?id=3045390.3045410},
acmid = {3045410},
publisher = {JMLR.org},
}
@inproceedings{prabhavalkar2017comparison,
title={A comparison of sequence-to-sequence models for speech recognition},
author={Prabhavalkar, Rohit and Rao, Kanishka and Sainath, Tara N and Li, Bo and Johnson, Leif and Jaitly, Navdeep},
booktitle={Proc. Interspeech},
pages={939--943},
year={2017}
}
@article{chiu2017state,
title={State-of-the-art speech recognition with sequence-to-sequence models},
author={Chiu, Chung-Cheng and Sainath, Tara N and Wu, Yonghui and Prabhavalkar, Rohit and Nguyen, Patrick and Chen, Zhifeng and Kannan, Anjuli and Weiss, Ron J and Rao, Kanishka and Gonina, Katya and others},
journal={arXiv preprint arXiv:1712.01769},
year={2017}
}
@misc{NVMixed,
title = {{NVIDA's Mixed-Precision Training - TensorFlow example}},
howpublished = {\url{https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/#example_tensorflow}},
author={NVIDIA},
note = {Accessed: 2018-10-09},
year={2018}
}
@article{gehring2017,
title={Convolutional sequence to sequence learning},
author={Gehring, Jonas and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N},
journal={arXiv preprint arXiv:1705.03122},
year={2017}
}
@article{collobert2016,
title={Wav2letter: an end-to-end convnet-based speech recognition system},
author={Collobert, Ronan and Puhrsch, Christian and Synnaeve, Gabriel},
journal={arXiv preprint arXiv:1609.03193},
year={2016}
}
@inproceedings{Zhang2016,
author={Ying Zhang and Mohammad Pezeshki and Philémon Brakel and Saizheng Zhang and César Laurent and Yoshua Bengio and Aaron Courville},
title={Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-1446},
url={http://dx.doi.org/10.21437/Interspeech.2016-1446},
pages={410--414}
}
@inproceedings{Zhang2017,
title={Very deep convolutional networks for end-to-end speech recognition},
author={Zhang, Yu, and Chan, William, and Jaitly, Navdeep},
booktitle={Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on},
year={2017},
organization={IEEE}
}
@article{Wang2017,
title={Tacotron: Towards End-to-End Speech Synthesis},
author={ Wang, Yuxuan, and Skerry-Ryan, RJ, and Stanton, Daisy and Wu, Yonghui and Weiss, Ron, and Jaitly, Navdeep and Yang, Zongheng and Xiao, Ying and Chen,Zhifeng and Bengio, Samy and Le, Quoc and Agiomyrgiannakis, Yannis and Clark,Rob and Saurous, Rif A.},
journal={arXiv preprint arXiv:1703.10135},
year={2017}
}
@inproceedings{shen2018natural,
title={Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions},
author={Shen, Jonathan and Pang, Ruoming and Weiss, Ron J and Schuster, Mike and Jaitly, Navdeep and Yang, Zongheng and Chen, Zhifeng and Zhang, Yu and Wang, Yuxuan and Skerrv-Ryan, Rj and others},
booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages={4779--4783},
year={2018},
organization={IEEE}
}
@article{griffin1984signal,
title={Signal estimation from modified short-time Fourier transform},
author={Griffin, Daniel and Lim, Jae},
journal={IEEE Transactions on Acoustics, Speech, and Signal Processing},
volume={32},
number={2},
pages={236--243},
year={1984},
publisher={IEEE}
}
@misc{ito2017lj,
title={The LJ speech dataset},
author={Ito, Keith and others},
year={2017}
}
@misc{mailabs,
title = {{The M-AILABS Speech Dataset}},
howpublished = {\url{http://www.m-ailabs.bayern/en/the-mailabs-speech-dataset/}},
author={M-AILABS},
note = {Accessed: 2018-10-09},
year={2018}
}
@article{merity2016pointer,
title={Pointer sentinel mixture models},
author={Merity, Stephen and Xiong, Caiming and Bradbury, James and Socher, Richard},
journal={arXiv preprint arXiv:1609.07843},
year={2016}
}
@inproceedings{socher2013recursive,
title={Recursive deep models for semantic compositionality over a sentiment treebank},
author={Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D and Ng, Andrew and Potts, Christopher},
booktitle={Proceedings of the 2013 conference on empirical methods in natural language processing},
pages={1631--1642},
year={2013}
}
@InProceedings{maas-EtAl:2011:ACL-HLT2011,
author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher},
title = {Learning Word Vectors for Sentiment Analysis},
booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
month = {June},
year = {2011},
address = {Portland, Oregon, USA},
publisher = {Association for Computational Linguistics},
pages = {142--150},
url = {http://www.aclweb.org/anthology/P11-1015}
}
@inproceedings{Povey2018SemiOrthogonalLM,
title={Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks},
author={Daniel Povey and Gaofeng Cheng and Yiming Wang and Ke Li and Hainan Xu and Mahsa Yarmohammadi and Sanjeev Khudanpur},
booktitle={Interspeech},
year={2018}
}
@article{CAPIO2017,
author = {Kyu J. Han and Akshay Chandrashekaran and Jungsuk Kim and Ian R. Lane},
title = {The {CAPIO} 2017 Conversational Speech Recognition System},
journal = {CoRR},
volume = {abs/1801.00059},
year = {2018},
url = {http://arxiv.org/abs/1801.00059},
archivePrefix = {arXiv},
eprint = {1801.00059},
timestamp = {Mon, 13 Aug 2018 16:49:10 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1801-00059},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{WaveNet,
author = {A{\"{a}}ron van den Oord and Sander Dieleman and Heiga Zen and Karen Simonyan and Oriol Vinyals and Alex Graves and Nal Kalchbrenner and Andrew W. Senior and Koray Kavukcuoglu},
title = {WaveNet: {A} Generative Model for Raw Audio},
journal = {CoRR},
volume = {abs/1609.03499},
year = {2016},
url = {http://arxiv.org/abs/1609.03499},
archivePrefix = {arXiv},
eprint = {1609.03499},
timestamp = {Mon, 13 Aug 2018 16:49:15 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/OordDZSVGKSK16},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{FacebookGERENGBackTranslation,
author = {Rico Sennrich and Barry Haddow and Alexandra Birch},
title = {Improving Neural Machine Translation Models with Monolingual Data},
journal = {CoRR},
volume = {abs/1511.06709},
year = {2015},
url = {http://arxiv.org/abs/1511.06709},
archivePrefix = {arXiv},
eprint = {1511.06709},
timestamp = {Mon, 13 Aug 2018 16:47:05 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/SennrichHB15a},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{GlobalStyleTokens,
author = {Yuxuan Wang and Daisy Stanton and Yu Zhang and R. J. Skerry{-}Ryan and Eric Battenberg and Joel Shor and Ying Xiao and Fei Ren and Ye Jia and Rif A. Saurous},
title = {Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis},
journal = {CoRR},
volume = {abs/1803.09017},
year = {2018},
url = {http://arxiv.org/abs/1803.09017},
archivePrefix = {arXiv},
eprint = {1803.09017},
timestamp = {Mon, 13 Aug 2018 16:46:53 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1803-09017},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{IoffeS15BatchNorm,
author = {Sergey Ioffe and Christian Szegedy},
title = {Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift},
journal = {CoRR},
volume = {abs/1502.03167},
year = {2015},
url = {http://arxiv.org/abs/1502.03167},
archivePrefix = {arXiv},
eprint = {1502.03167},
timestamp = {Mon, 13 Aug 2018 16:47:06 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/IoffeS15},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{kingma,
author = {Diederik P. Kingma and
Jimmy Ba},
title = {Adam: {A} Method for Stochastic Optimization},
journal = {CoRR},
volume = {abs/1412.6980},
year = {2014},
url = {http://arxiv.org/abs/1412.6980},
archivePrefix = {arXiv},
eprint = {1412.6980},
timestamp = {Mon, 13 Aug 2018 01:00:00 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/KingmaB14},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@incollection{Salimans2016WeightNorm,
title = {Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks},
author = {Salimans, Tim and Kingma, Durk P},
booktitle = {Advances in Neural Information Processing Systems 29},
editor = {D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett},
pages = {901--909},
year = {2016},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf}
}
@article{wu2016google,
title={Google's neural machine translation system: Bridging the gap between human and machine translation},
author={Wu, Yonghui and Schuster, Mike and Chen, Zhifeng and Le, Quoc V and Norouzi, Mohammad and Macherey, Zolfgang and Krikun, Maxim and Cao, Yuan and Gao, Qin and Macherey, Klaus and others},
journal={arXiv preprint arXiv:1609.08144},
year={2016}
}
@inproceedings{opennmt,
author = {Guillaume Klein and Yoon Kim and Yuntian Deng and Jean Senellart and Alexander M. Rush},
title = {OpenNMT: Open-Source Toolkit for Neural Machine Translation},
booktitle = {Proc. ACL},
year = {2017},
url = {https://doi.org/10.18653/v1/P17-4012},
doi = {10.18653/v1/P17-4012}
}
@article{paszke2017automatic,
title={Automatic differentiation in PyTorch},
author={Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory and Yang, Edward and DeVito, Zachary and Lin, Zeming and Desmaison, Alban and Antiga, Luca and Lerer, Adam},
year={2017}
}
@article{yu2014introduction,
title={An introduction to computational networks and the computational network toolkit},
author={Yu, Dong and Eversole, Adam and Seltzer, Mike and Yao, Kaisheng and Huang, Zhiheng and Guenter, Brian and Kuchaiev, Oleksii and Zhang, Yu and Seide, Frank and Wang, Huaming and others},
journal={Microsoft Technical Report MSR-TR-2014--112},
year={2014}
}
@article{nvidia2017v100,
title={V100 GPU architecture. The world’s most advanced data center GPU. Version WP-08608-001\_v1. 1},
author={NVIDIA, Tesla},
journal={NVIDIA. Aug},
pages={108},
year={2017}
}
@article{post2018call,
title={A call for clarity in reporting bleu scores},
author={Post, Matt},
journal={arXiv preprint arXiv:1804.08771},
year={2018}
}
@article{Ba2016LayerNorm,
author = {Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E Hinton},
title = {Layer normalization},
journal = {CoRR},
volume = {abs/1607.06450},
year = {2016},
url = {http://arxiv.org/abs/1607.06450},
archivePrefix = {arXiv},
}
@inproceedings{Dauphin2017GLU,
author = {Dauphin, Yann N. and Fan, Angela and Auli, Michael and Grangier, David},
title = {Language Modeling with Gated Convolutional Networks},
booktitle = {Proceedings of the 34th International Conference on Machine Learning - Volume 70},
series = {ICML'17},
year = {2017},
location = {Sydney, NSW, Australia},
pages = {933--941},
numpages = {9},
url = {http://dl.acm.org/citation.cfm?id=3305381.3305478},
acmid = {3305478},
publisher = {JMLR.org},
}
@incollection{Oord2016PixelCNN,
title = {Conditional Image Generation with PixelCNN Decoders},
author = {van den Oord, Aaron and Kalchbrenner, Nal and Espeholt, Lasse and kavukcuoglu, koray and Vinyals, Oriol and Graves, Alex},
booktitle = {Advances in Neural Information Processing Systems 29},
editor = {D. D. Lee and M. Sugiyama and U. V. Luxburg and I. Guyon and R. Garnett},
pages = {4790--4798},
year = {2016},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/6527-conditional-image-generation-with-pixelcnn-decoders.pdf}
}
@article{he2015,
title={Deep residual learning for image recognition},
author={K. He, and X. Zhang, and S. Ren, and J. Sun},
journal={arXiv preprint arXiv:1512.03385},
year={2015}
}
@article{huang2016,
title={Densely Connected Convolutional Networks},
author={Gao Huang, and Zhuang Liu, and Laurens van der Maaten, and Kilian Q. Weinberger},
journal={arXiv preprint arXiv:1608.06993},
year={2016}
}
@inproceedings{heafield2011kenlm,
title={KenLM: Faster and smaller language model queries},
author={Heafield, Kenneth},
booktitle={Proceedings of the sixth workshop on statistical machine translation},
pages={187--197},
year={2011},
organization={Association for Computational Linguistics}
}
@article{dai2018transformer,
title={Transformer-XL: Language Modeling with Longer-Term Dependency},
author={Dai, Zihang and Yang, Zhilin and Yang, Yiming and Cohen, William W and Carbonell, Jaime and Le, Quoc V and Salakhutdinov, Ruslan},
year={2018},
journal = {CoRR},
volume = {abs/1901.02860},
url = {http://arxiv.org/abs/1901.02860},
archivePrefix = {arXiv},
eprint = {1901.02860},
timestamp = {Fri, 01 Feb 2019 13:39:59 +0100},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1901-02860},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{Saon+2016,
author={George Saon and Tom Sercu and Steven Rennie and Hong-Kwang J. Kuo},
title={The IBM 2016 English Conversational Telephone Speech Recognition System},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-1460},
url={http://dx.doi.org/10.21437/Interspeech.2016-1460},
pages={7--11}
}
@INPROCEEDINGS{Sercu-2016,
author={T. {Sercu} and C. {Puhrsch} and B. {Kingsbury} and Y. {LeCun}},
booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={Very deep multilingual convolutional neural networks for LVCSR},
year={2016},
volume={},
number={},
pages={4955-4959},
keywords={natural language processing;neural nets;speech recognition;very deep multilingual convolutional neural networks;LVCSR;CNN;large vocabulary continuous speech recognition systems;word error rate;Training;Context;Hidden Markov models;Neural networks;Computer architecture;Kernel;Training data;Convolutional Networks;Multilingual;Acoustic Modeling;Speech Recognition;Neural Networks},
doi={10.1109/ICASSP.2016.7472620},
ISSN={2379-190X},
month={March},}
@inproceedings{Sercu+2016,
author={Tom Sercu and Vaibhava Goel},
title={Advances in Very Deep Convolutional Neural Networks for LVCSR},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-1033},
url={http://dx.doi.org/10.21437/Interspeech.2016-1033},
pages={3429--3433}
}
@INPROCEEDINGS{Xiong-2018,
author={W. {Xiong} and L. {Wu} and F. {Alleva} and J. {Droppo} and X. {Huang} and A. {Stolcke}},
booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={The Microsoft 2017 Conversational Speech Recognition System},
year={2018},
volume={},
number={},
pages={5934-5938},
keywords={convolution;feedforward neural nets;natural language processing;speaker recognition;speech processing;language model rescoring step;senone level;switchboard domains;character-based LSTM language models;NIST 2000 switchboard test set;frame level;word-level voting;acoustic model posteriors;dialog session aware LSTM language models;CNN-BLSTM acoustic model;Microsoft 2017 conversational speech recognition system;Acoustics;Error analysis;Training;Speech recognition;Switches;Computational modeling;Context modeling;Conversational speech recognition;CNN;LACE;BLSTM;LSTM-LM;system combination;human parity},
doi={10.1109/ICASSP.2018.8461870},
ISSN={2379-190X},
month={April},}
@inproceedings{zeyer2018improved,
author={Albert Zeyer and Kazuki Irie and Ralf Schlüter and Hermann Ney},
title={Improved Training of End-to-end Attention Models for Speech Recognition},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={7--11},
doi={10.21437/Interspeech.2018-1616},
url={http://dx.doi.org/10.21437/Interspeech.2018-1616}
}
@article{Wav2LetterV2,
author = {Vitaliy Liptchinsky and
Gabriel Synnaeve and
Ronan Collobert},
title = {Letter-Based Speech Recognition with Gated ConvNets},
journal = {CoRR},
volume = {abs/1712.09444},
year = {2017},
url = {http://arxiv.org/abs/1712.09444},
archivePrefix = {arXiv},
eprint = {1712.09444},
timestamp = {Mon, 13 Aug 2018 16:46:33 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1712-09444},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{zeghidour2018,
author = {Neil Zeghidour and
Qiantong Xu and
Vitaliy Liptchinsky and
Nicolas Usunier and
Gabriel Synnaeve and
Ronan Collobert},
title = {Fully Convolutional Speech Recognition},
journal = {CoRR},
volume = {abs/1812.06864},
year = {2018},
url = {http://arxiv.org/abs/1812.06864},
archivePrefix = {arXiv},
eprint = {1812.06864},
timestamp = {Tue, 01 Jan 2019 15:01:25 +0100},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1812-06864},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{Hadian2018,
author={Hossein Hadian and Hossein Sameti and Daniel Povey and Sanjeev Khudanpur},
title={End-to-end Speech Recognition Using Lattice-free MMI},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={12--16},
doi={10.21437/Interspeech.2018-1423},
url={http://dx.doi.org/10.21437/Interspeech.2018-1423}
}
@inproceedings{Tang2018,
author={Jian Tang and Yan Song and Lirong Dai and Ian McLoughlin},
title={Acoustic Modeling with Densely Connected Residual Network for Multichannel Speech Recognition},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={1783--1787},
doi={10.21437/Interspeech.2018-1089},
url={http://dx.doi.org/10.21437/Interspeech.2018-1089}
}
@article{Kurata2017LanguageMW,
title={Language modeling with highway LSTM},
author={Gakuto Kurata and Bhuvana Ramabhadran and George Saon and Abhinav Sethy},
journal={2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
year={2017},
pages={244-251}
}
@inproceedings{Saon2017,
author={George Saon and Gakuto Kurata and Tom Sercu and Kartik Audhkhasi and Samuel Thomas and Dimitrios Dimitriadis and Xiaodong Cui and Bhuvana Ramabhadran and Michael Picheny and Lynn-Li Lim and Bergul Roomi and Phil Hall},
title={English Conversational Telephone Speech Recognition by Humans and Machines},
year=2017,
booktitle={Proc. Interspeech 2017},
pages={132--136},
doi={10.21437/Interspeech.2017-405},
url={http://dx.doi.org/10.21437/Interspeech.2017-405}
}
@inproceedings{Povey+2016,
author={Daniel Povey and Vijayaditya Peddinti and Daniel Galvez and Pegah Ghahremani and Vimal Manohar and Xingyu Na and Yiming Wang and Sanjeev Khudanpur},
title={Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-595},
url={http://dx.doi.org/10.21437/Interspeech.2016-595},
pages={2751--2755}
}
@article{Yang2018,
author = {Xuerui Yang and
Jiwei Li and
Xi Zhou},
title = {A novel pyramidal-FSMN architecture with lattice-free {MMI} for speech
recognition},
journal = {CoRR},
volume = {abs/1810.11352},
year = {2018},
url = {http://arxiv.org/abs/1810.11352},
archivePrefix = {arXiv},
eprint = {1810.11352},
timestamp = {Wed, 31 Oct 2018 14:24:29 +0100},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1810-11352},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{liptchinsky2017based,
title={Letter-Based Speech Recognition with Gated ConvNets},
author={Liptchinsky, Vitaliy and Synnaeve, Gabriel and Collobert, Ronan},
journal={arXiv preprint arXiv:1712.09444},
year={2017}
}
@inproceedings{Weng2018,
author={Chao Weng and Jia Cui and Guangsen Wang and Jun Wang and Chengzhu Yu and Dan Su and Dong Yu},
title={Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={761--765},
doi={10.21437/Interspeech.2018-1030},
url={http://dx.doi.org/10.21437/Interspeech.2018-1030}
}
@INPROCEEDINGS{Battenberg2017,
author={E. {Battenberg} and J. {Chen} and R. {Child} and A. {Coates} and Y. G. Y. {Li} and H. {Liu} and S. {Satheesh} and A. {Sriram} and Z. {Zhu}},
booktitle={2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
title={Exploring neural transducers for end-to-end speech recognition},
year={2017},
volume={},
number={},
pages={206-213},
keywords={recurrent neural nets;speech recognition;Hub500 benchmark;CTC models;speech recognition pipeline;RNN-Transducer models;language model;Seq2Seq models;end-to-end speech recognition;neural transducers;Decoding;Hidden Markov models;Transducers;Task analysis;Speech;Mathematical model;Neural networks},
doi={10.1109/ASRU.2017.8268937},
ISSN={},
month={Dec},
}
@inproceedings{
loshchilov2018,
title={Decoupled Weight Decay Regularization},
author={Ilya Loshchilov and Frank Hutter},
booktitle={International Conference on Learning Representations},
year={2019},
url={https://openreview.net/forum?id=Bkg6RiCqY7},
}
@article{zhang2017ndadam,
author = {Zijun Zhang and Lin Ma and Zongpeng Li and Chuan Wu},
title = {Normalized Direction-preserving Adam},
journal = {arXiv e-prints arXiv:1709.04546},
year = {2017},
}
@article{park2019,
author = {{Park}, Daniel S. and {Chan}, William and {Zhang}, Yu and
{Chiu}, Chung-Cheng and {Zoph}, Barret and {Cubuk}, Ekin D. and
{Le}, Quoc V.},
title = "{SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition}",
journal = {arXiv e-prints},
year = "2019",
eid = {arXiv:1904.08779},
eprint = {1904.08779},
}
@article{novograd2019,
author = {{Ginsburg}, Boris and {Castonguay}, Patrice and {Hrinchuk}, Oleksii and
{Kuchaiev}, Oleksii and {Lavrukhin}, Vitaly and {Leary}, Ryan and
{Li}, Jason and {Nguyen}, Huyen and {Cohen}, Jonathan M.},
title = "{Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks}",
journal = {arXiv e-prints},
year = "2019",
eid = {arXiv:1905.11286},
eprint = {1905.11286},
}
@article{kriman2019quartznet,
title={Quartznet: {Deep} automatic speech recognition with 1d time-channel separable convolutions},
author={Kriman, Samuel and Beliaev, Stanislav and Ginsburg, Boris and Huang, Jocelyn and Kuchaiev, Oleksii and Lavrukhin, Vitaly and Leary, Ryan and Li, Jason and Zhang, Yang},
journal={arXiv preprint arXiv:1910.10261},
year={2019}
}
@misc{itu1988g711,
title={{ITU-T} {G.711} - {Pulse} code modulation ({PCM}) of voice frequencies},
author={ITU-T Geneva Switzerland},
year={1988},
}
@article{han2020contextnet,
title={ContextNet: Improving convolutional neural networks for automatic speech recognition with global context},
author={Han, Wei and Zhang, Zhengdong and Zhang, Yu and Yu, Jiahui and Chiu, Chung-Cheng and Qin, James and Gulati, Anmol and Pang, Ruoming and Wu, Yonghui},
journal={arXiv:2005.03191},
year={2020}
}
@inproceedings{hu2018squeeze,
title={Squeeze-and-excitation networks},
author={Hu, Jie and Shen, Li and Sun, Gang},
booktitle={ICVPR},
year={2018}
}
@article{koluguri2020speakernet,
title={SpeakerNet: 1D Depth-wise Separable Convolutional Network for Text-Independent Speaker Recognition and Verification},
author={Koluguri, Nithin Rao and Li, Jason and Lavrukhin, Vitaly and Ginsburg, Boris},
journal={arXiv preprint arXiv:2010.12653},
year={2020}
}
@article{gulati2020conformer,
title={Conformer: Convolution-augmented transformer for speech recognition},
author={Gulati, Anmol and Qin, James and Chiu, Chung-Cheng and Parmar, Niki and Zhang, Yu and Yu, Jiahui and Han, Wei and Wang, Shibo and Zhang, Zhengdong and Wu, Yonghui and others},
journal={arXiv preprint arXiv:2005.08100},
year={2020}
}
tts/NeMo/docs/source/asr/asr_language_modeling.rst
+292
View File
@@ -0,0 +1,292 @@
#####################
ASR Language Modeling
#####################
Language models have shown to help the accuracy of ASR models. NeMo supports the following two approaches to incorporate language models into the ASR models:
* :ref:`ngram_modeling`
* :ref:`neural_rescoring`
It is possible to use both approaches on the same ASR model.
.. _ngram_modeling:
************************
N-gram Language Modeling
************************
In this approach, an N-gram LM is trained on text data, then it is used in fusion with beam search decoding to find the
best candidates. The beam search decoders in NeMo support language models trained with KenLM library (
`https://github.com/kpu/kenlm <https://github.com/kpu/kenlm>`__).
The beam search decoders and KenLM library are not installed by default in NeMo, and you need to install them to be
able to use beam search decoding and N-gram LM.
Please refer to `scripts/ngram_lm/install_beamsearch_decoders.sh` on how to install them.
NeMo supports both character-based and BPE-based models for N-gram LMs. An N-gram LM can be used with beam search
decoders on top of the ASR models to produce more accurate candidates. The beam search decoder would incorporate
the scores produced by the N-gram LM into its score calculations as the following:
.. code::
final_score = acoustic_score + beam_alpha*lm_score + beam_beta*seq_length
where acoustic_score is the score predicted by the acoustic encoder and lm_score is the one estimated by the LM.
Parameter 'beam_alpha' specifies amount of importance to place on the N-gram language model, and 'beam_beta' is a
penalty term to consider the sequence length in the scores. Larger alpha means more importance on the LM and less
importance on the acoustic model. Negative values for beta will give penalty to longer sequences and make the decoder
to prefer shorter predictions, while positive values would result in longer candidates.
Train N-gram LM
===============
The script to train an N-gram language model with KenLM can be found at
`scripts/asr_language_modeling/ngram_lm/train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
This script would train an N-gram language model with KenLM library which can be used with the beam search decoders
on top of the ASR models. This script supports both character level and BPE level encodings and models which is
detected automatically from the type of the model.
You may train the N-gram model as the following:
.. code::
python train_kenlm.py --nemo_model_file <path to the .nemo file of the model> \
--train_file <path to the training text or JSON manifest file \
--kenlm_bin_path <path to the bin folder of KenLM library> \
--kenlm_model_file <path to store the binary KenLM model> \
--ngram_length <order of N-gram model>
The train file specified by `--train_file` can be a text file or JSON manifest. If the file's extension is anything
other than `.json`, it assumes that data format is plain text. For plain text format, each line should contain one
sample. For JSON manifest file, the file need to contain json formatted samples per each line like this:
.. code::
{"audio_filepath": "/data_path/file1.wav", "text": "The transcript of the audio file."}
It just extracts the `text` field from each line to create the training text file. After the N-gram model is trained,
it is stored at the path specified by `--kenlm_model_file`.
The following is the list of the arguments for the training script:
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| **Argument** | **Type** | **Default** | **Description** |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| nemo_model_file | str | Required | The path of the `.nemo` file of the ASR model. It is needed to extract the tokenizer. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| train_file | str | Required | Path to the training file, it can be a text file or JSON manifest. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| kenlm_model_file | str | Required | The path to store the KenLM binary model file. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| kenlm_bin_path | str | Required | The path to the bin folder of KenLM. It is a folder named `bin` under where KenLM is installed.|
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| ngram_length** | int | Required | Specifies order of N-gram LM. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
| do_lower_case | bool | ``False`` | Whether to make the training text all lower case. |
+------------------+----------+-------------+------------------------------------------------------------------------------------------------+
** Note: Recommend to use 6 as the order of the N-gram model for BPE-based models. Higher orders may need the re-compilation of KenLM to support it.
Evaluate by Beam Search Decoding and N-gram LM
==============================================
NeMo's beam search decoders are capable of using the KenLM's N-gram models to find the best candidates.
The script to evaluate an ASR model with beam search decoding and N-gram models can be found at
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py>`__.
You may evaluate an ASR model as the following:
.. code::
python eval_beamsearch_ngram.py --nemo_model_file <path to the .nemo file of the model> \
--input_manifest <path to the evaluation JSON manifest file \
--kenlm_model_file <path to the binary KenLM model> \
--acoustic_batch_size <batch size for calculating log probabilities> \
--beam_width <list of the beam widths> \
--beam_alpha <list of the beam alphas> \
--beam_width <list of the beam betas> \
--preds_output_folder <optional folder to store the predictions> \
--decoding_mode beam_search_ngram
It can evaluate a model in the three following modes by setting the argument `--decoding_mode`:
* greedy: Just greedy decoding is done, and no beam search decoding is performed.
* beamsearch: The beam search decoding is done but without using the N-gram language model, final results would be equivalent to setting the weight of LM (beam_beta) to zero.
* beamsearch_ngram: The beam search decoding is done with N-gram LM.
The `beamsearch` mode would evaluate by beam search decoding without any language model.
It would report the performances in terms of Word Error Rate (WER) and Character Error Rate (CER). Moreover,
the WER/CER of the model when the best candidate is selected among the candidates is also reported as the best WER/CER.
It can be an indicator of how good the predicted candidates are.
The script would initially load the ASR model and predict the outputs of the model's encoder as log probabilities.
This part would be computed in batches on a device selected by `--device`, which can be CPU (`--device=cpu`) or a
single GPU (`--device=cuda:0`). The batch size of this part can get specified by `--acoustic_batch_size`. You may use
the largest batch size feasible to speed up the step of calculating the log probabilities. You may also use `--use_amp`
to speed up the calculation of log probabilities and make it possible to use larger sizes for `--acoustic_batch_size`.
Currently multi-GPU is not supported for calculating the log probabilities, but using `--probs_cache_file` can help.
It stores the log probabilities produced from the model's encoder into a pickle file so that next time the first step
can get skipped.
The following is the list of the arguments for the evaluation script:
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| **Argument** |**Type**| **Default** | **Description** |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| nemo_model_file | str | Required | The path of the `.nemo` file of the ASR model to extract the tokenizer. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| input_manifest | str | Required | Path to the training file, it can be a text file or JSON manifest. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| kenlm_model_file | str | Required | The path to store the KenLM binary model file. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| preds_output_folder | str | None | The path to an optional folder to store the predictions. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| probs_cache_file | str | None | The cache file for storing the outputs of the model. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| acoustic_batch_size | int | 16 | The batch size to calculate log probabilities. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| use_amp | bool | ``False`` | Whether to use AMP if available to calculate log probabilities. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| device | str | cuda | The device to load the model onto to calculate log probabilities. |
| | | | It can `cpu`, `cuda`, `cuda:0`, `cuda:1`, ... |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| decoding_mode | str | beamsearch_ngram | The decoding scheme to be used for evaluation. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_width | float | Required | The width or list of the widths of the beam search decoding. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_alpha | float | Required | The alpha parameter or list of the alphas for the beam search decoding. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_beta | float | Required | The beta parameter or list of the betas for the beam search decoding. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_batch_size | int | 128 | The batch size to be used for beam search decoding. |
| | | | Larger batch size can be a little faster, but uses larger memory. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
Width of the beam search (`--beam_width`) specifies the number of top candidates/predictions the beam search decoder
would search for. Larger beams result in more accurate but slower predictions.
There is also a tutorial to learn more about evaluating the ASR models with N-gram LM here:
`Offline ASR Inference with Beam Search and External Language Model Rescoring <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/Offline_ASR.ipynb>`_
Hyperparameter Grid Search
--------------------------
Beam search decoding with N-gram LM has three main hyperparameters: `beam_width`, `beam_alpha`, and `beam_beta`.
The accuracy of the model is dependent to the values of these parameters, specially beam_alpha and beam_beta.
You may specify a single or list of values for each of these parameters to perform grid search. It would perform the
beam search decoding on all the combinations of the these three hyperparameters.
For instance, the following set of parameters would results in 2*1*2=4 beam search decodings:
.. code::
python eval_beamsearch_ngram.py ... \
--beam_width 64 128 \
--beam_alpha 1.0 \
--beam_beta 1.0 0.5
.. _neural_rescoring:
****************
Neural Rescoring
****************
In this approach a neural network is used which can gives scores to a candidate. A candidate is the text transcript predicted by the decoder of the ASR model.
The top K candidates produced by the beam search decoding (beam width of K) are given to a neural language model to rank them.
Ranking can be done by a language model which gives a score to each candidate.
This score is usually combined with the scores from the beam search decoding to produce the final scores and rankings.
Train Neural Rescorer
=====================
An example script to train such a language model with Transformer can be found at `examples/nlp/language_modeling/transformer_lm.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/transformer_lm.py>`__.
It trains a TransformerLMModel which can be used as a neural rescorer for an ASR system.
Evaluation
==========
Given a trained TransformerLMModel `.nemo` file, the script available at
`scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__
can be used to re-score beams obtained with ASR model. You need the `.tsv` file containing the candidates produced
by the acoustic model and the beam search decoding to use this script. The candidates can be the result of just the beam
search decoding or the result of fusion with an N-gram LM. You may generate this file by specifying `--preds_output_folder' for
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py>`__.
The neural rescorer would rescore the beams/candidates by using two parameters of `rescorer_alpha` and `rescorer_beta` as the following:
.. code::
final_score = beam_search_score + rescorer_alpha*neural_rescorer_score + rescorer_beta*seq_length
Parameter `rescorer_alpha` specifies amount of importance to place on the neural rescorer model, and `rescorer_beta` is
a penalty term to consider the sequence length in the scores. They have similar effects like the parameters
`beam_alpha` and `beam_beta` of beam search decoder and N-gram LM.
You may follow the following steps to evaluate a neural LM:
#. Obtain `.tsv` file with beams and their corresponding scores. Scores can be from a regular beam search decoder or
in fusion with an N-gram LM scores. For a given beam size `beam_size` and a number of examples
for evaluation `num_eval_examples`, it should contain (`num_eval_examples` x `beam_size`) lines of
form `beam_candidate_text \t score`. This file can be generated by `scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py>`__
#. Rescore the candidates by `scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__.
.. code::
python eval_neural_rescorer.py
--lm_model=[path to .nemo file of the LM]
--beams_file=[path to beams .tsv file]
--beam_size=[size of the beams]
--eval_manifest=[path to eval manifest .json file]
--batch_size=[batch size used for inference on the LM model]
--alpha=[the value for the parameter rescorer_alpha]
--beta=[the value for the parameter rescorer_beta]
--scores_output_file=[the optional path to store the rescored candidates]
The candidates along with their new scores would be stored at the file specified by `--scores_output_file`.
The following is the list of the arguments for the evaluation script:
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| **Argument** |**Type**| **Default** | **Description** |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| lm_model | str | Required | The path of the '.nemo' file of the ASR model to extract the tokenizer. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| eval_manifest | str | Required | Path to the evaluation manifest file (.json manifest file) |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beams_file | str | Required | path to beams file (.tsv) containing the candidates and their scores |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_size | int | Required | The width of the beams (number of candidates) generated by the decoder |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| alpha | float | None | The value for parameter rescorer_alpha |
| | | | Not passing value would enable linear search for rescorer_alpha |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beta | float | None | The value for parameter rescorer_beta |
| | | | Not passing value would enable linear search for rescorer_beta |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| batch_size | int | 16 | The batch size used to calculate the scores |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| scores_output_file | str | None | The optional file to store the rescored beams |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| use_amp | bool | ``False`` | Whether to use AMP if available calculate the scores |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| device | str | cuda | The device to load LM model onto to calculate the scores |
| | | | It can be 'cpu', 'cuda', 'cuda:0', 'cuda:1', ... |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
Hyperparameter Linear Search
----------------------------
This script also supports linear search for parameters `rescorer_alpha` and `rescorer_beta`. If any of the two is not
provided, a linear search is performed to find the best value for that parameter. When linear search is used, initially
`rescorer_beta` is set to zero and the best value for `rescorer_alpha` is found, then `rescorer_alpha` is fixed with
that value and another linear search is done to find the best value for `rescorer_beta`.
If any of the of these two parameters is already specified, then search for that one is skipped. After each search for a
parameter, the plot of WER% for different values of the parameter is also shown.
It is recommended to first use the linear search for both parameters on a validation set by not providing any values for `--alpha` and `--beta`.
Then check the WER curves and decide on the best values for each parameter. Finally, evaluate the best values on the test set.
tts/NeMo/docs/source/asr/configs.rst
+528
View File
@@ -0,0 +1,528 @@
NeMo ASR Configuration Files
============================
This section describes the NeMo configuration file setup that is specific to models in the ASR collection. For general information
about how to set up and run experiments that is common to all NeMo models (e.g. Experiment Manager and PyTorch Lightning trainer
parameters), see the :doc:`../core/core` section.
The model section of the NeMo ASR configuration files generally requires information about the dataset(s) being used, the preprocessor
for audio files, parameters for any augmentation being performed, as well as the model architecture specification. The sections on
this page cover each of these in more detail.
Example configuration files for all of the NeMo ASR scripts can be found in the
`config directory of the examples <https://github.com/NVIDIA/NeMo/tree/v1.0.2/examples/asr/conf>`_.
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections in the configuration file, respectively. Depending on the task, there may be arguments specifying the sample rate
of the audio files, the vocabulary of the dataset (for character prediction), whether or not to shuffle the dataset, and so on. You may
also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command-line at runtime.
Any initialization parameter that is accepted for the Dataset class used in the experiment can be set in the config file.
Refer to the `Datasets <./api.html#Datasets>`__ section of the API for a list of Datasets and their respective parameters.
An example ASR train and validation configuration should look similar to the following:
.. code-block:: yaml
# Specified at the beginning of the config file
labels: &labels [" ", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m",
"n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "'"]
model:
train_ds:
manifest_filepath: ???
sample_rate: 16000
labels: *labels # Uses the labels above
batch_size: 32
trim_silence: True
max_duration: 16.7
shuffle: True
is_tarred: False # If set to true, uses the tarred version of the Dataset
tarred_audio_filepaths: null # Not used if is_tarred is false
tarred_shard_strategy: "scatter" # Not used if is_tarred is false
num_workers: 8
pin_memory: true
validation_ds:
manifest_filepath: ???
sample_rate: 16000
labels: *labels # Uses the labels above
batch_size: 32
shuffle: False # No need to shuffle the validation data
num_workers: 8
pin_memory: true
Preprocessor Configuration
--------------------------
If you are loading audio files for your experiment, you will likely want to use a preprocessor to convert from the
raw audio signal to features (e.g. mel-spectrogram or MFCC). The ``preprocessor`` section of the config specifies the audio
preprocessor to be used via the ``_target_`` field, as well as any initialization parameters for that preprocessor.
An example of specifying a preprocessor is as follows:
.. code-block:: yaml
model:
...
preprocessor:
# _target_ is the audio preprocessor module you want to use
_target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
normalize: "per_feature"
window_size: 0.02
...
# Other parameters for the preprocessor
Refer to the `Audio Preprocessors <./api.html#Audio Preprocessors>`__ API section for the preprocessor options, expected arguments,
and defaults.
Augmentation Configurations
---------------------------
There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
configuration file using a ``spec_augment`` section.
For example, there are options for `Cutout <https://arxiv.org/abs/1708.04552>`_ and
`SpecAugment <https://arxiv.org/abs/1904.08779>`_ available via the ``SpectrogramAugmentation`` module.
The following example sets up both ``Cutout`` (via the ``rect_*`` parameters) and ``SpecAugment`` (via the ``freq_*``
and ``time_*`` parameters).
.. code-block:: yaml
model:
...
spec_augment:
_target_: nemo.collections.asr.modules.SpectrogramAugmentation
# Cutout parameters
rect_masks: 5 # Number of rectangles to cut from any given spectrogram
rect_freq: 50 # Max cut of size 50 along the frequency dimension
rect_time: 120 # Max cut of size 120 along the time dimension
# SpecAugment parameters
freq_masks: 2 # Cut two frequency bands
freq_width: 15 # ... of width 15 at maximum
time_masks: 5 # Cut out 10 time bands
time_width: 25 # ... of width 25 at maximum
You can use any combination of ``Cutout``, frequency/time ``SpecAugment``, or neither of them.
With NeMo ASR, you can also add augmentation pipelines that can be used to simulate various kinds of noise
added to audio in the channel. Augmentors in a pipeline are applied on the audio data read in the data layer. Online
augmentors can be specified in the config file using an ``augmentor`` section in ``train_ds``. The following example
adds an augmentation pipeline that first adds white noise to an audio sample with a probability of 0.5 and at a level
randomly picked between -50 dB and -10 dB and then passes the resultant samples through a room impulse response randomly
picked from the manifest file provided for ``impulse`` augmentation in the config file.
.. code-block:: yaml
model:
...
train_ds:
...
augmentor:
white_noise:
prob: 0.5
min_level: -50
max_level: -10
impulse:
prob: 0.3
manifest_path: /path/to/impulse_manifest.json
Refer to the `Audio Augmentors <./api.html#Audio Augmentors>`__ API section for more details.
Tokenizer Configurations
------------------------
Some models utilize sub-word encoding via an external tokenizer instead of explicitly defining their vocabulary.
For such models, a ``tokenizer`` section is added to the model config. ASR models currently support two types of
custom tokenizers:
- Google Sentencepiece tokenizers (tokenizer type of ``bpe`` in the config)
- HuggingFace WordPiece tokenizers (tokenizer type of ``wpe`` in the config)
In order to build custom tokenizers, refer to the ``ASR_with_Subword_Tokenization`` notebook available in the
ASR tutorials directory.
The following example sets up a ``SentencePiece Tokenizer`` at a path specified by the user:
.. code-block:: yaml
model:
...
tokenizer:
dir: "<path to the directory that contains the custom tokenizer files>"
type: "bpe" # can be "bpe" or "wpe"
For models which utilize sub-word tokenization, we share the decoder module (``ConvASRDecoder``) with character tokenization models.
All parameters are shared, but for models which utilize sub-word encoding, there are minor differences when setting up the config. For
such models, the tokenizer is utilized to fill in the missing information when the model is constructed automatically.
For example, a decoder config corresponding to a sub-word tokenization model should look similar to the following:
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoder
feat_in: *enc_final
num_classes: -1 # filled with vocabulary size from tokenizer at runtime
vocabulary: [] # filled with vocabulary from tokenizer at runtime
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment. Models in the NeMo ASR collection need
an ``encoder`` section and a ``decoder`` section, with the ``_target_`` field specifying the module to use for each.
Here is the list of the parameters in the model section which are shared among most of the ASR models:
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=================================+
| :code:`log_prediction` | bool | Whether a random sample should be printed in the output at each step, along with its predicted transcript. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`ctc_reduction` | string | Specifies the reduction type of CTC loss. Defaults to ``mean_batch`` which would take the average over the | :code:`none`, |
| | | batch after taking the average over the length of each sample. | :code:`mean_batch` |
| | | | :code:`mean`, :code:`sum` |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
The following sections go into more detail about the specific configurations of each model architecture.
For more information about the ASR models, refer to the :doc:`Models <./models>` section.
Jasper and QuartzNet
~~~~~~~~~~~~~~~~~~~~
The `Jasper <./models.html#Jasper>`__ and `QuartzNet <./models.html#QuartzNet>`__ models are very similar, and as such the components in their
configs are very similar as well.
Both architectures use the ``ConvASREncoder`` for the ``encoder``, with parameters detailed in the table below. The encoder parameters
include details about the Jasper/QuartzNet ``[BxR]`` encoder architecture, including how many blocks to use (``B``), how many times
to repeat each sub-block (``R``), and the convolution parameters for each block.
The number of blocks ``B`` is determined by the number of list elements under ``jasper`` minus the one prologue and two epilogue blocks.
The number of sub-blocks ``R`` is determined by setting the ``repeat`` parameter.
To use QuartzNet (which uses more compact time-channel separable convolutions) instead of Jasper, add :code:`separable: true` to all
but the last block in the architecture.
Change the parameter name ``jasper``.
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=====================================+
| :code:`feat_in` | int | The number of input features. Should be equal to :code:`features` in the preprocessor parameters. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| :code:`activation` | string | Which activation function to use in the encoder. | :code:`hardtanh`, :code:`relu`, |
| | | | :code:`selu`, :code:`swish` |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| :code:`conv_mask` | bool | Whether to use masked convolutions in the encoder. Defaults to ``true``. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
| :code:`jasper` | | A list of blocks that specifies your encoder architecture. Each entry in this list represents one block in | |
| | | the architecture and contains the parameters for that block, including convolution parameters, dropout, and | |
| | | the number of times the block is repeated. Refer to the `Jasper <https://arxiv.org/pdf/1904.03288.pdf>`_ and | |
| | | `QuartzNet <https://arxiv.org/pdf/1910.10261.pdf>`_ papers for details about specific model configurations. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+-------------------------------------+
A QuartzNet 15x5 (fifteen blocks, each sub-block repeated five times) encoder configuration should look similar to the following example:
.. code-block:: yaml
# Specified at the beginning of the file for convenience
n_mels: &n_mels 64 # Used for both the preprocessor and encoder as number of input features
repeat: &repeat 5 # R=5
dropout: &dropout 0.0
separable: &separable true # Set to true for QN. Set to false for Jasper.
model:
...
encoder:
_target_: nemo.collections.asr.modules.ConvASREncoder
feat_in: *n_mels # Should match "features" in the preprocessor.
activation: relu
conv_mask: true
jasper: # This field name should be "jasper" for both types of models.
# Prologue block
- dilation: [1]
dropout: *dropout
filters: 256
kernel: [33]
repeat: 1 # Prologue block is not repeated.
residual: false
separable: *separable
stride: [2]
# Block 1
- dilation: [1]
dropout: *dropout
filters: 256
kernel: [33]
repeat: *repeat
residual: true
separable: *separable
stride: [1]
... # Entries for blocks 2~14
# Block 15
- dilation: [1]
dropout: *dropout
filters: 512
kernel: [75]
repeat: *repeat
residual: true
separable: *separable
stride: [1]
# Two epilogue blocks
- dilation: [2]
dropout: *dropout
filters: 512
kernel: [87]
repeat: 1 # Epilogue blocks are not repeated
residual: false
separable: *separable
stride: [1]
- dilation: [1]
dropout: *dropout
filters: &enc_filters 1024
kernel: [1]
repeat: 1 # Epilogue blocks are not repeated
residual: false
stride: [1]
Both Jasper and QuartzNet use the ``ConvASRDecoder`` as the decoder. The decoder parameters are detailed in the following table.
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=================================+
| :code:`feat_in` | int | The number of input features to the decoder. Should be equal to the number of filters in the last block of | |
| | | the encoder. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`vocabulary` | list | A list of the valid output characters for your model. For example, for an English dataset, this could be a | |
| | | list of all lowercase letters, space, and apostrophe. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`num_classes` | int | Number of output classes, i.e. the length of :code:`vocabulary`. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
For example, a decoder config corresponding to the encoder above should look similar to the following:
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoder
feat_in: *enc_filters
vocabulary: *labels
num_classes: 28 # Length of the vocabulary list
Citrinet
~~~~~~~~
The `Citrinet <./models.html#Citrinet>`__ and `QuartzNet <./models.html#QuartzNet>`__ models are very similar, and as such the
components in their configs are very similar as well. Citrinet utilizes Squeeze and Excitation, as well as sub-word tokenization, in
contrast to QuartzNet. Depending on the dataset, we utilize different tokenizers. For Librispeech, we utilize the HuggingFace WordPiece
tokenizer, and for all other datasets we utilize the Google Sentencepiece tokenizer - usually the ``unigram`` tokenizer type.
Both architectures use the ``ConvASREncoder`` for the ``encoder``, with parameters detailed above. The encoder parameters include
details about the Citrinet-C encoder architecture, including how many filters are used per channel (``C``). The Citrinet-C
configuration is a shortform notation for Citrinet-21x5xC, such that ``B = 21`` and ``R = 5`` are the default and should generally
not be changed.
To use Citrinet instead of QuartzNet, refer to the ``citrinet_512.yaml`` configuration found inside the ``examples/asr/conf/citrinet``
directory. Citrinet is primarily comprised of the same :class:`~nemo.collections.asr.parts.submodules.jasper.JasperBlock` as ``Jasper`` or
``QuartzNet`.
While the configs for Citrinet and QuartzNet are similar, we note the additional flags used for Citrinet below. Refer to the
``JasperBlock`` documentation for the meaning of these arguments.
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+===========================+==================+===========================================================================================================+===================================+
| :code:`se` | bool | Whether to apply squeeze-and-excitation mechanism or not. | :code:`true` or :code:`false` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| :code:`se_context_size` | int | SE context size. -1 means global context. | :code:`-1` or :code:`+ve int` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| :code:`stride_last` | bool | Stride on the final repeated block or all repeated blocks. | :code:`true` or :code:`false` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
| :code:`residual_mode` | str | Type of residual branch to construct. | :code:`"add"` or |
| | | Can be pointwise residual addition or pointwise strided residual attention | :code:`"stride_add"` |
+---------------------------+------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------+
A Citrinet-512 config should look similar to the following:
.. code-block:: yaml
model:
...
# Specify some defaults across the entire model
model_defaults:
repeat: 5
dropout: 0.1
separable: true
se: true
se_context_size: -1
...
encoder:
_target_: nemo.collections.asr.modules.ConvASREncoder
feat_in: *n_mels # Should match "features" in the preprocessor.
activation: relu
conv_mask: true
jasper: # This field name should be "jasper" for the JasperBlock (which constructs Citrinet).
# Prologue block
- filters: 512
repeat: 1
kernel: [5]
stride: [1]
dilation: [1]
dropout: 0.0
residual: false
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
# Block 1
- filters: 512
repeat: ${model.model_defaults.repeat}
kernel: [11]
stride: [2]
dilation: [1]
dropout: ${model.model_defaults.dropout}
residual: true
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
stride_last: true
residual_mode: "stride_add"
... # Entries for blocks 2~21
# Block 22
- filters: 512
repeat: ${model.model_defaults.repeat}
kernel: [39]
stride: [1]
dilation: [1]
dropout: ${model.model_defaults.dropout}
residual: true
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
# Epilogue block
- filters: &enc_final 640
repeat: 1
kernel: [41]
stride: [1]
dilation: [1]
dropout: 0.0
residual: false
separable: ${model.model_defaults.separable}
se: ${model.model_defaults.se}
se_context_size: ${model.model_defaults.se_context_size}
As mentioned above, Citrinet uses the ``ConvASRDecoder`` as the decoder layer similar to QuartzNet. Only the configuration must be
changed slightly as Citrinet utilizes sub-word tokenization.
.. note::
The following information is relevant to any of the above models that implements its encoder as an :class:`~nemo.collections.asr.modules.conv_asr.ConvASREncoder`, and utilizes the ``SqueezeExcite`` mechanism.
The ``SqueezeExcite`` block within a :class:`~nemo.collections.asr.modules.conv_asr.ConvASREncoder` network can be modified to utilize a different context window after the model has been instantiated (even after the model has been trained) so as to evaluate the model with limited context. This can be achieved using the :meth:`~nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin.change_conv_asr_se_context_window`
.. code-block:: python
# Here, model can be any model that has a `ConvASREncoder` as its encoder, and utilized `SqueezeExcite` blocks
# `context_window` : It is an integer representing the number of timeframes (each corresponding to some window stride).
# `update_config` : Bool flag which determines whether the config of the model should be updated to reflect the new context window.
# Here, we specify that 128 timeframes of 0.01s stride should be the context window
# This is equivalent to 128 * 0.01s context window for `SqueezeExcite`
model.change_conv_asr_se_context_window(context_window=128, update_config=True)
Conformer-CTC
~~~~~~~~~~~~~
The config files for Conformer-CTC model contain character-based encoding and sub-word encoding at
``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_char.yaml`` and ``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_bpe.yaml``
respectively. Some components of the configs of `Conformer-CTC <./models.html#Conformer-CTC>`__ include the following datasets:
- ``train_ds``, ``validation_ds``, and ``test_ds``
- opimizer (``optim``)
- augmentation (``spec_augment``)
- ``decoder``
- ``trainer``
``exp_manager``
These datasets are similar to other ASR models like `QuartzNet <./models.html#QuartzNet>`__. There should be a tokenizer section where you can
specify the tokenizer if you want to use sub-word encoding instead of character-based encoding.
The encoder section includes the details about the Conformer-CTC encoder architecture. You may find more information in the
config files and also :doc:`nemo.collections.asr.modules.ConformerEncoder<./api.html#nemo.collections.asr.modules.ConformerEncoder>`.
Fine-tuning Configurations
-------------------------
All ASR scripts support easy fine-tuning by partially/fully loading the pretrained weights from a checkpoint into the currently instantiated model. Pre-trained weights can be provided in multiple ways -
1) Providing a path to a NeMo model (via ``init_from_nemo_model``)
2) Providing a name of a pretrained NeMo model (which will be downloaded via the cloud) (via ``init_from_pretrained_model``)
3) Providing a path to a Pytorch Lightning checkpoint file (via ``init_from_ptl_ckpt``)
Fine-tuning via a NeMo model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: sh
python examples/asr/script_to_<script_name>.py \
--config-path=<path to dir of configs> \
--config-name=<name of config without .yaml>) \
model.train_ds.manifest_filepath="<path to manifest file>" \
model.validation_ds.manifest_filepath="<path to manifest file>" \
trainer.gpus=-1 \
trainer.max_epochs=50 \
+init_from_nemo_model="<path to .nemo model file>"
Fine-tuning via a NeMo pretrained model name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: sh
python examples/asr/script_to_<script_name>.py \
--config-path=<path to dir of configs> \
--config-name=<name of config without .yaml>) \
model.train_ds.manifest_filepath="<path to manifest file>" \
model.validation_ds.manifest_filepath="<path to manifest file>" \
trainer.gpus=-1 \
trainer.max_epochs=50 \
+init_from_pretrained_model="<name of pretrained checkpoint>"
Fine-tuning via a Pytorch Lightning checkpoint
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: sh
python examples/asr/script_to_<script_name>.py \
--config-path=<path to dir of configs> \
--config-name=<name of config without .yaml>) \
model.train_ds.manifest_filepath="<path to manifest file>" \
model.validation_ds.manifest_filepath="<path to manifest file>" \
trainer.gpus=-1 \
trainer.max_epochs=50 \
+init_from_ptl_ckpt="<name of pytorch lightning checkpoint>"
tts/NeMo/docs/source/asr/data/asr_results.csv
+21
View File
@@ -0,0 +1,21 @@
Model Name,Model Base Class,Model Card
QuartzNet15x5Base-En,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels"
stt_en_jasper10x5dr,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr"
stt_en_citrinet_256,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256"
stt_en_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512"
stt_en_citrinet_1024,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024"
stt_ca_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5"
stt_it_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5"
stt_fr_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5"
stt_es_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5"
stt_de_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5"
stt_pl_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5"
stt_ru_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5"
stt_zh_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5"
stt_zh_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512"
stt_en_conformer_ctc_small,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small"
stt_en_conformer_ctc_medium,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium"
stt_en_conformer_ctc_large,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large"
stt_en_conformer_ctc_small_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls"
stt_en_conformer_ctc_medium_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls"
stt_en_conformer_ctc_large_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls"
1 Model Name Model Base Class Model Card
2 QuartzNet15x5Base-En EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels
3 stt_en_jasper10x5dr EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr
4 stt_en_citrinet_256 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256
5 stt_en_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512
6 stt_en_citrinet_1024 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024
7 stt_ca_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5
8 stt_it_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5
9 stt_fr_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5
10 stt_es_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5
11 stt_de_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5
12 stt_pl_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5
13 stt_ru_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5
14 stt_zh_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5
15 stt_zh_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512
16 stt_en_conformer_ctc_small EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small
17 stt_en_conformer_ctc_medium EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium
18 stt_en_conformer_ctc_large EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large
19 stt_en_conformer_ctc_small_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls
20 stt_en_conformer_ctc_medium_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls
21 stt_en_conformer_ctc_large_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls
tts/NeMo/docs/source/asr/data/asrlm_results.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Model Base Class,Model Card
asrlm_en_transformer_large_ls,TransformerLMModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls"
1 Model Name Model Base Class Model Card
2 asrlm_en_transformer_large_ls TransformerLMModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls
tts/NeMo/docs/source/asr/data/benchmark_ca.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class,Model Card
stt_ca_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_ca_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_de.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_de_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_de_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_en.csv
+14
View File
@@ -0,0 +1,14 @@
Model Name,Model Base Class,Model Card
QuartzNet15x5Base-En,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels"
stt_zh_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5"
stt_en_jasper10x5dr,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr"
stt_en_citrinet_256,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256"
stt_en_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512"
stt_en_citrinet_1024,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024"
stt_zh_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512"
stt_en_conformer_ctc_small,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small"
stt_en_conformer_ctc_medium,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium"
stt_en_conformer_ctc_large,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large"
stt_en_conformer_ctc_small_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls"
stt_en_conformer_ctc_medium_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls"
stt_en_conformer_ctc_large_ls,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls"
1 Model Name Model Base Class Model Card
2 QuartzNet15x5Base-En EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels
3 stt_zh_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5
4 stt_en_jasper10x5dr EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_jasper10x5dr
5 stt_en_citrinet_256 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_256
6 stt_en_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_512
7 stt_en_citrinet_1024 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_citrinet_1024
8 stt_zh_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512
9 stt_en_conformer_ctc_small EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small
10 stt_en_conformer_ctc_medium EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium
11 stt_en_conformer_ctc_large EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large
12 stt_en_conformer_ctc_small_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls
13 stt_en_conformer_ctc_medium_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls
14 stt_en_conformer_ctc_large_ls EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls
tts/NeMo/docs/source/asr/data/benchmark_es.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_es_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_es_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_fr.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_fr_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_fr_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_it.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_it_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_it_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_pl.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class,Model Card
stt_pl_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_pl_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_pl_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_ru.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_ru_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5"
1 Model Model Base Class Model Card
2 stt_ru_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_quartznet15x5
tts/NeMo/docs/source/asr/data/benchmark_zh.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class,Model Card
stt_zh_quartznet15x5,EncDecCTCModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5"
stt_zh_citrinet_512,EncDecCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512"
1 Model Model Base Class Model Card
2 stt_zh_quartznet15x5 EncDecCTCModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_quartznet15x5
3 stt_zh_citrinet_512 EncDecCTCBPEModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_citrinet_512
tts/NeMo/docs/source/asr/datasets.rst
+268
View File
@@ -0,0 +1,268 @@
Datasets
========
NeMo has scripts to convert several common ASR datasets into the format expected by the ``nemo_asr`` collection. You can get started
with those datasets by following the instructions to run those scripts in the section appropriate to each dataset below.
If the user has their own data and want to preprocess it to use with NeMo ASR models, refer to the `Preparing Custom ASR Data`_ section.
If the user already has a dataset that you want to convert to a tarred format, refer to the `Tarred Datasets`_ section.
.. _LibriSpeech_dataset:
LibriSpeech
-----------
Run the following scripts to download the LibriSpeech data and convert it into the format expected by `nemo_asr`. At least 250GB free
space is required.
.. code-block:: bash
# install sox
sudo apt-get install sox
mkdir data
python get_librispeech_data.py --data_root=data --data_set=ALL
After this, the ``data`` folder should contain wav files and ``.json`` manifests for NeMo ASR datalayer.
Each line is a training example. ``audio_filepath`` contains the path to the wav file, ``duration`` is the duration in seconds, and ``text`` is the transcript:
.. code-block:: json
{"audio_filepath": "<absolute_path_to>/1355-39947-0000.wav", "duration": 11.3, "text": "psychotherapy and the community both the physician and the patient find their place in the community the life interests of which are superior to the interests of the individual"}
{"audio_filepath": "<absolute_path_to>/1355-39947-0001.wav", "duration": 15.905, "text": "it is an unavoidable question how far from the higher point of view of the social mind the psychotherapeutic efforts should be encouraged or suppressed are there any conditions which suggest suspicion of or direct opposition to such curative work"}
Fisher English Training Speech
------------------------------
Run these scripts to convert the Fisher English Training Speech data into a format expected by the ``nemo_asr`` collection.
In brief, the following scripts convert the ``.sph`` files to ``.wav``, slices those files into smaller audio samples, matches the
smaller slices with their corresponding transcripts, and splits the resulting audio segments into train, validation, and test sets
(with one manifest each).
.. note::
- 106 GB of space is required to run the ``.wav`` conversion
- additional 105 GB is required for the slicing and matching
- ``sph2pipe`` is required in order to run the ``.wav`` conversion
**Instructions**
The following scripts assume that you already have the Fisher dataset from the Linguistic Data Consortium, with a directory structure
that looks similar to the following:
.. code-block:: bash
FisherEnglishTrainingSpeech/
├── LDC2004S13-Part1
│   ├── fe_03_p1_transcripts
│   ├── fisher_eng_tr_sp_d1
│   ├── fisher_eng_tr_sp_d2
│   ├── fisher_eng_tr_sp_d3
│   └── ...
└── LDC2005S13-Part2
├── fe_03_p2_transcripts
├── fe_03_p2_sph1
├── fe_03_p2_sph2
├── fe_03_p2_sph3
└── ...
The transcripts that will be used are located in the ``fe_03_p<1,2>_transcripts/data/trans`` directory. The audio files (``.sph``)
are located in the remaining directories in an ``audio`` subdirectory.
#. Convert the audio files from ``.sph`` to ``.wav`` by running:
.. code-block:: bash
cd <nemo_root>/scripts
python fisher_audio_to_wav.py \
--data_root=<fisher_root> --dest_root=<conversion_target_dir>
This will place the unsliced ``.wav`` files in ``<conversion_target_dir>/LDC200[4,5]S13-Part[1,2]/audio-wav/``. It will take several
minutes to run.
#. Process the transcripts and slice the audio data.
.. code-block:: bash
python process_fisher_data.py \
--audio_root=<conversion_target_dir> --transcript_root=<fisher_root> \
--dest_root=<processing_target_dir> \
--remove_noises
This script splits the full dataset into train, validation, test sets, and places the audio slices in the corresponding folders
in the destination directory. One manifest is written out per set, which includes each slice's transcript, duration, and path.
This will likely take around 20 minutes to run. Once finished, delete the 10 minute long ``.wav`` files.
2000 HUB5 English Evaluation Speech
-----------------------------------
Run the following script to convert the HUB5 data into a format expected by the ``nemo_asr`` collection.
Similarly, to the Fisher dataset processing scripts, this script converts the ``.sph`` files to ``.wav``, slices the audio files and
transcripts into utterances, and combines them into segments of some minimum length (default is 10 seconds). The resulting segments
are all written out to an audio directory and the corresponding transcripts are written to a manifest JSON file.
.. note::
- 5 GB of free space is required to run this script
- ``sph2pipe`` is also required to be installed
This script assumes you already have the 2000 HUB5 dataset from the Linguistic Data Consortium.
Run the following command to process the 2000 HUB5 English Evaluation Speech samples:
.. code-block:: bash
python process_hub5_data.py \
--data_root=<path_to_HUB5_data> \
--dest_root=<target_dir>
You can optionally include ``--min_slice_duration=<num_seconds>`` if you would like to change the minimum audio segment duration.
AN4 Dataset
-----------
This is a small dataset recorded and distributed by Carnegie Mellon University. It consists of recordings of people spelling out
addresses, names, etc. Information about this dataset can be found on the `official CMU site <http://www.speech.cs.cmu.edu/databases/an4/>`_.
#. `Download and extract the dataset <http://www.speech.cs.cmu.edu/databases/an4/an4_sphere.tar.gz>`_ (which is labeled "NIST's Sphere audio (.sph) format (64M)".
#. Convert the ``.sph`` files to ``.wav`` using sox, and build one training and one test manifest.
.. code-block:: bash
python process_an4_data.py --data_root=<path_to_extracted_data>
After the script finishes, the ``train_manifest.json`` and ``test_manifest.json`` can be found in the ``<data_root>/an4/`` directory.
Aishell-1
---------
To download the Aishell-1 data and convert it into a format expected by ``nemo_asr``, run:
.. code-block:: bash
# install sox
sudo apt-get install sox
mkdir data
python get_aishell_data.py --data_root=data
After the script finishes, the ``data`` folder should contain a ``data_aishell`` folder which contains a wav file, a transcript folder, and related ``.json`` and ``vocab.txt`` files.
Aishell-2
---------
To process the AIShell-2 dataset, in the command below, set the data folder of AIShell-2 using ``--audio_folder`` and where to push
these files using ``--dest_folder``. In order to generate files in the supported format of ``nemo_asr``, run:
.. code-block:: bash
python process_aishell2_data.py --audio_folder=<data directory> --dest_folder=<destination directory>
After the script finishes, the ``train.json``, ``dev.json``, ``test.json``, and ``vocab.txt`` files can be found in the ``dest_folder`` directory.
Preparing Custom ASR Data
-------------------------
The ``nemo_asr`` collection expects each dataset to consist of a set of utterances in individual audio files plus
a manifest that describes the dataset, with information about one utterance per line (``.json``).
The audio files can be of any format supported by `Pydub <https://github.com/jiaaro/pydub>`_, though we recommend
WAV files as they are the default and have been most thoroughly tested.
There should be one manifest file per dataset that will be passed in, therefore, if the user wants separate training and validation
datasets, they should also have separate manifests. Otherwise, thay will be loading validation data with their training data and vice
versa.
Each line of the manifest should be in the following format:
.. code::
{"audio_filepath": "/path/to/audio.wav", "text": "the transcription of the utterance", "duration": 23.147}
The :code:`audio_filepath` field should provide an absolute path to the ``.wav`` file corresponding to the utterance.
The :code:`text` field should contain the full transcript for the utterance, and the :code:`duration` field should
reflect the duration of the utterance in seconds.
Each entry in the manifest (describing one audio file) should be bordered by '{' and '}' and must
be contained on one line. The fields that describe the file should be separated by commas, and have the form :code:`"field_name": value`,
as shown above. There should be no extra lines in the manifest, i.e. there should be exactly as many lines in the manifest as
there are audio files in the dataset.
Since the manifest specifies the path for each utterance, the audio files do not have to be located
in the same directory as the manifest, or even in any specific directory structure.
Once there is a manifest that describes each audio file in the dataset, use the dataset by passing
in the manifest file path in the experiment config file, e.g. as ``training_ds.manifest_filepath=<path/to/manifest,json>``.
Tarred Datasets
---------------
If experiments are run on a cluster with datasets stored on a distributed file system, the user will likely
want to avoid constantly reading multiple small files and would prefer tarring their audio files.
There are tarred versions of some NeMo ASR dataset classes for this case, such as the ``TarredAudioToCharDataset``
(corresponding to the ``AudioToCharDataset``) and the ``TarredAudioToBPEDataset`` (corresponding to the
``AudioToBPEDataset``). The tarred audio dataset classes in NeMo use `WebDataset <https://github.com/tmbdev/webdataset>`_.
To use an existing tarred dataset instead of a non-tarred dataset, set ``is_tarred: true`` in
the experiment config file. Then, pass in the paths to all of the audio tarballs in ``tarred_audio_filepaths``, either as a list
of filepaths, e.g. ``['/data/shard1.tar', '/data/shard2.tar']``, or in a single brace-expandable string, e.g.
``'/data/shard_{1..64}.tar'`` or ``'/data/shard__OP_1..64_CL_'`` (recommended, see note below).
.. note::
For brace expansion, there may be cases where ``{x..y}`` syntax cannot be used due to shell interference. This occurs most commonly
inside SLURM scripts. Therefore, we provide a few equivalent replacements. Supported opening braces (equivalent to ``{``) are ``(``,
``[``, ``<`` and the special tag ``_OP_``. Supported closing braces (equivalent to ``}``) are ``)``, ``]``, ``>`` and the special
tag ``_CL_``. For SLURM based tasks, we suggest the use of the special tags for ease of use.
As with non-tarred datasets, the manifest file should be passed in ``manifest_filepath``. The dataloader assumes that the length
of the manifest after filtering is the correct size of the dataset for reporting training progress.
The ``tarred_shard_strategy`` field of the config file can be set if you have multiple shards and are running an experiment with
multiple workers. It defaults to ``scatter``, which preallocates a set of shards per worker which do not change during runtime.
For more information about the individual tarred datasets and the parameters available, including shuffling options,
see the corresponding class APIs in the `Datasets <./api.html#Datasets>`__ section.
.. warning::
If using multiple workers, the number of shards should be divisible by the world size to ensure an even
split among workers. If it is not divisible, logging will give a warning but training will proceed, but likely hang at the last epoch.
In addition, if using distributed processing, each shard must have the same number of entries after filtering is
applied such that each worker ends up with the same number of files. We currently do not check for this in any dataloader, but the user's
program may hang if the shards are uneven.
Conversion to Tarred Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can easily convert your existing NeMo-compatible ASR datasets using the
`conversion script here <https://github.com/NVIDIA/NeMo/blob/v1.0.2/scripts/speech_recognition/convert_to_tarred_audio_dataset.py>`_.
.. code::
python convert_to_tarred_audio_dataset.py \
--manifest_path=<path to the manifest file> \
--target_dir=<path to output directory> \
--num_shards=<number of tarfiles that will contain the audio>
--max_duration=<float representing maximum duration of audio samples> \
--min_duration=<float representing minimum duration of audio samples> \
--shuffle --shuffle_seed=0
This script shuffles the entries in the given manifest (if ``--shuffle`` is set, which we recommend), filter
audio files according to ``min_duration`` and ``max_duration``, and tar the remaining audio files to the directory
``--target_dir`` in ``n`` shards, along with separate manifest and metadata files.
The files in the target directory should look similar to the following:
.. code::
target_dir/
├── audio_1.tar
├── audio_2.tar
├── ...
├── metadata.yaml
└── tarred_audio_manifest.json
Note that file structures are flattened such that all audio files are at the top level in each tarball. This ensures that
filenames are unique in the tarred dataset and the filepaths do not contain "-sub" and forward slashes in each ``audio_filepath`` are
simply converted to underscores. For example, a manifest entry for ``/data/directory1/file.wav`` would be ``_data_directory1_file.wav``
in the tarred dataset manifest, and ``/data/directory2/file.wav`` would be converted to ``_data_directory2_file.wav``.
tts/NeMo/docs/source/asr/images/citrinet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 135 KiB

tts/NeMo/docs/source/asr/images/conformer_ctc.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 103 KiB

tts/NeMo/docs/source/asr/images/ctc_asr.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

tts/NeMo/docs/source/asr/images/jasper_layers.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

tts/NeMo/docs/source/asr/images/jasper_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 273 KiB

tts/NeMo/docs/source/asr/images/quartz_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 257 KiB

tts/NeMo/docs/source/asr/intro.rst
+59
View File
@@ -0,0 +1,59 @@
Automatic Speech Recognition (ASR)
==================================
ASR, or Automatic Speech Recognition, refers to the problem of getting a program to automatically transcribe spoken language
(speech-to-text). Our goal is usually to have a model that minimizes the Word Error Rate (WER) metric when transcribing speech input.
In other words, given some audio file (e.g. a WAV file) containing speech, how do we transform this into the corresponding text with
as few errors as possible?
Traditional speech recognition takes a generative approach, modeling the full pipeline of how speech sounds are produced in order to
evaluate a speech sample. We would start from a language model that encapsulates the most likely orderings of words that are generated
(e.g. an n-gram model), to a pronunciation model for each word in that ordering (e.g. a pronunciation table), to an acoustic model that
translates those pronunciations to audio waveforms (e.g. a Gaussian Mixture Model).
Then, if we receive some spoken input, our goal would be to find the most likely sequence of text that would result in the given audio
according to our generative pipeline of models. Overall, with traditional speech recognition, we try to model ``Pr(audio|transcript)*Pr(transcript)``,
and take the argmax of this over possible transcripts.
Over time, neural nets advanced to the point where each component of the traditional speech recognition model could be replaced by a
neural model that had better performance and that had a greater potential for generalization. For example, we could replace an n-gram
model with a neural language model, and replace a pronunciation table with a neural pronunciation model, and so on. However, each of
these neural models need to be trained individually on different tasks, and errors in any model in the pipeline could throw off the
whole prediction.
Thus, we can see the appeal of end-to-end ASR architectures: discriminative models that simply take an audio input and give a textual
output, and in which all components of the architecture are trained together towards the same goal. The model's encoder would be
akin to an acoustic model for extracting speech features, which can then be directly piped to a decoder which outputs text. If desired,
we could integrate a language model that would improve our predictions, as well.
And the entire end-to-end ASR model can be trained at once--a much easier pipeline to handle!
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
datasets
asr_language_modeling
results
configs
api
Resources and Documentation
---------------------------
Hands-on speech recognition tutorial notebooks can be found under `the ASR tutorials folder <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/asr/>`_.
If you are a beginner to NeMo, consider trying out the `ASR with NeMo <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb>`_ tutorial.
This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular ASR model, or would like to find out more about the model
architectures available in the `nemo_asr` collection, refer to the :doc:`Models <./models>` section.
NeMo includes preprocessing scripts for several common ASR datasets. The :doc:`Datasets <./datasets>` section contains instructions on
running those scripts. It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), as well as a list of the checkpoints
available on NGC are located on the :doc:`Checkpoints <./results>` section.
Documentation regarding the configuration files specific to the ``nemo_asr`` models can be found on the :doc:`Configuration Files <./configs>` section.
tts/NeMo/docs/source/asr/models.rst
+97
View File
@@ -0,0 +1,97 @@
Models
======
This section gives a brief overview of the models that NeMo's ASR collection currently supports.
Each of these models can be used with the example ASR scripts (in the ``<NeMo_git_root>/examples/asr`` directory) by
specifying the model architecture in the config file used. Examples of config files for each model can be found in
the ``<NeMo_git_root>/examples/asr/conf`` directory.
For more information about the config files and how they should be structured, refer to the :doc:`./configs` section.
Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found in the :doc:`./results`
section. You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets. The checkpoints section
also contains benchmark results for the available ASR models.
.. _Jasper_model:
Jasper
------
Jasper ("Just Another Speech Recognizer") :cite:`asr-models-li2019jasper` is a deep time delay neural network (TDNN) comprising of
blocks of 1D-convolutional layers. The Jasper family of models are denoted as ``Jasper_[BxR]`` where ``B`` is the number of blocks
and ``R`` is the number of convolutional sub-blocks within a block. Each sub-block contains a 1-D convolution, batch normalization,
ReLU, and dropout:
.. image:: images/jasper_vertical.png
:align: center
:alt: japer model
:scale: 50%
Jasper models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModel` class.
QuartzNet
---------
QuartzNet :cite:`asr-models-kriman2019quartznet` is a version of Jasper :cite:`asr-models-li2019jasper` model with separable
convolutions and larger filters. It can achieve performance similar to Jasper but with an order of magnitude fewer parameters.
Similarly to Jasper, the QuartzNet family of models are denoted as ``QuartzNet_[BxR]`` where ``B`` is the number of blocks and ``R``
is the number of convolutional sub-blocks within a block. Each sub-block contains a 1-D *separable* convolution, batch normalization,
ReLU, and dropout:
.. image:: images/quartz_vertical.png
:align: center
:alt: quartznet model
:scale: 40%
QuartzNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModel` class.
Citrinet
--------
Citrinet is a version of QuartzNet :cite:`asr-models-kriman2019quartznet` that extends ContextNet :cite:`asr-models-han2020contextnet`,
utilizing subword encoding (via Word Piece tokenization) and Squeeze-and-Excitation mechanism :cite:`asr-models-hu2018squeeze` to
obtain highly accurate audio transcripts while utilizing a non-autoregressive CTC based decoding scheme for efficient inference.
.. image:: images/citrinet_vertical.png
:align: center
:alt: citrinet model
:scale: 50%
Citrinet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModelBPE` class.
.. _Conformer-CTC_model:
Conformer-CTC
-------------
Conformer-CTC is a CTC-based variant of the Conformer model introduced in :cite:`asr-models-gulati2020conformer`. Conformer-CTC has a
similar encoder as the original Conformer but uses CTC loss and decoding instead of RNNT loss, which makes it a non-autoregressive model.
We also drop the LSTM decoder and instead use a linear decoder on the top of the encoder. This model uses the combination of
self-attention and convolution modules to achieve the best of the two approaches, the self-attention layers can learn the global
interaction while the convolutions efficiently capture the local correlations. The self-attention modules support both regular
self-attention with absolute positional encoding, and also Transformer-XL's self-attention with relative positional encodings.
Here is the overall architecture of the encoder of Conformer-CTC:
.. image:: images/conformer_ctc.png
:align: center
:alt: Conformer-CTC Model
:scale: 50%
This model supports both the sub-word level and character level encodings. You can find more details on the config files for the
Conformer-CTC models at `Conformer-CTC <./configs.html#conformer-ctc>`. The variant with sub-word encoding is a BPE-based model
which can be instantiated using the :class:`~nemo.collections.asr.models.EncDecCTCModelBPE` class, while the
character-based variant is based on :class:`~nemo.collections.asr.models.EncDecCTCModel`.
You may find the example config files of Conformer-CTC model with character-based encoding at
``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_char.yaml`` and
with sub-word encoding at ``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_bpe.yaml``.
References
----------
.. bibliography:: asr_all.bib
:style: plain
:labelprefix: ASR-MODELS
:keyprefix: asr-models-
tts/NeMo/docs/source/asr/results.rst
+181
View File
@@ -0,0 +1,181 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo:
* Using the :code:`restore_from()` method to load a local checkpoint file (``.nemo``), or
* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
Refer to the following sections for instructions and examples for each.
Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning. For resuming an unfinished
training experiment, use the Experiment Manager to do so by setting the ``resume_if_exists`` flag to ``True``.
Loading Local Checkpoints
-------------------------
NeMo automatically saves checkpoints of a model that is trained in a ``.nemo`` format. Alternatively, to manually save the model at any
point, issue :code:`model.save_to(<checkpoint_path>.nemo)`.
If there is a local ``.nemo`` checkpoint that you'd like to load, use the :code:`restore_from()` method:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
Where the model base class is the ASR model class of the original checkpoint, or the general ``ASRModel`` class.
NGC Pretrained Checkpoints
--------------------------
The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks. These checkpoints are
obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
The model cards on NGC contain more information about each of the checkpoints available.
The tables below list the ASR models available from NGC. The models can be accessed via the :code:`from_pretrained()` method inside
the ASR Model class. In general, you can load any of these models with code in the following format:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.from_pretrained(model_name="<MODEL_NAME>")
Where the model name is the value under "Model Name" entry in the tables below.
For example, to load the base English QuartzNet model for speech recognition, run:
.. code-block:: python
model = nemo_asr.models.ASRModel.from_pretrained(model_name="QuartzNet15x5Base-En")
You can also call :code:`from_pretrained()` from the specific model class (such as :code:`EncDecCTCModel`
for QuartzNet) if you need to access a specific model functionality.
If you would like to programatically list the models available for a particular base class, you can use the
:code:`list_available_models()` method.
.. code-block:: python
nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
Transcribing/Inference
^^^^^^^^^^^^^^^^^^^^^^
To perform inference and transcribe a sample of speech after loading the model, use the ``transcribe()`` method:
.. code-block:: python
model.transcribe(paths2audio_files=[list of audio files], batch_size=BATCH_SIZE, logprobs=False)
Setting the argument ``logprobs`` to ``True`` returns the log probabilities instead of transcriptions. For more information, see :doc:`./api.html#modules`.
The audio files should be 16KHz monochannel wav files.
Fine-tuning on Different Datasets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are multiple ASR tutorials provided in the :ref:`Tutorials <tutorials>` section. Most of these tutorials explain how to instantiate a pre-trained model, prepare the model for fine-tuning on some dataset (in the same language) as a demonstration.
Automatic Speech Recognition Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/asr_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
Language Models for ASR
^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/asrlm_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
Speech Recognition (Languages)
------------------------------
English
^^^^^^^
.. csv-table::
:file: data/benchmark_en.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Mandarin
^^^^^^^^
.. csv-table::
:file: data/benchmark_zh.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
German
^^^^^^
.. csv-table::
:file: data/benchmark_de.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Polish
^^^^^^
.. csv-table::
:file: data/benchmark_pl.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Italian
^^^^^^^
.. csv-table::
:file: data/benchmark_it.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Russian
^^^^^^^
.. csv-table::
:file: data/benchmark_ru.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Spanish
^^^^^^^
.. csv-table::
:file: data/benchmark_es.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
-----------------------------
Catalan
^^^^^^^
.. csv-table::
:file: data/benchmark_ca.csv
:align: left
:widths: 40, 10, 50
:header-rows: 1
tts/NeMo/docs/source/asr/speaker_diarization/api.rst
+17
View File
@@ -0,0 +1,17 @@
NeMo Speaker Diarization API
=============================
Model Classes
-------------
.. autoclass:: nemo.collections.asr.models.ClusteringDiarizer
:show-inheritance:
:members: init_speaker_model, set_vad_model, init_vad_model, setup_vad_test_data, setup_spkr_test_data, run_vad, extract_embeddings, path2audio_files_to_manifest, diarize, make_nemo_file_from_folder, save_to, unpack_nemo_file, restore_from
Mixins
------
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.DiarizationMixin
:show-inheritance:
:members:
tts/NeMo/docs/source/asr/speaker_diarization/configs.rst
+65
View File
@@ -0,0 +1,65 @@
NeMo Speaker Diarization Configuration Files
============================================
Since speaker diarization model here is not a fully-trainble End-to-End model but an inference pipeline, we use **diarizer** instead of **model** which is used in other tasks.
The diarizer section will generally require information about the dataset(s) being
used, models used in this pipline, as well as inference related parameters such as postprocessing of each models.
The sections on this page cover each of these in more detail.
Example configuration files for speaker diarization can be found in the
``<NeMo_git_root>/examples/speaker_recognition/conf/speaker_diarization.yaml>``
.. note::
For model details and deep understanding about configs, finetuning, tuning threshold, and evaluation,
please refer to ``<NeMo_git_root>/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb``;
for other applications such as possible integration with ASR, have a look at ``<NeMo_git_root>/tutorials/speaker_recognition/ASR_with_SpeakerDiarization.ipynb``.
Dataset Configuration
-----------------------
In contrast to other ASR related tasks or models in NeMo, speaker diarization supported in NeMo is a modular inference pipeline.
Datasets here denotes the data you would like to perform speaker diarization on.
An example Speaker Diarization dataset configuration could look like:
.. code-block:: yaml
diarizer:
oracle_num_speakers: ??? # if number of speakers is known input it or pass null if not known. Accepts int or path to file containing uniq-id and num of speakers of that session
max_num_speakers: 8 # max number of speakers for each recording. If oracle num speakers is passed this value is ignored
out_dir: ???
paths2audio_files: ??? # file containing paths to audio files for diarzation
path2groundtruth_rttm_files: null # (Optional) file containing paths to rttm files (this can be passed if we need to calculate DER rate based on ground truth rttm files).
...
.. note::
We expect audio and the corresponding RTTM to have the same base name and the name should be unique.
Diarizer Architecture Configurations
-------------------------------------
.. code-block:: yaml
diarizer:
...
vad:
model_path: null #.nemo local model path or pretrained model name or none
window_length_in_sec: 0.15
shift_length_in_sec: 0.01
threshold: 0.5 # tune threshold on dev set. Check <NeMo_git_root>/scripts/voice_activity_detection/vad_tune_threshold.py
vad_decision_smoothing: True
smoothing_params:
method: "median"
overlap: 0.875
speaker_embeddings:
oracle_vad_manifest: null # leave it null if to perform diarization with above VAD model else path to manifest file genrerated as shown in Datasets section
model_path: ??? #.nemo local model path or pretrained model name
window_length_in_sec: 1.5 # for better performace try with 3 sec and 1.5 sec
shift_length_in_sec: 0.75 # for better performace try with 1.5 sec and 0.75 sec
tts/NeMo/docs/source/asr/speaker_diarization/data/diarization_results.csv
+4
View File
@@ -0,0 +1,4 @@
Model Name,Model Base Class,Model Card
vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
speakerdiarization_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet"
1 Model Name Model Base Class Model Card
2 vad_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet
3 vad_telephony_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet
4 speakerdiarization_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet
tts/NeMo/docs/source/asr/speaker_diarization/datasets.rst
+72
View File
@@ -0,0 +1,72 @@
Datasets
========
Check out page :doc:`Speech Classification Datasets <../speech_classification/datasets>` and :doc:`Speaker Recogniton Datasets <../speaker_recognition/datasets>`
for preparing datasets for training and validating VAD and speaker embedding models respectively.
For Speaker Diarization inference, ``diarizer`` expects following arguments :
- paths2audio_files
- oracle_num_speakers
- path2groundtruth_rttm_files [Optional]
- oracle_vad_manifest (if vad/rttm files are known)[Optional but suggested for good performace. This avoids FPR and TPR]
paths2audio_files:
- a file containing absolute paths to audio files or list of paths to audio files.
For generating a file that contains paths to audio files, you can simply use ``find`` bash command as shown below:
.. code-block:: bash
find $PWD/{data_dir} -iname '*.wav' > path_to_audiofiles.txt
.. note::
We expect audio and the corresponding RTTM files to have the same base name and the name should be unique (uniq-id).
oracle_num_speakers:
- if number of speakers is known input it or pass null if not known. Accepts int or path to file containing uniq-id with num of speakers of that session
Sample line (uniq-id `iaaa` session has 2 unique speakers )::
iaaa 2
path2groundtruth_rttm_files [Optional]:
- To evaluate the diarizer system with known rttm files, One needs to provide a txt file (path2groundtruth_rttm_files) for groundtruth label files.
use above mentioned ``find`` command to get all reference rttm files (use '*.rttm' as search pattern)
Each groundtruth label file should be in NIST Rich Transcription Time Marked (RTTM) format. Take one line from a RTTM file for example:
.. code-block:: bash
SPEAKER TS3012d.Mix-Headset 1 331.573 0.671 <NA> <NA> MTD046ID <NA> <NA>
oracle_vad_manifest [Optional but suggested for good performace. This reduces FPR and TPR]:
- To perform just oracle diarization, that is taking speech activity time stamps from groundtruth RTTMs or another VAD from wild instead of NeMo VAD output, ``diarizer`` expects an oracle manifest json file that contains paths to audio files with offset for start time and duration of segment.
To prepare an oracle manifest file, use the script from ``scripts`` folder as shown below:
.. code-block:: bash
python $NeMo/scripts/ --paths2rttm_files=<path2groundtruth_rttm_files.txt> --paths2audio_files=<paths2audio_files.txt> --manifest_file=<output_oracle_manifest_file.json>
Here ``paths2audio_files`` and ``path2groundtruth_rttm_files`` are files containing paths to audio files as shown above.
AMI Meeting Corpus
------------------
The following are the suggested parameters for getting Speaker Error Rate (SER) of 4.1% on AMI Lapel test set corpus:
- diarizer.oracle_num_speakers = 4 (since there are exactly 4 speakers per each Lapel test set session)
- diarizer.speaker_embeddings.model_path = ``speakerverification_speakernet`` (This model is trained on voxceleb dataset. ``Use this model for simialr non-telephonic speech datasets``)
- diarizer.speaker_embeddings.window_length_in_sec = 3
- diarizer.speaker_embeddings.shift_length_in_sec = 1.5
Input paths2audio_files, paths2rttm_files and oracle_vad_manifest by following steps as shown above
CallHome LDC97S42 (CH109)
-------------------------
The following are the suggested parameters for getting Speaker Error Rate (SER) of 5.4% on CH109 set:
- diarizer.oracle_num_speakers = 2 (since there are exactly 2 speakers per each ch109 session)
- diarizer.speaker_embeddings.model_path = ``speakerdiarization_speakernet`` (This model is trained on voxceleb and telephonic speech Fisher and SWBD. ``Use this model for similar telephonic speech datasets``)
- diarizer.speaker_embeddings.window_length_in_sec = 1.5
- diarizer.speaker_embeddings.shift_length_in_sec = 0.75
Input paths2audio_files, paths2rttm_files and oracle_vad_manifest by following steps as shown above
tts/NeMo/docs/source/asr/speaker_diarization/images/sd_pipeline.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

tts/NeMo/docs/source/asr/speaker_diarization/intro.rst
+52
View File
@@ -0,0 +1,52 @@
Speaker Diarization
==================================
Speaker Diarization (SD) is the task of segmenting audio recordings by speaker labels, that is Who Speaks When?
.. image:: images/sd_pipeline.png
:align: center
:alt: Speaker Diarization pipeline [todo]
:scale: 50%
A diarization system consists of a **Voice Activity Detection (VAD)** model to get the time stamps of audio where speech is being spoken while ignoring the background noise and
a **Speaker Embeddings** model to get speaker embeddings on speech segments obtained from VAD time stamps.
These speaker embeddings would then be clustered into clusters based on number of speakers present in the audio recording.
In NeMo we support both **oracle VAD** and **non-oracle VAD** diarization.
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
datasets
results
configs
api
Resource and Documentation Guide
--------------------------------
Hands-on speaker diarization tutorial notebooks can be found under ``<NeMo_git_root>/tutorials/speaker_recognition/``.
There are tutorials for peformming inference using :ref:`MarbleNet_model` and :ref:`SpeakerNet_model`,
and how one can get ASR transcriptions combined with Speaker labels along with voice activity time stamps with NeMo asr collections.
Most of the tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular model used for speaker diarization inference, or would like to find out more about the model
architectures available in the `nemo_asr` collection, check out the :doc:`Models <./models>` page.
Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
NeMo includes preprocessing scripts for several common ASR datasets, and this page contains instructions on running
those scripts.
It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
:doc:`Configuration Files <./configs>` page.
tts/NeMo/docs/source/asr/speaker_diarization/models.rst
+5
View File
@@ -0,0 +1,5 @@
Models
======
Currenlty NeMo's Speaker Diarization pipeline uses `MarbleNet <../speech_classification/models.html#marblenet-vad>`__ model for Voice Activity Detection (VAD) and `SpeakerNet <../speaker_recognition/models.html#speakernet>`__ model for Speaker Embedding Extraction.
tts/NeMo/docs/source/asr/speaker_diarization/results.rst
+76
View File
@@ -0,0 +1,76 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo as introduced in `loading ASR checkpoints <../results.html#checkpoints>`__.
In speaker diarization, the diarizer loads checkpoints that are passed through the config file. For example:
Loading Local Checkpoints
---------------------------
Load VAD model
.. code-block:: bash
pretrained_vad_model='/path/to/vad_marblenet.nemo' # local .nemo or pretrained vad model name
...
# pass with hydra config
config.diarizer.vad.model_path=pretrained_vad_model
Load Speaker Embedding model
.. code-block:: bash
pretrained_speaker_model='/path/to/speakerdiarization_speakernet.nemo' # local .nemo or pretrained speakernet model name
...
# pass with hydra config
config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
Inference
----------
.. note::
For details and deep understanding, please refer to ``<NeMo_git_root>/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb``.
Check out :doc:`Datasets <./datasets>` for preparing audio files and optional label files.
Run and evaluate speaker diarizer with below command:
.. code-block:: bash
# Have a look at the instruction inside the script and pass the arguments you might need.
python <NeMo_git_root>/examples/speaker_recognition/speaker_diarize.py
NGC Pretrained Checkpoints
----------------------------
The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
The model cards on NGC contain more information about each of the checkpoints available.
In general, you can load models with model name in the following format,
.. code-block:: python
pretrained_vad_model='vad_telephony_marblenet'
pretrained_speaker_model='speakerdiarization_speakernet'
...
config.diarizer.vad.model_path=retrained_vad_model \
config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
where the model name is the value under "Model Name" entry in the tables below.
Models for Speaker Diarization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/diarization_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
tts/NeMo/docs/source/asr/speaker_recognition/configs.rst
+117
View File
@@ -0,0 +1,117 @@
NeMo ASR Configuration Files
============================
This page covers NeMo configuration file setup that is specific to speaker recognition models.
For general information about how to set up and run experiments that is common to all NeMo models (e.g.
experiment manager and PyTorch Lightning trainer parameters), see the :doc:`../../core/core` page.
The model section of NeMo ASR configuration files will generally require information about the dataset(s) being
used, the preprocessor for audio files, parameters for any augmentation being performed, as well as the
model architecture specification.
The sections on this page cover each of these in more detail.
Example configuration files for all of the Speaker related scripts can be found in the
config directory of the examples ``{NEMO_ROOT/examples/speaker_recognition/conf}``.
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections of your configuration file, respectively.
Depending on the task, you may have arguments specifying the sample rate of your audio files, max time length to consider for each audio file , whether or not to shuffle the dataset, and so on.
You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command line
at runtime.
Any initialization parameters that are accepted for the Dataset class used in your experiment
can be set in the config file.
An example SpeakerNet train and validation configuration could look like:
.. code-block:: yaml
model:
train_ds:
manifest_filepath: ???
sample_rate: 16000
labels: None # finds labels based on manifest file
batch_size: 32
trim_silence: False
time_length: 8
shuffle: True
validation_ds:
manifest_filepath: ???
sample_rate: 16000
labels: None # Keep None, to match with labels extracted during training
batch_size: 32
shuffle: False # No need to shuffle the validation data
If you would like to use tarred dataset, have a look at `Datasets Configuration <../configs.html#dataset-configuration>`__.
Preprocessor Configuration
--------------------------
Preprocessor helps to compute MFCC or mel spectrogram features that are given as inputs to model.
For details on how to write this section, refer to `Preprocessor Configuration <../configs.html#preprocessor-configuration>`__
Augmentation Configurations
---------------------------
For SpeakerNet training we use on-the fly augmentations with MUSAN and RIR impulses using ``noise`` augmentor section
The following example sets up musan augmentation with audio files taken from manifest path and
minimum and maximum SNR specified with min_snr and max_snr respectively. This section can be added to
``train_ds`` part in model
.. code-block:: yaml
model:
...
train_ds:
...
augmentor:
noise:
manifest_path: /path/to/musan/manifest_file
prob: 0.2 # probability to augment the incoming batch audio with augmentor data
min_snr_db: 5
max_snr_db: 15
See the :class:`nemo.collections.asr.parts.preprocessing.perturb.AudioAugmentor` API section for more details.
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment.
Models in the NeMo ASR collection need a ``encoder`` section and a ``decoder`` section, with the ``_target_`` field
specifying the module to use for each.
The following sections go into more detail about the specific configurations of each model architecture.
For more information about the SpeakerNet Encoder models, see the :doc:`Models <./models>` page and at `Jasper and QuartzNet <../configs.html#jasper-and-quartznet>`__
Decoder Configurations
------------------------
After features have been computed from speakernet encoder, we pass these features to the decoder to compute embeddings and then to compute log probabilities
for training models.
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.SpeakerDecoder
feat_in: *enc_final
num_classes: 7205 # Total number of classes in training manifest file
pool_mode: xvector # xvector for variance and mean bases statistics pooling
emb_sizes: 256 # number of inermediate emb layers. can be comma separated for additional layers like 512,512
angular: true # if true then loss will be changed to angular softmax loss and consider scale and margin from loss section else train with cross-entrophy loss
loss:
scale: 30
margin 0.2
tts/NeMo/docs/source/asr/speaker_recognition/data/speaker_results.csv
+4
View File
@@ -0,0 +1,4 @@
Model Name,Model Base Class,Model Card
speakerverification_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet"
speakerrecognition_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet"
speakerdiarization_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet"
1 Model Name Model Base Class Model Card
2 speakerverification_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet
3 speakerrecognition_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet
4 speakerdiarization_speakernet EncDecSpeakerLabelModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerdiarization_speakernet
tts/NeMo/docs/source/asr/speaker_recognition/datasets.rst
+57
View File
@@ -0,0 +1,57 @@
Datasets
========
.. _HI-MIA:
HI-MIA
--------
Run the script to download and process ``hi-mia`` dataset in order to generate files in the supported format of ``nemo_asr``. You should set the data folder of
hi-mia using ``--data_root``. These scripts are present in ``<nemo_root>/scripts``
.. code-block:: bash
python get_hi-mia_data.py --data_root=<data directory>
After download and conversion, your `data` folder should contain directories with following set of files as:
* `data/<set>/train.json`
* `data/<set>/dev.json`
* `data/<set>/{set}_all.json`
* `data/<set>/utt2spk`
All-other Datasets
------------------
These methods can be applied to any dataset to get similar training manifest files.
First we prepare scp file(s) containing absolute paths to all the wav files required for each of the train, dev, and test set. This can be easily prepared by using ``find`` bash command as follows:
.. code-block:: bash
!find {data_dir}/{train_dir} -iname "*.wav" > data/train_all.scp
!head -n 3 data/train_all.scp
Since we created the scp file for the train, we use `scp_to_manifest.py` to convert this scp file to a manifest file and then optionally split the files to train \& dev for evaluating the models while training by using the --split flag.
We wont be needing the --split option for the test folder. Accordingly please mention the id number, which is the field num separated by / to be considered as the speaker label.
After the download and conversion, your data folder should contain directories with manifest files as:
* `data/<path>/train.json`
* `data/<path>/dev.json`
* `data/<path>/train_all.json`
Each line in the manifest file describes a training sample - audio_filepath contains the path to the wav file, duration it's duration in seconds, and label is the speaker class label:
.. code-block:: json
{"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": 3.9, "label": "speaker_id"}
Tarred Datasets
---------------
Similarly to ASR, you can tar your audio files and use ASR Dataset class ``TarredAudioToSpeechLabelDataset`` (corresponding to the ``AudioToSpeechLabelDataset``) for this case.
If you want to use tarred dataset, have a look at `ASR Tarred Datasets <../datasets.html#tarred-datasets>`__.
tts/NeMo/docs/source/asr/speaker_recognition/images/ICASPP_SpeakerNet.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 207 KiB

tts/NeMo/docs/source/asr/speaker_recognition/intro.rst
+48
View File
@@ -0,0 +1,48 @@
Speaker Recognition (SR)
========================
Speaker Recognition (SR) is a broad research area which solves two major tasks: speaker identification (who is speaking?) and speaker verification (is the speaker who they claim to be?).
We focus on far-field, text-independent speaker recognition when the identity of the speaker is based on how the speech is spoken, not necessarily in what is being said.
Typically such SR systems operate on unconstrained speech utterances, which are converted into vectors of fixed length, called speaker embeddings.
Speaker embeddings can also be used in automatic speech recognition (ASR) and speech synthesis.
As the goal of most speaker recognition systems is to get good speaker level embeddings that could help distinguish from other speakers, we shall first train these embeddings in end-to-end manner optimizing the QuatzNet based encoder model on cross-entropy loss.
We modify the decoder to get these fixed size embeddings irrespective of the length of the input audio and employ a mean and variance based statistics pooling method to grab these embeddings.
In Speaker Identification we typically train on a larger training set with cross-entrophy loss and finetune later on your preferred set of labels where one would want to classify only known set of speakers.
And in Speaker verification we train with Angular Softmax loss and compare embedings extracted from one audio file coming from a single speaker with
embeddings extracted from another file of same or another speaker by employing backend scoring techniques like cosine similarity.
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
configs
datasets
results
Resource and Documentation Guide
--------------------------------
Hands-on speaker recognition tutorial notebooks can be found under
`the speaker recognition tutorials folder <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/speaker_recognition/>`_. This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular SpeakerNet model, or would like to find out more about the model
architectures available in the ``nemo_asr`` collection, check out the :doc:`Models <./models>` page.
Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
NeMo includes preprocessing and other scripts for speaker_recognition in <nemo/scripts/speaker_recognition/> folder, and this page contains instructions on running
those scripts. It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
:doc:`Configuration Files <./configs>` page.
For a clear step-by-step tutorial we advice you to refer tutorials found in `folder <https://github.com/NVIDIA/NeMo/tree/v1.0.2/tutorials/speaker_recognition/>`_.
tts/NeMo/docs/source/asr/speaker_recognition/models.rst
+52
View File
@@ -0,0 +1,52 @@
Models
======
Examples of config files for the following quartznet model can be found in the ``<NeMo_git_root>/examples/speaker_recogniton/conf`` directory.
For more information about the config files and how they should be structured, see the :doc:`./configs` page.
Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found on the :doc:`./results` page.
You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets.
The Checkpoints page also contains benchmark results for the available speaker recognition models.
.. _SpeakerNet_model:
SpeakerNet
-----------
The model is based on the QuartzNet ASR architecture :cite:`sr-models-koluguri2020speakernet`
comprising of an encoder and decoder structure. We use the encoder of the QuartzNet model as a top-level feature extractor, and feed the output to the statistics pooling layer, where
we compute the mean and variance across channel dimensions to capture the time-independent utterance-level speaker features.
The QuartzNet encoder used for speaker embeddings shown in figure below has the following structure: a QuartzNet BxR
model has B blocks, each with R sub-blocks. Each sub-block applies the following operations: a 1D convolution, batch norm, ReLU, and dropout. All sub-blocks in a block have the same number of output channels. These blocks are connected with residual connections. We use QuartzNet with 3 blocks, 2 sub-blocks, and 512 channels, as the Encoder for Speaker Embeddings. All conv layers have stride 1 and dilation 1.
.. image:: images/ICASPP_SpeakerNet.png
:align: center
:alt: speakernet model
:scale: 40%
Top level acoustic Features, obtained from the output of
encoder are used to compute intermediate features that are
then passed to the decoder for getting utterance level speaker
embeddings. The intermediate time-independent features are
computed using a statistics pooling layer, where we compute the mean and standard deviation of features across
time-channels, to get a time-independent feature representation S of size Batch_size × 3000.
The intermediate features, S are passed through the Decoder consisting of two layers each of output size 512 for a
linear transformation from S to the final number of classes
N for the larger (L) model, and a single linear layer of output size 256 to the final number of classes N for the medium
(M) model. We extract q-vectors after the final linear layer
of fixed size 512, 256 for SpeakerNet-L and SpeakerNet-M
models respectively.
SpeakerNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecSpeakerLabelModel` class.
References
-----------
.. bibliography:: ../asr_all.bib
:style: plain
:labelprefix: SR-MODELS
:keyprefix: sr-models-
tts/NeMo/docs/source/asr/speaker_recognition/results.rst
+85
View File
@@ -0,0 +1,85 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo:
* Using the :code:`restore_from()` method to load a local checkpoint file (`.nemo`), or
* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
See the following sections for instructions and examples for each.
Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning.
For resuming an unfinished training experiment, please use the experiment manager to do so by setting the
``resume_if_exists`` flag to True.
Loading Local Checkpoints
-------------------------
NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
If you have a local ``.nemo`` checkpoint that you'd like to load, simply use the :code:`restore_from()` method:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
Where the model base class is the ASR model class of the original checkpoint, or the general `ASRModel` class.
Inference
----------
The audio files should be 16KHz monochannel wav files.
Write audio files to a ``manifest.json`` file with lines as in format:
.. code-block:: json
{"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": "duration of file in sec", "label": "speaker_id"}
This python call will download best pretrained model from NGC and writes embeddings pickle file to current working directory
.. code-block:: bash
python examples/speaker_recognition/extract_speaker_embeddings.py --manifest=manifest.json
NGC Pretrained Checkpoints
--------------------------
The SpeakerNet-ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
`Speaker_Recognition <https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet>`_ and `Speaker_Verification <https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet>`_ model cards on NGC contain more information about each of the checkpoints available.
The tables below list the SpeakerNet models available from NGC, and the models can be accessed via the
:code:`from_pretrained()` method inside the EncDecSpeakerLabelModel Model class.
.. note:: While loading, remember to use EncDecSpeakerLabelModel for recognition tasks and ExtractSpeakerEmbeddingsModel while extracting embeddings.
In general, you can load any of these models with code in the following format.
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_CLASS_NAME>.from_pretrained(model_name="<MODEL_NAME>")
Where the model name is the value under "Model Name" entry in the tables below.
If you would like to programatically list the models available for a particular base class, you can use the
:code:`list_available_models()` method.
.. code-block:: python
nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
Speaker Recognition Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: data/speaker_results.csv
:align: left
:widths: 30, 30, 40
:header-rows: 1
tts/NeMo/docs/source/asr/speech_classification/configs.rst
+104
View File
@@ -0,0 +1,104 @@
NeMo Speech Classification Configuration Files
================================================
This page covers NeMo configuration file setup that is specific to models in the Speech Classification collection.
For general information about how to set up and run experiments that is common to all NeMo models (e.g.
experiment manager and PyTorch Lightning trainer parameters), see the :doc:`../../core/core` page.
The model section of NeMo Speech Classification configuration files will generally require information about the dataset(s) being
used, the preprocessor for audio files, parameters for any augmentation being performed, as well as the
model architecture specification.
The sections on this page cover each of these in more detail.
Example configuration files for all of the NeMo ASR scripts can be found in the
``<NeMo_git_root>/examples/asr/conf>``.
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections of your configuration file, respectively.
Depending on the task, you may have arguments specifying the sample rate of your audio files, labels, whether or not to shuffle the dataset, and so on.
You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command line
at runtime.
Any initialization parameters that are accepted for the Dataset class used in your experiment
can be set in the config file.
See the `Datasets <../api.html#Datasets>`__ section of the API for a list of Datasets and their respective parameters.
An example Speech Classification train and validation configuration could look like:
.. code-block:: yaml
model:
sample_rate: 16000
repeat: 2 # number of convolutional sub-blocks within a block, R in <MODEL>_[BxRxC]
dropout: 0.0
kernel_size_factor: 1.0
labels: ['bed', 'bird', 'cat', 'dog', 'down', 'eight', 'five', 'four', 'go', 'happy', 'house', 'left', 'marvin',
'nine', 'no', 'off', 'on', 'one', 'right', 'seven', 'sheila', 'six', 'stop', 'three', 'tree', 'two', 'up',
'wow', 'yes', 'zero']
train_ds:
manifest_filepath: ???
sample_rate: ${model.sample_rate}
labels: ${model.labels} # Uses the labels above
batch_size: 128
shuffle: True
validation_ds:
manifest_filepath: ???
sample_rate: ${model.sample_rate}
labels: ${model.labels} # Uses the labels above
batch_size: 128
shuffle: False # No need to shuffle the validation data
If you would like to use tarred dataset, have a look at `Datasets Configuration <../configs.html#dataset-configuration>`__.
Preprocessor Configuration
--------------------------
Preprocessor helps to compute MFCC or mel spectrogram features that are given as inputs to model.
For details on how to write this section, refer to `Preprocessor Configuration <../configs.html#preprocessor-configuration>`__
Check config yaml files in ``<NeMo_git_root>/examples/asr/conf`` to find the processors been used by speech classification models.
Augmentation Configurations
---------------------------
There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
configuration file using the ``augmentor`` and ``spec_augment`` section.
For details on how to write this section, refer to `Augmentation Configuration <../configs.html#augmentation-configurations>`__
Check config yaml files in ``<NeMo_git_root>/tutorials/asr/conf`` to find the processors been used by speech classification models.
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment.
Models in the NeMo ASR collection need a ``encoder`` section and a ``decoder`` section, with the ``_target_`` field
specifying the module to use for each.
The following sections go into more detail about the specific configurations of each model architecture.
The `MatchboxNet <./models.html#matchboxnet-speech-commands>`__ and `MarbleNet <./models.html#marblenet-vad>`__ models are very similar, and they are based on `QuartzNet <../models.html#quartznet>`__ and as such the components in their
configs are very similar as well.
Decoder Configurations
------------------------
After features have been computed from ConvASREncoder, we pass the features to decoder to compute embeddings and then to compute log_probs
for training models.
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoderClassification
feat_in: *enc_final_filters
return_logits: true # return logits if true, else return softmax output
pooling_type: 'avg' # AdaptiveAvgPool1d 'avg' or AdaptiveMaxPool1d 'max'
tts/NeMo/docs/source/asr/speech_classification/data/classification_results.csv
+9
View File
@@ -0,0 +1,9 @@
Model Name,Model Base Class,Model Card
vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
commandrecognition_en_matchboxnet3x1x64_v1,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v1"
commandrecognition_en_matchboxnet3x2x64_v1,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v1"
commandrecognition_en_matchboxnet3x1x64_v2,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2"
commandrecognition_en_matchboxnet3x2x64_v2,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2"
commandrecognition_en_matchboxnet3x1x64_v2_subset_task,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2_subset_task"
commandrecognition_en_matchboxnet3x2x64_v2_subset_task,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2_subset_task"
1 Model Name Model Base Class Model Card
2 vad_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet
3 vad_telephony_marblenet EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet
4 commandrecognition_en_matchboxnet3x1x64_v1 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v1
5 commandrecognition_en_matchboxnet3x2x64_v1 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v1
6 commandrecognition_en_matchboxnet3x1x64_v2 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2
7 commandrecognition_en_matchboxnet3x2x64_v2 EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2
8 commandrecognition_en_matchboxnet3x1x64_v2_subset_task EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x1x64_v2_subset_task
9 commandrecognition_en_matchboxnet3x2x64_v2_subset_task EncDecClassificationModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:commandrecognition_en_matchboxnet3x2x64_v2_subset_task
tts/NeMo/docs/source/asr/speech_classification/datasets.rst
+131
View File
@@ -0,0 +1,131 @@
Datasets
========
NeMo has scripts to convert several common ASR datasets into the format expected by the `nemo_asr` collection.
You can get started with those datasets by following the instructions to run those scripts in the section appropriate to each dataset below.
If you have your own data and want to preprocess it to use with NeMo ASR models, check out the `Preparing Custom Speech Classification Data`_ section at the bottom of the page.
.. _Freesound-dataset:
Freesound
-----------
`Freesound <http://www.freesound.org/>`_ is a website that aims to create a huge open collaborative database of audio snippets, samples, recordings, bleeps.
Most audio samples are released under Creative Commons licenses that allow their reuse.
Researchers and developers can access Freesound content using the Freesound API to retrieve meaningful sound information such as metadata, analysis files, and the sounds themselves.
**Instructions**
Go to ``<NeMo_git_root>/scripts/freesound_download_resample`` and follow the below steps to download and convert freedsound data into a format expected by the `nemo_asr` collection.
1. We will need some required libraries including freesound, requests, requests_oauthlib, joblib, librosa and sox. If they are not installed, please run `pip install -r freesound_requirements.txt`
2. Create an API key for freesound.org at https://freesound.org/help/developers/
3. Create a python file called `freesound_private_apikey.py` and add lined `api_key = <your Freesound api key> and client_id = <your Freesound client id>`
4. Authorize by run `python freesound_download.py --authorize` and visit the website and paste response code
5. Feel free to change any arguments in `download_resample_freesound.sh` such as max_samples and max_filesize
6. Run `bash download_resample_freesound.sh <numbers of files you want> <download data directory> <resampled data directory>` . For example:
.. code-block:: bash
bash download_resample_freesound.sh 4000 ./freesound ./freesound_resampled_background
Note that downloading this dataset may take hours. Change categories in download_resample_freesound.sh to include other (speech) categories audio files.
Then, you should have 16khz mono wav files in `<resampled data directory>`.
.. _Google-Speech-Commands-Dataset:
Google Speech Commands Dataset
------------------------------
Google released two versions of the dataset with the first version containing 65k samples over 30 classes and the second containing 110k samples over 35 classes.
We refer to these datasets as `v1` and `v2` respectively.
Run the script `process_speech_commands_data.py` to process Google Speech Commands dataset in order to generate files in the supported format of `nemo_asr`,
which can be found in ``<NeMo_git_root>/scripts/dataset_processing/``.
You should set the data folder of Speech Commands using :code:`--data_root` and the version of the dataset using :code:`--data_version` as an int.
You can further rebalance the train set by randomly oversampling files inside the manifest by passing the `--rebalance` flag.
.. code-block:: bash
python process_speech_commands_data.py --data_root=<data directory> --data_version=<1 or 2> {--rebalance}
Then, you should have `train_manifest.json`, `validation_manifest.json` and `test_manifest.json`
in the directory `{data_root}/google_speech_recognition_v{1/2}`.
.. note::
You should have at least 4GB or 6GB of disk space available if you use v1 or v2 respectively.
Also, it will take some time to download and process, so go grab a coffee.
Each line is a training example.
.. code-block:: bash
{"audio_filepath": "<absolute path to dataset>/two/8aa35b0c_nohash_0.wav", "duration": 1.0, "label": "two"}
{"audio_filepath": "<absolute path to dataset>/two/ec5ab5d5_nohash_2.wav", "duration": 1.0, "label": "two"}
Speech Command & Freesound for VAD
------------------------------------
Speech Command & Freesound (SCF) dataset is used to train MarbleNet in the `paper <https://arxiv.org/pdf/2010.13886.pdf>`_. Here we show how to download and process it.
This script assumes that you already have the Freesound dataset, if not, have a look at :ref:`Freesound-dataset`.
We will use the open-source :ref:`Google-Speech-Commands-Dataset` (we will use V2 of the dataset for SCF dataset, but require very minor changes to support V1 dataset) as our speech data.
These scripts below will download the Google Speech Commands v2 dataset and convert speech and background data to a format suitable for use with nemo_asr.
.. note::
You may additionally pass :code:`--test_size` or :code:`--val_size` flag for splitting train val and test data.
You may additionally pass :code:`--seg_len` flag for indicating the segment length. Default is 0.63s.
You may additionally pass a :code:`-rebalance_method='fixed|over|under'` at the end of the script to rebalance the class samples in the manifest.
* `'fixed'`: Fixed number of sample for each class. Train 5000, val 1000, and test 1000. (Change number in script if you want)
* `'over'`: Oversampling rebalance method
* `'under'`: Undersampling rebalance method
.. code-block:: bash
mkdir './google_dataset_v2'
python process_vad_data.py --out_dir='./manifest/' --speech_data_root='./google_dataset_v2'--background_data_root=<resampled freesound data directory> --log --rebalance_method='fixed'
After download and conversion, your `manifest` folder should contain a few json manifest files:
* `(balanced_)background_testing_manifest.json`
* `(balanced_)background_training_manifest.json`
* `(balanced_)background_validation_manifest.json`
* `(balanced_)speech_testing_manifest.json`
* `(balanced_)speech_training_manifest.json`
* `(balanced_)speech_validation_manifest.json`
Each line is a training example. `audio_filepath` contains path to the wav file, `duration` is duration in seconds, `offset` is offset in seconds, and `label` is label (class):
.. code-block:: bash
{"audio_filepath": "<absolute path to dataset>/two/8aa35b0c_nohash_0.wav", "duration": 0.63, "label": "speech", "offset": 0.0}
{"audio_filepath": "<absolute path to dataset>/Emergency_vehicle/id_58368 simambulance.wav", "duration": 0.63, "label": "background", "offset": 4.0}
Preparing Custom Speech Classification Data
--------------------------------------------
Preparing Custom Speech Classification Data is almost identical to `Preparing Custom ASR Data <../datasets.html#preparing-custom-asr-data>`__.
Instead of :code:`text` entry in manifest, you need :code:`label` to determine class of this sample
Tarred Datasets
---------------
Similarly to ASR, you can tar your audio files and use ASR Dataset class ``TarredAudioToClassificationLabelDataset`` (corresponding to the ``AudioToClassificationLabelDataset``) for this case.
If you would like to use tarred dataset, have a look at `ASR Tarred Datasets <../datasets.html#tarred-datasets>`__.
tts/NeMo/docs/source/asr/speech_classification/images/marblenet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 384 KiB

tts/NeMo/docs/source/asr/speech_classification/images/matchboxnet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 479 KiB

tts/NeMo/docs/source/asr/speech_classification/intro.rst
+47
View File
@@ -0,0 +1,47 @@
Speech Classification
==================================
Speech Classification refers to a set of tasks or problems of getting a program to automatically classify input utterance or audio segment into categories,
such as Speech Command Recognition (multi-class), Voice Activity Detection (binary or multi-class), and Audio Sentiment Classification (typically multi-class), etc.
**Speech Command Recognition** is the task of classifying an input audio pattern into a discrete set of classes.
It is a subset of Automatic Speech Recognition, sometimes referred to as Key Word Spotting, in which a model is constantly analyzing speech patterns to detect certain "command" classes.
Upon detection of these commands, a specific action can be taken by the system.
It is often the objective of command recognition models to be small and efficient so that they can be deployed onto low-power sensors and remain active for long durations of time.
**Voice Activity Detection (VAD)** also known as speech activity detection or speech detection, is the task of predicting which parts of input audio contain speech versus background noise.
It is an essential first step for a variety of speech-based applications including Automatic Speech Recognition.
It serves to determine which samples to be sent to the model and when to close the microphone.
The full documentation tree is as follows:
.. toctree::
:maxdepth: 8
models
datasets
results
configs
Resource and Documentation Guide
--------------------------------
Hands-on speech classification tutorial notebooks can be found under ``<NeMo_git_repo>/tutorials/asr/``.
There are training and offline & online microphone inference tutorials for Speech Command Detection and Voice Activity Detection tasks.
This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
If you are looking for information about a particular Speech Classification model or would like to find out more about the model
architectures available in the `nemo_asr` collection, check out the :doc:`Models <./models>` page.
Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
NeMo includes preprocessing scripts for several common ASR datasets, and this page contains instructions on running
those scripts.
It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
:doc:`Configuration Files <./configs>` page.
tts/NeMo/docs/source/asr/speech_classification/models.rst
+71
View File
@@ -0,0 +1,71 @@
Models
======
This page gives a brief overview of the models that NeMo's Speech Classification collection currently supports.
For Speech Classification, we support Speech Command (Keyword) Detection and Voice Activity Detection (VAD).
Each of these models can be used with the example ASR scripts (in the ``<NeMo_git_root>/examples/asr`` directory) by
specifying the model architecture in the config file used.
Examples of config files for each model can be found in the ``<NeMo_git_root>/examples/asr/conf`` directory.
For more information about the config files and how they should be structured, see the :doc:`./configs` page.
Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found on the :doc:`./results` page.
You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets.
The Checkpoints page also contains benchmark results for the available ASR models.
.. _MatchboxNet_model:
MatchboxNet (Speech Commands)
------------------------------
MatchboxNet :cite:`sc-models-matchboxnet` is an end-to-end neural network for speech command recognition based on `QuartzNet <../models.html#QuartzNet>`__.
Similarly to QuartzNet, the MatchboxNet family of models are denoted as MatchBoxNet_[BxRxC] where B is the number of blocks, and R is the number of convolutional sub-blocks within a block, and C is the number of channels. Each sub-block contains a 1-D *separable* convolution, batch normalization, ReLU, and dropout:
.. image:: images/matchboxnet_vertical.png
:align: center
:alt: MatchboxNet model
:scale: 50%
It can reach state-of-the art accuracy on the Google Speech Commands dataset while having significantly fewer parameters than similar models.
The `_v1` and `_v2` are denoted for models trained on `v1` (30-way classification) and `v2` (35-way classification) datasets;
And we use _subset_task to represent (10+2)-way subset (10 specific classes + other remaining classes + silence) classification task.
MatchboxNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecClassificationModel` class.
.. note::
For model details and deep understanding about Speech Command Detedction training, inference, finetuning and etc.,
please refer to ``<NeMo_git_root>/tutorials/asr/03_Speech_Commands.ipynb`` and ``<NeMo_git_root>/tutorials/asr/04_Online_Offline_Speech_Commands_Demo.ipynb``.
.. _MarbleNet_model:
MarbleNet (VAD)
------------------
MarbleNet :cite:`sc-models-marblenet` an end-to-end neural network for speech command recognition based on :ref:`MatchboxNet_model`,
Similarly to MatchboxNet, the MarbleNet family of models are denoted as MarbleNet_[BxRxC] where B is the number of blocks, and R is the number of convolutional sub-blocks within a block, and C is the number of channels. Each sub-block contains a 1-D *separable* convolution, batch normalization, ReLU, and dropout:
.. image:: images/marblenet_vertical.png
:align: center
:alt: MarbleNet model
:scale: 30%
It can reach state-of-the art performance on the difficult `AVA speech dataset <https://research.google.com/ava/download.html#ava_speech_download>`_ while having significantly fewer parameters than similar models even training on simple data.
MarbleNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecClassificationModel` class.
.. note::
For model details and deep understanding about VAD training, inference, postprocessing, threshold tuning and etc.,
please refer to ``<NeMo_git_root>/tutorials/asr/06_Voice_Activiy_Detection.ipynb`` and ``<NeMo_git_root>/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb``.
References
----------------
.. bibliography:: ../asr_all.bib
:style: plain
:labelprefix: SC-MODELS
:keyprefix: sc-models-
tts/NeMo/docs/source/asr/speech_classification/results.rst
+105
View File
@@ -0,0 +1,105 @@
Checkpoints
===========
There are two main ways to load pretrained checkpoints in NeMo:
* Using the :code:`restore_from()` method to load a local checkpoint file (`.nemo`), or
* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
See the following sections for instructions and examples for each.
Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning.
For resuming an unfinished training experiment, please use the experiment manager to do so by setting the
``resume_if_exists`` flag to True.
Loading Local Checkpoints
-------------------------
NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
If you have a local ``.nemo`` checkpoint that you'd like to load, simply use the :code:`restore_from()` method:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
Where the model base class is the ASR model class of the original checkpoint, or the general `ASRModel` class.
Transcribing/Inference
-----------------------
The audio files should be 16KHz monochannel wav files.
**Transcribe speech command segment:**
You may perform inference and transcribe a sample of speech after loading the model by using its 'transcribe()' method:
.. code-block:: python
mbn_model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="<MODEL_NAME>")
mbn_model.transcribe([list of audio files], batch_size=BATCH_SIZE, logprobs=False)
Setting argument ``logprobs`` to True would return the log probabilities instead of transcriptions. You may find more details in `Modules <../api.html#modules>`__.
Learn how to fine tune on your own data or on subset classes in ``<NeMo_git_root>/tutorials/asr/03_Speech_Commands.ipynb``
f
**Run VAD inference:**
.. code-block:: bash
python examples/speaker_recognition/vad_infer.py --vad_model="vad_marblenet" --dataset=<FULL PATH OF MANIFEST TO BE PERFORMED INFERENCE ON> --out_dir='frame/demo' --time_length=0.63
Have a look at scripts under ``<NeMo-git-root>/scripts/voice_activity_detection`` for posterior processing and threshold tuning.
NGC Pretrained Checkpoints
--------------------------
The Speech Classification collection has checkpoints of several models trained on various datasets for a variety of tasks.
These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`_.
The model cards on NGC contain more information about each of the checkpoints available.
The tables below list the Speech Classification models available from NGC, and the models can be accessed via the
:code:`from_pretrained()` method inside the ASR Model class.
In general, you can load any of these models with code in the following format.
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="<MODEL_NAME>")
Where the model name is the value under "Model Name" entry in the tables below.
For example, to load the MatchboxNet3x2x64_v1 model for speech command detection, run:
.. code-block:: python
model = nemo_asr.models.EncDecClassificationModel.from_pretrained(model_name="commandrecognition_en_matchboxnet3x2x64_v1")
You can also call :code:`from_pretrained()` from the specific model class (such as :code:`EncDecClassificationModel`
for MatchboxNet and MarbleNet) if you will need to access specific model functionality.
If you would like to programatically list the models available for a particular base class, you can use the
:code:`list_available_models()` method.
.. code-block:: python
nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
Speech Classification Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. tabularcolumns:: 30 30 40
.. csv-table::
:file: data/classification_results.csv
:header-rows: 1
:class: longtable
:widths: 1 1 1