Move model code from Rust monorepo to improve Rust compile times

This will need work done to support w2l.
This commit is contained in:
Brandon Thomas
2022-11-13 17:49:37 -05:00
parent f738b88f9b
commit 5163650b86
1110 changed files with 223803 additions and 0 deletions
@@ -0,0 +1,39 @@
Common Collection
=================
The common collection contains things that could be used across all collections.
Tokenizers
----------
.. automodule:: nemo.collections.common.tokenizers.AutoTokenizer
:special-members: __init__
.. automodule:: nemo.collections.common.tokenizers.SentencePieceTokenizer
:special-members: __init__
.. automodule:: nemo.collections.common.tokenizers.TokenizerSpec
:special-members: __init__
Losses
------
.. automodule:: nemo.collections.common.losses.AggregatorLoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.CrossEntropyLoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.MSELoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.SmoothedCrossEntropyLoss
:special-members: __init__
.. automodule:: nemo.collections.common.losses.SpanningLoss
:special-members: __init__
Metrics
-------
.. autoclass:: nemo.collections.common.metrics.Perplexity
:show-inheritance:
:members:
:undoc-members: