storytold/storyteller-ml
2
0
Fork 0
mirror of https://github.com/storytold/storyteller-ml.git synced 2026-10-17 00:23:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Introduce code from Rust monorepo. This is a mess

Browse Source
This commit is contained in:
Brandon Thomas
2022-02-28 05:43:54 -05:00
parent 921fb49f9a
commit ba326bfcfd
1058 changed files with 192192 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tts/NeMo/docs/source/nlp/api.rst
Executable
+74
View File
@@ -0,0 +1,74 @@
NeMo NLP collection API
=======================
Model Classes
-------------
.. autoclass:: nemo.collections.nlp.models.TextClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact, classifytext
.. autoclass:: nemo.collections.nlp.models.GLUEModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.nlp.models.PunctuationCapitalizationModel
:show-inheritance:
:members: add_punctuation_capitalization, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, multi_validation_epoch_end, register_artifact
.. autoclass:: nemo.collections.nlp.models.TokenClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.nlp.models.QAModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, inference, validation_epoch_end, test_epoch_end
.. autoclass:: nemo.collections.nlp.models.BERTLMModel
:show-inheritance:
:members: setup_training_data, setup_optimization
Modules
-------
.. autoclass:: nemo.collections.nlp.modules.BertModule
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.MegatronBertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.AlbertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.BertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.DistilBertEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.RobertaEncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.SequenceClassifier
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.SequenceRegression
:show-inheritance:
:members:
.. autoclass:: nemo.collections.nlp.modules.SequenceTokenClassifier
:show-inheritance:
:members:
.. autofunction:: nemo.collections.nlp.modules.get_lm_model
.. autofunction:: nemo.collections.nlp.modules.get_pretrained_lm_models_list
.. autofunction:: nemo.collections.nlp.modules.get_megatron_lm_models_list
tts/NeMo/docs/source/nlp/bert_pretraining.rst
Executable
+133
View File
@@ -0,0 +1,133 @@
.. _bert_pretraining:
BERT
====
BERT is an autoencoding language model with a final loss composed of:
- masked language model loss
- next sentence prediction
The model architecture is published in `BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ :cite:`nlp-bert-devlin2018bert`.
The model is originally trained on English Wikipedia and BookCorpus. BERT is often used as a language model encoder for downstream tasks, for example, :ref:`token_classification`, :ref:`text_classification`, :ref:`question_answering`, etc.
Domain-specific BERT models can be advantageous for a wide range of applications. One notable application is the domain-specific BERT in a biomedical setting,
e.g. BioBERT :cite:`nlp-bert-lee2019biobert` or its improved derivative BioMegatron :cite:`nlp-bert-shin2020biomegatron`. For the latter, refer to :ref:`megatron_finetuning`.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import BERTLMModel
# to get the list of pre-trained models
BERTLMModel.list_available_models()
# Download and load the pre-trained BERT-based model
model = BERTLMModel.from_pretrained("bertbaseuncased")
Available Models
^^^^^^^^^^^^^^^^
.. list-table:: *Pretrained Models*
:widths: 5 10
:header-rows: 1
* - Model
- Pretrained Checkpoint
* - BERT-base uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:bertbaseuncased
* - BERT-large uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:bertlargeuncased
.. _dataset_bert_pretraining:
Data Input for the BERT model
-----------------------------
Data preprocessing can be either done on-the-fly during training or offline before training. The latter is optimized and recommended
for large text corpora. This was also used in the original paper to train the model on Wikipedia and BookCorpus. For on-the-fly data
processing, provide text files with sentences for training and validation, where words are separated by spaces, i.e.: ``[WORD] [SPACE] [WORD] [SPACE] [WORD]``.
To use this pipeline in training, use the dedicated configuration file `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml`.
To process data offline in advance, refer to the `BERT Quick Start Guide <https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling/BERT#quick-start-guide>`__.
To recreate the original Wikipedia and BookCorpus datasets, follow steps 1-5 in the Quick Start Guide and run the script ``./data/create_datasets_from_start.sh`` inside the Docker container.
The ``downloaded`` folder should include two sub folders ``lower_case_[0,1]_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5``
and ``lower_case_[0,1]_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5``, containing sequences of length 128 with a maximum of 20 masked tokens
and sequences of length 512 with a maximum of 80 masked tokens respectively. To use this pipeline in training, use the dedicated configuration file ``NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml``
and specify the path to the created hd5f files.
Training the BERT model
-----------------------
Example of model configuration for on-the-fly data preprocessing: `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml>`__.
Example of model configuration for offline data preprocessing: `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml>`__.
The specification can be grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the datasets: **model.train_ds**, **model.validation_ds**
- Parameters that describe the model: **model**, **model.tokenizer**, **model.language_model**
More details about parameters in the config file can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.only_mlm_loss** | bool | Only uses masked language model without next sentence prediction. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.data_file** | string | Name of the text file or hdf5 data directory. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.num_samples** | integer | Number of samples to use from the training dataset, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
More details about parameters for offline data preprocessing can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.max_predictions_per_seq** | integer | Maximum number of masked tokens in a sequence in the preprocessed data. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
More details about parameters for online data preprocessing can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.max_seq_length** | integer | The maximum total input sequence length after tokenization. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.mask_prob** | float | Probability of masking a token in the input text during data processing. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.short_seq_prob** | float | Probability of having a sequence shorter than the maximum sequence length. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
.. note::
For offline data preprocessing, **model.tokenizer** is null. For downstream task, use the same tokenizer that was used for
offline preprocessing. For online data preprocessing, **model.tokenizer** needs to be specified. See also :ref:`nlp_model` for
details.
Example of the command for training the model:
.. code::
python bert_pretraining.py \
model.train_ds.data_file=<PATH_TO_DATA> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.gpus=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>]
Fine-tuning on Downstream Tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To use a trained BERT model checkpoint on a NeMo NLP downstream task, e.g. :ref:`question_answering`, specify
:code:`model.language_model.lm_checkpoint=<PATH_TO_CHECKPOINT>`.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-BERT
:keyprefix: nlp-bert-
tts/NeMo/docs/source/nlp/glue_benchmark.rst
+10
View File
@@ -0,0 +1,10 @@
.. _glue_benchmark:
GLUE Benchmark
==============
We recommend you try the GLUE Benchmark model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_): `NeMo/tutorials/nlp/GLUE_Benchmark.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/GLUE_Benchmark.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train the model can be found here: `NeMo/examples/nlp/glue_benchmark/glue_benchmark.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/glue_benchmark/glue_benchmark.py>`__.
tts/NeMo/docs/source/nlp/information_retrieval.rst
+10
View File
@@ -0,0 +1,10 @@
.. _information_retrieval:
Information Retrieval
=====================
We recommend you try the Information Retrieval model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_): `NeMo/tutorials/nlp/Information_Retrieval_MSMARCO.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Information_Retrieval_MSMARCO.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for hardware the accelerator),
An example script on how to train the model can be found here: `NeMo/examples/nlp/information_retrieval <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp/information_retrieval>`__.
tts/NeMo/docs/source/nlp/intro.rst
Executable
+15
View File
@@ -0,0 +1,15 @@
Natural Language Processing (NLP)
=================================
.. toctree::
:maxdepth: 2
models
megatron_finetuning
api
Example scripts for the NLP collection can be found under `NeMo/example/nlp/ <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp>`__.
NLP notebook-tutorials are located under `NeMo/tutorials/nlp/ <https://github.com/NVIDIA/NeMo/tree/main/tutorials/nlp>`__. Most NeMo
tutorials can be run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.), with more details in the :ref:`tutorials` section.
tts/NeMo/docs/source/nlp/joint_intent_slot.rst
+244
View File
@@ -0,0 +1,244 @@
.. _intent_slot:
Joint Intent and Slot Classification
====================================
Joint Intent and Slot classification is a NLU task for classifying an intent and detecting all
relevant slots (Entities) for the intent in a query. For example, in the query ``What is the weather in Santa Clara tomorrow morning?``,
we would like to classify the query as a ``weather intent``, detect ``Santa Clara`` as a `location slot`,
and ``tomorrow morning`` as a ``date_time slot``. Intent and Slot names are usually task-specific and
defined as labels in the training data. This is a fundamental step that is executed in any
task-driven conversational assistant.
Our BERT-based model implementation allows you to train and detect both of these tasks together.
.. note::
We recommend you try the Joint Intent and Slot Classification model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.): `NeMo/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train the model can be found here: `NeMo/examples/nlp/intent_slot_classification <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp/intent_slot_classification>`__.
NeMo Data Format
----------------
When training the model, the dataset should be first converted to the required data format, which requires the following files:
- :code:`dict.intents.csv` - A list of all intent names in the data. One line per an intent name. The index of the intent line
(starting from ``0``) is used to identify the appropriate intent in ``train.tsv`` and ``test.tsv`` files.
.. code::
weather
alarm
meeting
...
- :code:`dict.slots.csv` - A list of all slot names in the data. One line per slot name. The index of the slot line
(starting from ``0``) is used to identify the appropriate slots in the queries in ``train_slot.tsv`` and ``test_slot.tsv`` files.
In the last line of this dictionary ``O`` slot name is used to identify all ``out of scope`` slots, which are usually the majority of the tokens
in the queries.
.. code::
date
time
city
...
O
- :code:`train.tsv/test.tsv` - A list of original queries, one per line, with the intent number
separated by a tab (e.g. "what alarms do i have set right now <TAB> 0"). Intent numbers are
set according to the intent line in the intent dictionary file (:code:`dict.intents.csv`),
starting from ``0``. The first line in these files should contain the header line ``sentence
<tab> label``.
- :code:`train_slot.tvs/test_slot.tsv` - A list that contains one line per query, when each word from the original text queries
is replaced by a token number from the slots dictionary file (``dict.slots.csv``), counted starting from ``0``. All the words
which do not contain a relevant slot are replaced by ``out-of scope`` token number, which is also a part of the slot dictionary file,
usually as the last entry there. For example a line from these files should look similar to: "54 0 0 54 54 12 12" (the numbers are
separated by a space). These files do not contain a header line.
Dataset Conversion
------------------
To convert to the format of the model data, use the ``import_datasets`` utility, which implements
the conversion for the Assistant dataset. Download the dataset `here <https://github.com/xliuhw/NLU-Evaluation-Data>`_ or you can
write your own converter for the format that you are using for data annotation.
For a dataset that follows your own annotation format, we recommend using one text file for all
samples of the same intent, with the name of the file as the name of the intent. Use one line per
query, with brackets to define slot names. This is very similar to the assistant format, and you can
adapt this converter utility or your own format with small changes:
::
did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]
Run the ``dataset_converter`` command:
.. code::
python examples/nlp/intent_slot_classification/data/import_datasets.py
--source_data_dir=`source_data_dir` \
--target_data_dir=`target_data_dir` \
--dataset_name=['assistant'|'snips'|'atis']
- :code:`source_data_dir`: the directory location of the your dataset
- :code:`target_data_dir`: the directory location where the converted dataset should be saved
- :code:`dataset_name`: one of the implemented dataset names
After conversion, ``target_data_dir`` should contain the following files:
.. code::
.
|--target_data_dir
|-- dict.intents.csv
|-- dict.slots.csv
|-- train.tsv
|-- train_slots.tsv
|-- test.tsv
|-- test_slots.tsv
Model Training
--------------
This is a pretrained BERT based model with 2 linear classifier heads on the top of it, one for classifying an intent of the query and
another for classifying slots for each token of the query. This model is trained with the combined loss function on the Intent and Slot
classification task on the given dataset. The model architecture is based on the paper `BERT for Joint Intent Classification and Slot Filling <https://arxiv.org/pdf/1902.10909.pdf>`__:cite:`nlp-jis-chen2019bert`.
For each query, the model classifies it as one the intents from the intent dictionary and for each word of the query it will classify
it as one of the slots from the slot dictionary, including out of scope slot for all the remaining words in the query which does not
fall in another slot category. Out of scope slot (``O``) is a part of slot dictionary that the model is trained on.
Example of model configuration file for training the model can be found at: `NeMo/examples/nlp/intent_slot_classification/conf/intent_slot_classification.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/intent_slot_classification/conf/intent_slot_classification_config.yaml>`__.
In the configuration file, define the parameters of the training and the model, although most of the default values will work well.
The specification can be roughly grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the model: **model**
- Parameters that describe the datasets: **model.train_ds**, **model.validation_ds**, **model.test_ds**,
More details about parameters in the spec file can be found below:
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.data_dir** | string | -- | The path of the data converted to the specified format. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.class_balancing** | string | ``null`` | Choose from ``[null, weighted_loss]``. The ``weighted_los``s enables weighted class balancing of the loss. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.intent_loss_weight** | float | ``0.6`` | The elation of intent-to-slot loss in the total loss. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.pad_label** | integer | ``-1`` | A value to pad the inputs. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.ignore_extra_tokens** | boolean | ``false`` | A flag that specifies whether to ignore extra tokens. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.ignore_start_end** | boolean | ``true`` | A flag that specifies whether to not use the first and last token for slot training. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.num_output_layers** | integer | ``2`` | The number of fully connected layers of the classifier on top of the BERT model. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.fc_dropout** | float | ``0.1`` | The dropout ratio of the fully connected layers. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.prefix** | string | ``train`` | A prefix for the training file names. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.prefix** | string | ``dev`` | A prefix for the validation file names. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| **test_ds.prefix** | string | ``test`` | A prefix for the test file names. |
+-------------------------------------------+-----------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
For additional config parameters common to all NLP models, refer to the `nlp_model doc <https://github.com/NVIDIA/NeMo/blob/main/docs/source/nlp/nlp_model.rst#model-nlp>`__.
The following is an example of the command for training the model:
.. code::
python examples/nlp/intent_slot_classification/intent_slot_classification.py
model.data_dir=<PATH_TO_DATA_DIR> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.gpus=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>]
Required Arguments for Training
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`model.data_dir`: the dataset directory
Optional Arguments
^^^^^^^^^^^^^^^^^^
Most of the default parameters in the existing configuration file are already set appropriately, however, there are some parameters
you may want to experiment with.
- ``trainer.max_epochs``: the number of training epochs (reasonable to be between 10 to 100)
- ``model.class_balancing`` - value ``weighted_loss`` may help to train the model when there is unbalanced set of classes
- ``model.intent_loss_weight`` - a number between 0 to 1 that defines a weight of the intent lost versus a slot loss during training. A
default value 0.6 gives a slight preference for the intent lose optimization.
Training Procedure
^^^^^^^^^^^^^^^^^^
At the start of evaluation, NeMo will print out a log of the experiment specification, a summary of the training dataset, and the
model architecture.
As the model starts training, you should see a progress bar per epoch. During training, after each epoch, NeMo will display accuracy
metrics on the validation dataset for every intent and slot separately, as well as the total accuracy. You can expect these numbers
to grow up to 50-100 epochs, depending on the size of the trained data. Since this is a joint iIntent and slot training, usually
intent's accuracy will grow first for the initial 10-20 epochs, and after that, slot's accuracy will start improving as well.
At the end of training, NeMo saves the best checkpoint on the validation dataset at the path specified by the experiment spec file
before finishing.
.. code::
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
[NeMo W 2021-01-28 14:52:19 exp_manager:299] There was no checkpoint folder at checkpoint_dir :results/checkpoints. Training from scratch.
[NeMo I 2021-01-28 14:52:19 exp_manager:186] Experiments will be logged at results
...
label precision recall f1 support
weather.weather (label_id: 0) 0.00 0.00 0.00 128
weather.temperature (label_id: 1) 0.00 0.00 0.00 0
weather.temperature_yes_no (label_id: 2) 0.00 0.00 0.00 0
weather.rainfall (label_id: 3) 0.00 0.00 0.00 0
weather.rainfall_yes_no (label_id: 4) 0.00 0.00 0.00 0
weather.snow (label_id: 5) 0.00 0.00 0.00 0
weather.snow_yes_no (label_id: 6) 0.00 0.00 0.00 0
weather.humidity (label_id: 7) 0.00 0.00 0.00 0
weather.humidity_yes_no (label_id: 8) 0.00 0.00 0.00 0
weather.windspeed (label_id: 9) 0.00 0.00 0.00 0
weather.sunny (label_id: 10) 0.00 0.00 0.00 0
weather.cloudy (label_id: 11) 0.00 0.00 0.00 0
weather.alert (label_id: 12) 0.00 0.00 0.00 0
context.weather (label_id: 13) 0.00 0.00 0.00 0
context.continue (label_id: 14) 0.00 0.00 0.00 0
context.navigation (label_id: 15) 0.00 0.00 0.00 0
context.rating (label_id: 16) 0.00 0.00 0.00 0
context.distance (label_id: 17) 0.00 0.00 0.00 0
-------------------
micro avg 0.00 0.00 0.00 128
macro avg 0.00 0.00 0.00 128
weighted avg 0.00 0.00 0.00 128
Model Evaluation and Inference
------------------------------
There is no separate script for the evaluation and inference of this model in NeMo, however, inside of the example file `examples/nlp/intent_slot_classification/intent_slot_classification.py`
after the training part is finished, you can see the code that evaluates the trained model on an evaluation test set and then an example of doing inference using a list of given queries.
For the deployment in the production environment, refer to `NVIDIA Jarvis <https://developer.nvidia.com/nvidia-jarvis-getting-started>`__ and `NVIDIA TLT documentation <https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/nlp/index.html>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-JIS
:keyprefix: nlp-jis-
tts/NeMo/docs/source/nlp/language_modeling.rst
+6
View File
@@ -0,0 +1,6 @@
.. _language_modeling:
Language Modeling
=================
An example script on how to train the model can be found here: `NeMo/examples/nlp/language_modeling <https://github.com/NVIDIA/NeMo/tree/main/examples/nlp/language_modeling>`__.
tts/NeMo/docs/source/nlp/machine_translation.rst
+466
View File
@@ -0,0 +1,466 @@
.. _machine_translation:
Machine Translation Models
==========================
Machine translation is the task of translating text from one language to another. For example, from English to Spanish. Models are
based on the Transformer sequence-to-sequence architecture :cite:`nlp-machine_translation-vaswani2017attention`.
An example script on how to train the model can be found here: `NeMo/examples/nlp/machine_translation/enc_dec_nmt.py <https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/nlp/machine_translation/enc_dec_nmt.py>`__.
The default configuration file for the model can be found at: `NeMo/examples/nlp/machine_translation/conf/aayn_base.yaml <https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/nlp/machine_translation/conf/aayn_base.yaml>`__.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import MTEncDecModel
# To get the list of pre-trained models
MTEncDecModel.list_available_models()
# Download and load the a pre-trained to translate from English to Spanish
model = MTEncDecModel.from_pretrained("nmt_en_es_transformer12x2")
# Translate a sentence or list of sentences
translations = model.translate(["Hello!"], source_lang="en", target_lang="es")
Available Models
^^^^^^^^^^^^^^^^
.. list-table:: *Pretrained Models*
:widths: 5 10
:header-rows: 1
* - Model
- Pretrained Checkpoint
* - English -> German
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_de_transformer12x2
* - German -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_de_en_transformer12x2
* - English -> Spanish
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_es_transformer12x2
* - Spanish -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_es_en_transformer12x2
* - English -> French
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_fr_transformer12x2
* - French -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_fr_en_transformer12x2
* - English -> Russian
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_ru_transformer6x6
* - Russian -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_ru_en_transformer6x6
* - English -> Chinese
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_en_zh_transformer6x6
* - Chinese -> English
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:nmt_zh_en_transformer6x6
Data Format
-----------
Supervised machine translation models require parallel corpora which comprises many examples of sentences in a source language and
their corresponding translation in a target language. We use parallel data formatted as seperate text files for source and target
languages where sentences in corresponding files are aligned like in the table below.
.. list-table:: *Parallel Coprus*
:widths: 10 10
:header-rows: 1
* - train.english.txt
- train.spanish.txt
* - Hello .
- Hola .
* - Thank you .
- Gracias .
* - You can now translate from English to Spanish in NeMo .
- Ahora puedes traducir del inglés al español en NeMo .
It is common practice to apply data cleaning, normalization, and tokenization to the data prior to training a translation model and
NeMo expects already cleaned, normalized, and tokenized data. The only data pre-processing NeMo does is subword tokenization with BPE
:cite:`nlp-machine_translation-sennrich2015neural`.
Data Cleaning, Normalization & Tokenization
-------------------------------------------
We recommend applying the following steps to clean, normalize, and tokenize your data. All pre-trained models released, apply these data pre-processing steps.
#. Language ID filtering - This step filters out examples from your training dataset that aren't in the correct language. For example,
many datasets contain examples where source and target sentences are in the same language. You can use a pre-trained language ID
classifier from `fastText <https://fasttext.cc/docs/en/language-identification.html>`__. You can then run our script using the
``lid.176.bin`` model downloaded from the fastText website.
.. code ::
python NeMo/scripts/neural_machine_translation/filter_by_language.py \
--input-src train.en \
--input-tgt train.es \
--output-src train_lang_filtered.en \
--output-tgt train_lang_filtered.es \
--source-lang en \
--target-lang es \
--removed-src train_noise.en \
--removed-tgt train_noise.de \
--fasttext-model lid.176.bin
#. Length filtering - We filter out sentences from the data that are below a minimum length (1) or exceed a maximum length (250). We
also filter out sentences where the ratio between source and target lengths exceeds 1.3 except for English <-> Chinese models.
`Moses <https://github.com/moses-smt/mosesdecoder>`__ is a statistical machine translation toolkit that contains many useful
pre-processing scripts.
.. code ::
perl mosesdecoder/scripts/training/clean-corpus-n.perl -ratio 1.3 train en es train.filter 1 250
#. Data cleaning - While language ID filtering can sometimes help with filtering out noisy sentences that contain too many puncutations,
it does not help in cases where the translations are potentially incorrect, disfluent, or incomplete. We use `bicleaner <https://github.com/bitextor/bicleaner>`__
a tool to identify such sentences. It trains a classifier based on many features included pre-trained language model fluency, word
alignment scores from a word-alignment model like `Giza++ <https://github.com/moses-smt/giza-pp>`__ etc. We use their available
pre-trained models wherever possible and train models ourselves using their framework for remaining languages. The following script
applies a pre-trained bicleaner model to the data and pick sentences that are clean with probability > 0.5.
.. code ::
awk '{print "-\t-"}' train.en \
| paste -d "\t" - train.filter.en train.filter.es \
| bicleaner-classify - - </path/to/bicleaner.yaml> > train.en-es.bicleaner.score
#. Data deduplication - We use `bifixer <https://github.com/bitextor/bifixer>`__ (which uses xxHash) to hash the source and target
sentences based on which we remove duplicate entries from the file. You may want to do something similar to remove training examples
that are in the test dataset.
.. code ::
cat train.en-es.bicleaner.score \
| parallel -j 25 --pipe -k -l 30000 python bifixer.py --ignore-segmentation -q - - en es \
> train.en-es.bifixer.score
awk -F awk -F "\t" '!seen[$6]++' train.en-es.bifixer.score > train.en-es.bifixer.dedup.score
#. Filter out data that bifixer assigns probability < 0.5 to.
.. code ::
awk -F "\t" '{ if ($5>0.5) {print $3}}' train.en-es.bifixer.dedup.score > train.cleaned.en
awk -F "\t" '{ if ($5>0.5) {print $4}}' train.en-es.bifixer.dedup.score > train.cleaned.es
#. Punctuation Normalization - Punctuation, especially things like quotes can be written in different ways.
It's often useful to normalize the way they appear in text. We use the moses punctuation normalizer on all languages except Chinese.
.. code ::
perl mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l es < train.cleaned.es > train.normalized.es
perl mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l en < train.cleaned.en > train.normalized.en
For example:
.. code ::
Before - Aquí se encuentran joyerías como Tiffany`s entre negocios tradicionales suizos como la confitería Sprüngli.
After - Aquí se encuentran joyerías como Tiffany's entre negocios tradicionales suizos como la confitería Sprüngli.
#. Tokenization and word segmentation for Chinese - Naturally written text often contains punctuation markers like commas, full-stops
and apostrophes that are attached to words. Tokenization by just splitting a string on spaces will result in separate token IDs for
very similar items like ``NeMo`` and ``NeMo.``. Tokenization splits punctuation from the word to create two separate tokens. In the
previous example ``NeMo.`` becomes ``NeMo .`` which when split by space, results in two tokens and adressess the earlier problem.
For example:
.. code ::
Before - Especialmente porque se enfrentará "a Mathieu (Debuchy), Yohan (Cabaye) y Adil (Rami) ", recuerda.
After - Especialmente porque se enfrentará " a Mathieu ( Debuchy ) , Yohan ( Cabaye ) y Adil ( Rami ) " , recuerda .
We use the Moses tokenizer for all languages except Chinese.
.. code ::
perl mosesdecoder/scripts/tokenizer/tokenizer.perl -l es -no-escape < train.normalized.es > train.tokenized.es
perl mosesdecoder/scripts/tokenizer/tokenizer.perl -l en -no-escape < train.normalized.en > train.tokenized.en
For languages like Chinese where there is no explicit marker like spaces that separate words, we use `Jieba <https://github.com/fxsjy/jieba>`__ to segment a string into words that are space separated.
For example:
.. code ::
Before - 同时，卫生局认为有必要接种的其他人员，包括公共部门，卫生局将主动联络有关机构取得名单后由卫生中心安排接种。
After - 同时 ， 卫生局 认为 有 必要 接种 的 其他 人员 ， 包括 公共部门 ， 卫生局 将 主动 联络 有关 机构 取得 名单 后 由 卫生 中心 安排 接种 。
Training a BPE Tokenization
---------------------------
Byte-pair encoding (BPE) :cite:`nlp-machine_translation-sennrich2015neural` is a sub-word tokenization algorithm that is commonly used
to reduce the large vocabulary size of datasets by splitting words into frequently occuring sub-words. Currently, mMachine translation
only supports the `YouTokenToMe <https://github.com/VKCOM/YouTokenToMe>`__ BPE tokenizer. One can set the tokenization configuration
as follows:
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.tokenizer_name** | str | ``yttm`` | BPE library name. Only supports ``yttm`` for now. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.tokenizer_model** | str | ``null`` | Path to an existing YTTM BPE model. If ``null``, will train one from scratch on the provided data. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.vocab_size** | int | ``null`` | Desired vocabulary size after BPE tokenization. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.bpe_dropout** | float | ``null`` | BPE dropout probability. :cite:`nlp-machine_translation-provilkov2019bpe`. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.{encoder_tokenizer,decoder_tokenizer}.vocab_file** | str | ``null`` | Path to pre-computed vocab file if exists. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
| **model.shared_tokenizer** | bool | ``True`` | Whether to share the tokenizer between the encoder and decoder. |
+-----------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------+
Applying BPE Tokenization, Batching, Bucketing and Padding
----------------------------------------------------------
Given BPE tokenizers, and a cleaned parallel corpus, the following steps are applied to create a `TranslationDataset <https://github.com/NVIDIA/NeMo/blob/v1.0.2/nemo/collections/nlp/data/machine_translation/machine_translation_dataset.py#L64>`__ object.
#. Text to IDs - This performs subword tokenization with the BPE model on an input string and maps it to a sequence of tokens for the
source and target text.
#. Bucketing - Sentences vary in length and when creating minibatches, we'd like sentences in them to have roughly the same length to
minimize the number of ``<pad>`` tokens and to maximize computational efficiency. This step groups sentences roughly the same length
into buckets.
#. Batching and padding - Creates minibatches with a maximum number of tokens specified by ``model.{train_ds,validation_ds,test_ds}.tokens_in_batch``
from buckets and pads, so they can be packed into a tensor.
Datasets can be configured as follows:
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.src_file_name** | str | ``null`` | Path to the source language file. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tgt_file_name** | str | ``null`` | Path to the target language file. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tokens_in_batch** | int | ``512`` | Maximum number of tokens per minibatch. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.clean** | bool | ``true`` | Whether to clean the dataset by discarding examples that are greater than ``max_seq_length``. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.max_seq_length** | int | ``512`` | Maximum sequence to be used with the ``clean`` argument above. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.shuffle** | bool | ``true`` | Whether to shuffle minibatches in the PyTorch DataLoader. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.num_samples** | int | ``-1`` | Number of samples to use. ``-1`` for the entire dataset. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.drop_last** | bool | ``false`` | Drop last minibatch if it is not of equal size to the others. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.pin_memory** | bool | ``false`` | Whether to pin memory in the PyTorch DataLoader. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.num_workers** | int | ``8`` | Number of workers for the PyTorch DataLoader. |
+-------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------------+
Tarred Datasets for Large Corpora
---------------------------------
When training with ``DistributedDataParallel``, each process has its own copy of the dataset. For large datasets, this may not always
fit in CPU memory. `Webdatasets <https://github.com/tmbdev/webdataset>`__ circumvents this problem by efficiently iterating over
tar files stored on disk. Each tar file can contain hundreds to thousands of pickle files, each containing a single minibatch.
We recommend using this method when working with datasets with > 1 million sentence pairs.
Tarred datasets can be configured as follows:
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.use_tarred_dataset** | bool | ``false`` | Whether to use tarred datasets. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tar_files** | str | ``null`` | String specifying path to all tar files. Example with 100 tarfiles ``/path/to/tarfiles._OP_1..100_CL_.tar``. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.metadata_file** | str | ``null`` | Path to JSON metadata file that contains only a single entry for the total number of batches in the dataset. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.lines_per_dataset_fragment** | int | ``1000000`` | |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.num_batches_per_tarfile** | int | ``100`` | Maximum sequence to be used with the ``clean`` argument above. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.tar_shuffle_n** | int | ``100`` | Whether to cache IDs to avoid re-tokenizing data. This will be deprecated in favor of tarred datasets. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.{train_ds,validation_ds,test_ds}.shard_strategy** | str | ``scatter`` | Whether to cache IDs in each of the nodes in multi-node training. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
| **model.preproc_out_dir** | str | ``null`` | Path to folder that contains processed tar files or directory where new tar files are written. |
+-----------------------------------------------------------------------+-----------------+----------------+----------------------------------------------------------------------------------------------------------------+
Tarred datasets can be created in two ways:
#. Using the Hydra config and `training script <https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/nlp/machine_translation/enc_dec_nmt.py>`__.
For example:
.. code ::
python examples/nlp/machine_translation/enc_dec_nmt.py \
-cn aayn_base \
do_training=false \
model.preproc_out_dir=/path/to/preproc_dir \
model.train_ds.use_tarred_dataset=true \
model.train_ds.lines_per_dataset_fragment=1000000 \
model.train_ds.num_batches_per_tarfile=200 \
model.train_ds.src_file_name=train.tokenized.en \
model.train_ds.tgt_file_name=train.tokenized.es \
model.validation_ds.src_file_name=validation.tokenized.en \
model.validation_ds.tgt_file_name=validation.tokenized.es \
model.encoder_tokenizer.vocab_size=32000 \
model.decoder_tokenizer.vocab_size=32000 \
~model.test_ds \
trainer.gpus=[0,1,2,3] \
+trainer.fast_dev_run=true \
exp_manager=null \
The above script processes the parallel tokenized text files into tarred datasets that are written to ``/path/to/preproc_dir``. Since
``do_training`` is set to ``False``, the above script only creates tarred datasets and then exits. If ``do_training`` is set ``True``,
then one of two things happen:
(a) If no tar files are present in ``model.preproc_out_dir``, the script first creates those files and then commences training.
(b) If tar files are already present in ``model.preproc_out_dir``, the script starts training from the provided tar files.
#. Using a separate script without Hydra.
Tarred datasets for parallel corpora can also be created with a script that doesn't require specifying a configs via Hydra and
just uses Python argparse.
For example:
.. code ::
python examples/nlp/machine_translation/create_tarred_parallel_dataset.py \
--shared_tokenizer \
--clean \
--bpe_dropout 0.1 \
--src_fname train.tokenized.en \
--tgt_fname train.tokenized.es \
--out_dir /path/to/preproc_dir \
--vocab_size 32000 \
--max_seq_length 512 \
--min_seq_length 1 \
--tokens_in_batch 8192 \
--lines_per_dataset_fragment 1000000 \
--num_batches_per_tarfile 200
You can then set `model.preproc_out_dir=/path/to/preproc_dir` and `model.train_ds.use_tarred_dataset=true` to train with this data.
Model Configuration and Training
--------------------------------
The overall model consists of an encoder, decoder, and classification head. Encoders and decoders have the following configuration
options:
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.max_sequence_length** | int | ``512`` | Maximum sequence length of positional encodings. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.embedding_dropout** | float | ``0.1`` | Path to JSON metadata file that contains only a single entry for the total number of batches in the dataset. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.learn_positional_encodings** | bool | ``false`` | If ``True``, this is a regular learnable embedding layer. If ``False``, fixes position encodings to sinusoidal. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.hidden_size** | int | ``512`` | Size of the transformer hidden states. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.num_layers** | int | ``6`` | Number of transformer layers. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.inner_size** | int | ``2048`` | Size of the hidden states within the feedforward layers. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.num_attention_heads** | int | ``8`` | Number of attention heads. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.ffn_dropout** | float | ``0.1`` | Dropout probability within the feedforward layers. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.attn_score_dropout** | float | ``0.1`` | Dropout probability of the attention scores before softmax normalization. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.attn_layer_dropout** | float | ``0.1`` | Dropout probability of the attention query, key, and value projection activations. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.hidden_act** | str | ``relu`` | Activation function throughout the network. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.mask_future** | bool | ``false``, ``true`` | Whether to mask future timesteps for attention. Defaults to ``True`` for decoder and ``False`` for encoder. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| **model.{encoder,decoder}.pre_ln** | bool | ``false`` | Whether to apply layer-normalization before (``true``) or after (``false``) a sub-layer. |
+-------------------------------------------------------------------+-----------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
Our pre-trained models are optimized with Adam, with a maximum learning of 0.0004, beta of (0.9, 0.98), an inverse square root learning
rate schedule from :cite:`nlp-machine_translation-vaswani2017attention`. The **model.optim** section sets the optimization parameters.
The following script creates tarred datasets based on the provided parallel corpus and train a model based on the ``base`` configuration
from :cite:`nlp-machine_translation-vaswani2017attention`.
.. code ::
python examples/nlp/machine_translation/enc_dec_nmt.py \
-cn aayn_base \
do_training=true \
trainer.gpus=8 \
~trainer.max_epochs \
+trainer.max_steps=100000 \
+trainer.val_check_interval=1000 \
+exp_manager.exp_dir=/path/to/store/results \
+exp_manager.create_checkpoint_callback=True \
+exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU \
+exp_manager.checkpoint_callback_params.mode=max \
+exp_manager.checkpoint_callback_params.save_top_k=5 \
model.preproc_out_dir=/path/to/preproc_dir \
model.train_ds.use_tarred_dataset=true \
model.train_ds.lines_per_dataset_fragment=1000000 \
model.train_ds.num_batches_per_tarfile=200 \
model.train_ds.src_file_name=train.tokenized.en \
model.train_ds.tgt_file_name=train.tokenized.es \
model.validation_ds.src_file_name=validation.tokenized.en \
model.validation_ds.tgt_file_name=validation.tokenized.es \
model.encoder_tokenizer.vocab_size=32000 \
model.decoder_tokenizer.vocab_size=32000 \
~model.test_ds \
The trainer keeps track of the sacreBLEU score :cite:`nlp-machine_translation-post2018call` on the provided validation set and saves
the checkpoints that had the top 5 (by default) sacreBLEU scores.
At the end of training, a ``.nemo`` file is written to the result directory which allows to run inference on a test set.
Multi-Validation
----------------
To run validation on multiple datasets, specify ``validation_ds.src_file_name`` and ``validation_ds.tgt_file_name`` with a list of file paths:
.. code-block:: bash
model.validation_ds.src_file_name=[/data/wmt13-en-de.src,/data/wmt14-en-de.src] \
model.validation_ds.tgt_file_name=[/data/wmt13-en-de.ref,/data/wmt14-en-de.ref] \
When using ``val_loss`` or ``val_sacreBLEU`` for the ``exp_manager.checkpoint_callback_params.monitor``
then the 0th indexed dataset will be used as the monitor.
To use other indexes, append the index:
.. code-block:: bash
exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU_dl_index_1
Multiple test datasets work exactly the same way as validation datasets, simply replace ``validation_ds`` by ``test_ds`` in the above examples.
Model Inference
---------------
To generate translations on a test set and compute sacreBLEU scores, run the inference script:
.. code ::
python examples/nlp/machine_translation/nmt_transformer_infer.py \
--model /path/to/model.nemo \
--srctext test.en \
--tgtout test.en-es.translations \
--batch_size 128 \
--source_lang en \
--target_lang es
The ``--srctext`` file must be provided before tokenization and normalization. The resulting ``--tgtout`` file is detokenized and
can be used to compute sacreBLEU scores.
.. code ::
cat test.en-es.translations | sacrebleu test.es
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: nlp-machine_translation
:keyprefix: nlp-machine_translation-
tts/NeMo/docs/source/nlp/megatron_finetuning.rst
+216
View File
@@ -0,0 +1,216 @@
.. _megatron_finetuning:
Megatron-LM for Downstream Tasks
================================
Megatron-LM :cite:`nlp-megatron-shoeybi2019megatron` is a large, powerful transformer developed by the Applied Deep Learning Research
team at NVIDIA. Unlike BERT, the position of the layer normalization and the residual connection in the model architecture (similar to
GPT-2 architucture) are swapped, which allows the models to continue to improve as they were scaled up. This model reaches higher
scores compared to BERT on a range of Natural Language Processing (NLP) tasks. More details on efficient, model-parallel and multi-node
pre-training of GPT and BERT using mixed precision can be found in the `Megatron-LM GitHub repo <https://github.com/NVIDIA/Megatron-LM>`_.
Fine-tuning
-----------
Pre-trained Megatron-LM (BERT) can be used for most of the NLP downstream tasks from `NeMo/examples/nlp <https://github.com/NVIDIA/NeMo/tree/master/examples/nlp>`_.
Specify the ``model.language_model.pretrained_model_name`` parameter:
.. code-block:: bash
model.language_model.pretrained_model_name=megatron-bert-345m-uncased
Available pre-trained Megatron-LM models:
- ``megatron-bert-345m-cased``
- ``megatron-bert-345m-uncased``
- ``biomegatron-bert-345m-uncased``
- ``biomegatron-bert-345m-cased``
For example, to fine-tune SQuAD v1.1 with Megatron-LM, run:
.. code-block:: bash
python question_answering_squad.py \
model.train_ds.file=<TRAIN_JSON_FILE> \
model.validation_ds=<VAL_JSON_FILE> \
model.language_model.pretrained_model_name=megatron-bert-345m-uncased
If you have a different checkpoint or model configuration (pre-trained with `Megatron-LM GitHub repo <https://github.com/NVIDIA/Megatron-LM>`_),
use ``model.language_model.pretrained_model_name=megatron-bert-uncased`` or ``model.language_model.pretrained_model_name=megatron-bert-cased``
and specify ``--bert_config`` and ``--bert_checkpoint`` for your model.
.. note::
Megatron-LM has its own set of training arguments (including tokenizer) that are ignored during fine-tuning in NeMo. Use downstream
task training scripts for all NeMo supported arguments.
BioMegatron
-----------
BioMegatron has the same network architecture as the Megatron-LM, but is pretrained on a different dataset - `PubMed <https://catalog.data.gov/dataset/pubmed>`_,
a large biomedical text corpus, which achieves better performance in biomedical downstream tasks than the original Megatron-LM.
Examples of using BioMegatron on biomedical downstream tasks can be found at (can be executed with `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_):
`NeMo/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb>`__ and `NeMo/tutorials/nlp/Token_Classification-BioMegatron.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification-BioMegatron.ipynb>`__.
Model Parallelism
-----------------
`Megatron-LM <https://github.com/NVIDIA/Megatron-LM>`_ is a highly optimized and efficient library for training large language models.
With Megatron model parallelism, language models can be trained with billions of weights and then used in NeMo for downstream tasks.
NeMo handles pretrained model parallel checkpoints from Megatron-LM automatically and model parallel models in NeMo have the all
the same features as other NeMo Models.
.. note::
Currently, NeMo only supports tensor model parallelism.
Training
^^^^^^^^
All of the necessary logic to train model parallel models in NeMo with PyTorch Lightning is contained in the ``NLPDDPPlugin``.
The ``NLPDDPPlugin`` subclasses the PyTorch Lightning training type plugin ``DDPPlugin``.
See `plugins <https://pytorch-lightning.readthedocs.io/en/latest/extensions/plugins.html>`_ for more information on PyTorch Lightning Plugins.
To enable model parallel training in NeMo:
.. code-block:: python
trainer = Trainer(plugins=[NLPDDPPlugin()], **cfg.trainer)
Megatron-LM checkpoints have a specific format. One checkpoint is saved for each model parallel rank:
.. code-block:: bash
iter_0080000/
├── mp_rank_00
│ └── model_optim_rng.pt
└── mp_rank_01
└── model_optim_rng.pt
To start fine-tuning from a Megatron-LM checkpoint, simply pass the path to the Megatron-LM checkpoint
via the language model config:
.. code-block:: bash
model.language_model.lm_checkpoint=/raid/megatron/bert/iter_0080000 \
We also need to input the model configuration. This can be done via json:
.. code-block:: json
{
"hidden-size": 1024,
"num-attention-heads": 16,
"num-layers": 24,
"max-seq-length": 512
}
And input via command line:
.. code-block:: bash
model.language_model.config_file=/raid/data/megatron/bert/config.json \
Or the model configuration can be input via YAML:
.. code-block:: YAML
model:
language_model:
config:
hidden_size: 1024
num_attention_heads: 16
num_layers: 24
max_position_embeddings: 512
Additionally, Megatron-LM requires a vocab file:
.. code-block:: bash
model.tokenizer.vocab_file=/path/to/vocab.txt
If using the Megatron-LM default tokenizer for training BERT the vocab file can be omitted:
.. code-block:: bash
# uncased model
model.tokenizer.tokenizer_name=megatron-bert-uncased
.. code-block:: bash
# cased model
model.tokenizer.tokenizer_name=megatron-bert-uncased
Auto-Resume
^^^^^^^^^^^
Resuming training with NeMo experiment manager and PyTorch Lightning works exactly the same as other NeMo models.
While training with PTL, model parallel checkpoint will be saved and loaded properly.
.. code-block:: bash
checkpoints/
├── mp_rank_00
│ ├── mp_autoresume-last.ckpt
│ ├── mp_autoresume---val_loss=0.35-epoch=0.ckpt
│ ├── mp_autoresume---val_loss=0.38-epoch=1.ckpt
│ └── mp_autoresume---val_loss=0.39-epoch=2.ckpt
└── mp_rank_01
├── mp_autoresume-last.ckpt
├── mp_autoresume---val_loss=0.35-epoch=0.ckpt
├── mp_autoresume---val_loss=0.38-epoch=1.ckpt
└── mp_autoresume---val_loss=0.39-epoch=2.ckpt
Save and Restore
^^^^^^^^^^^^^^^^
Model parallel .nemo files behave the same as all other .nemo files. Calling ``.save_to`` will save
a checkpoint for each model parallel rank inside the .nemo file:
.. code-block:: bash
text_class_350m
├── megatron-bert-uncased_encoder_config.json
├── megatron_checkpoint_version.json
├── model_config.yaml
├── mp_rank_00
│ └── model_weights.ckpt
├── mp_rank_01
│ └── model_weights.ckpt
├── tokenizer_vocab_dict.json
└── tokenizer.vocab_file
When restoring a model parallel .nemo file, we must pass in the ``Trainer`` as model parallel requires DDP:
.. code-block:: python
model = TokenClassificationModel.restore_from(cfg.pretrained_model, trainer=trainer)
Evaluation
^^^^^^^^^^
Since model parallel models always require more than one GPU, the ``Trainer`` is needed for evaluation:
.. code-block:: python
trainer = pl.Trainer(plugins=[NLPDDPPlugin()], **cfg.trainer)
model = TextClassificationModel.restore_from(cfg.model.nemo_path, trainer=trainer)
model.setup_test_data(test_data_config=cfg.model.test_ds)
trainer.test(model=model, ckpt_path=None)
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-MEGATRON
:keyprefix: nlp-megatron-
tts/NeMo/docs/source/nlp/models.rst
Executable
+22
View File
@@ -0,0 +1,22 @@
.. _nlp_models:
Models
======
NeMo's NLP collection supports the following models:
.. toctree::
:maxdepth: 1
punctuation_and_capitalization
token_classification
joint_intent_slot
text_classification
bert_pretraining
language_modeling
question_answering
sgd_qa
glue_benchmark
information_retrieval
nlp_model
machine_translation
tts/NeMo/docs/source/nlp/nlp_all.bib
+109
View File
@@ -0,0 +1,109 @@
@article{devlin2018bert,
title={Bert: Pre-training of deep bidirectional transformers for language understanding},
author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal={arXiv preprint arXiv:1810.04805},
year={2018}
}
@article{shoeybi2019megatron,
title={Megatron-lm: Training multi-billion parameter language models using model parallelism},
author={Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanzaro, Bryan},
journal={arXiv preprint arXiv:1909.08053},
year={2019}
}
@InProceedings{maas2011,
author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher},
title = {Learning Word Vectors for Sentiment Analysis},
booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
month = {June},
year = {2011},
address = {Portland, Oregon, USA},
publisher = {Association for Computational Linguistics},
pages = {142--150},
url = {http://www.aclweb.org/anthology/P11-1015}
}
@inproceedings{socher2013,
title = "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank",
author = "Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D. and Ng, Andrew and Potts, Christopher",
booktitle = "Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2013",
address = "Seattle, Washington, USA",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/D13-1170",
pages = "1631--1642",
}
@article{lim2018chemical,
title={Chemical--gene relation extraction using recursive neural network},
author={Lim, Sangrak and Kang, Jaewoo},
journal={Database},
volume={2018},
year={2018},
publisher={Oxford Academic}
}
@inproceedings{li2007scalable,
title={Scalable term selection for text categorization},
author={Li, Jingyang and Sun, Maosong},
booktitle={Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
pages={774--782},
year={2007}
}
@misc{lee2019biobert,
title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
author={Jinhyuk Lee and Wonjin Yoon and Sungdong Kim and Donghyeon Kim and Sunkyu Kim and Chan Ho So and Jaewoo Kang},
year={2019},
eprint={1901.08746},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@misc{shin2020biomegatron,
title={BioMegatron: Larger Biomedical Domain Language Model},
author={Hoo-Chang Shin and Yang Zhang and Evelina Bakhturina and Raul Puri and Mostofa Patwary and Mohammad Shoeybi and Raghav Mani},
year={2020},
eprint={2010.06060},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@article{vaswani2017attention,
title={Attention is all you need},
author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia},
journal={arXiv preprint arXiv:1706.03762},
year={2017}
}
@article{sennrich2015neural,
title={Neural machine translation of rare words with subword units},
author={Sennrich, Rico and Haddow, Barry and Birch, Alexandra},
journal={arXiv preprint arXiv:1508.07909},
year={2015}
}
@article{provilkov2019bpe,
title={Bpe-dropout: Simple and effective subword regularization},
author={Provilkov, Ivan and Emelianenko, Dmitrii and Voita, Elena},
journal={arXiv preprint arXiv:1910.13267},
year={2019}
}
@article{post2018call,
title={A call for clarity in reporting BLEU scores},
author={Post, Matt},
journal={arXiv preprint arXiv:1804.08771},
year={2018}
}
@misc{zhang2021sgdqa,
title={SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services},
author={Yang Zhang and Vahid Noroozi and Evelina Bakhturina and Boris Ginsburg},
year={2021},
eprint={2105.08049},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
tts/NeMo/docs/source/nlp/nlp_model.rst
Executable
+43
View File
@@ -0,0 +1,43 @@
.. _nlp_model:
Model NLP
=========
The config file for NLP models contain three main sections:
- ``trainer``: contains the configs for PTL training. For more information, refer to :doc:`../core/core` and `PTL Trainer class API <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-class-api>'.
- ``exp_manager``: the configs of the experiment manager. For more information, refer to :doc:`../core/core`.
- ``model``: contains the configs of the datasets, model architecture, tokenizer, optimizer, scheduler, etc.
The following sub-sections of the model section are shared among most of the NLP models.
- ``tokenizer``: specifies the tokenizer
- ``language_model``: specifies the underlying model to be used as the encoder
- ``optim``: the configs of the optimizer and scheduler :doc:`../core/core`
The ``tokenizer`` and ``language_model`` sections have the following parameters:
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.tokenizer_name** | string | Tokenizer name will be filled automatically based on ``model.language_model.pretrained_model_name``. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.vocab_file** | string | Path to tokenizer vocabulary. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.tokenizer_model** | string | Path to tokenizer model (only for sentencepiece tokenizer). |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.pretrained_model_name** | string | Pre-trained language model name, for example: ``bert-base-cased`` or ``bert-base-uncased``. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.lm_checkpoint** | string | Path to the pre-trained language model checkpoint. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.config_file** | string | Path to the pre-trained language model config file. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.config** | dictionary | Config of the pre-trained language model. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
The parameter **model.language_model.pretrained_model_name** can be one of the following:
- ``megatron-bert-345m-uncased``, ``megatron-bert-345m-cased``, ``biomegatron-bert-345m-uncased``, ``biomegatron-bert-345m-cased``, ``bert-base-uncased``, ``bert-large-uncased``, ``bert-base-cased``, ``bert-large-cased``
- ``distilbert-base-uncased``, ``distilbert-base-cased``
- ``roberta-base``, ``roberta-large``, ``distilroberta-base``
- ``albert-base-v1``, ``albert-large-v1``, ``albert-xlarge-v1``, ``albert-xxlarge-v1``, ``albert-base-v2``, ``albert-large-v2``, ``albert-xlarge-v2``, ``albert-xxlarge-v2``
tts/NeMo/docs/source/nlp/punctuation_and_capitalization.rst
Executable
+300
View File
@@ -0,0 +1,300 @@
.. _punctuation_and_capitalization:
Punctuation and Capitalization Model
====================================
Automatic Speech Recognition (ASR) systems typically generate text with no punctuation and capitalization of the words.
There are two issues with non-punctuated ASR output:
- it could be difficult to read and understand
- models for some downstream tasks, such as named entity recognition, machine translation, or text-to-speech, are usually trained
on punctuated datasets and using raw ASR output as the input to these models could deteriorate their performance
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import PunctuationCapitalizationModel
# to get the list of pre-trained models
PunctuationCapitalizationModel.list_available_models()
# Download and load the pre-trained BERT-based model
model = PunctuationCapitalizationModel.from_pretrained("punctuation_en_bert")
# try the model on a few examples
model.add_punctuation_capitalization(['how are you', 'great how about you'])
Model Description
-----------------
For each word in the input text, the Punctuation and Capitalization model:
- predicts a punctuation mark that should follow the word (if any). By default, the model supports commas, periods, and question marks.
- predicts if the word should be capitalized or not
In the Punctuation and Capitalization model, we are jointly training two token-level classifiers on top of a pre-trained
language model, such as `BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ :cite:`nlp-punct-devlin2018bert`.
.. note::
We recommend you try this model in a Jupyter notebook (run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.): `NeMo/tutorials/nlp/Punctuation_and_Capitalization.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Punctuation_and_Capitalization.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train the model can be found at: `NeMo/examples/nlp/token_classification/punctuation_capitalization_train.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/punctuation_capitalization_train.py>`__.
An example script on how to run evaluation and inference can be found at: `NeMo/examples/nlp/token_classification/punctuation_capitalization_evaluate.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/punctuation_capitalization_evaluate.py>`__.
The default configuration file for the model can be found at: `NeMo/examples/nlp/token_classification/conf/punctuation_capitalization_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/punctuation_capitalization_config.yaml>`__.
.. _raw_data_format_punct:
Raw Data Format
---------------
The Punctuation and Capitalization model can work with any text dataset, although it is recommended to balance the data, especially
for the punctuation task. Before pre-processing the data to the format expected by the model, the data should be split into ``train.txt``
and ``dev.txt`` (and optionally ``test.txt``). Each line in the ``train.txt/dev.txt/test.txt`` should represent one or more full
and/or truncated sentences.
Example of the ``train.txt``/``dev.txt`` file:
.. code::
When is the next flight to New York?
The next flight is ...
....
The ``source_data_dir`` structure should look similar to the following:
.. code::
.
|--sourced_data_dir
|-- dev.txt
|-- train.txt
NeMo Data Format
----------------
The Punctuation and Capitalization model expects the data in the following format:
The training and evaluation data is divided into 2 files:
- ``text.txt``
- ``labels.txt``
Each line of the ``text.txt`` file contains text sequences, where words are separated with spaces.
[WORD] [SPACE] [WORD] [SPACE] [WORD], for example:
::
when is the next flight to new york
the next flight is ...
...
The ``labels.txt`` file contains corresponding labels for each word in ``text.txt``, the labels are separated with spaces.
Each label in ``labels.txt`` file consists of 2 symbols:
- the first symbol of the label indicates what punctuation mark should follow the word (where ``O`` means no punctuation needed)
- the second symbol determines if a word needs to be capitalized or not (where ``U`` indicates that the word should be upper cased,
and ``O`` - no capitalization needed)
By default, the following punctuation marks are considered: commas, periods, and question marks; the remaining punctuation marks were
removed from the data. This can be changed by introducing new labels in the ``labels.txt`` files.
Each line of the ``labels.txt`` should follow the format: ``[LABEL] [SPACE] [LABEL] [SPACE] [LABEL]`` (for ``labels.txt``). For example,
labels for the above ``text.txt`` file should be:
::
OU OO OO OO OO OO OU ?U
OU OO OO OO ...
...
The complete list of all possible labels used in this tutorial are:
- ``OO``
- ``O``
- ``.O``
- ``?O``
- ``OU``
- <blank space>
- ``U``
- ``.U``
- ``?U``
Converting Raw Data to NeMo Format
----------------------------------
To pre-process the raw text data, stored under :code:`sourced_data_dir` (see the :ref:`raw_data_format_punct`
section), run the following command:
.. code::
python examples/nlp/token_classification/data/prepare_data_for_punctuation_capitalization.py \
-s <PATH_TO_THE_SOURCE_FILE>
-o <PATH_TO_THE_OUTPUT_DIRECTORY>
Required Argument for Dataset Conversion
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`-s` or :code:`--source_file`: path to the raw file
- :code:`-o` or :code:`--output_dir` - path to the directory to store the converted files
After the conversion, the :code:`output_dir` should contain :code:`labels_*.txt` and :code:`text_*.txt` files. The default names
for the training and evaluation in the :code:`conf/punctuation_capitalization_config.yaml` are the following:
.. code::
.
|--output_dir
|-- labels_dev.txt
|-- labels_train.txt
|-- text_dev.txt
|-- text_train.txt
Training Punctuation and Capitalization Model
---------------------------------------------
The language model is initialized with the pre-trained model from `HuggingFace Transformers <https://github.com/huggingface/transformers>`__,
unless the user provides a pre-trained checkpoint for the language model. Example of model configuration file for training the model can be found at: `NeMo/examples/nlp/token_classification/conf/punctuation_capitalization_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/punctuation_capitalization_config.yaml>`__.
The specification is roughly grouped into the following categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the datasets: **model.dataset**, **model.train_ds**, **model.validation_ds**
- Parameters that describe the model: **model**
More details about parameters in the config file can be found below and in the `model's config file <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/punctuation_capitalization_config.yaml>`__:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **pretrained_model** | string | Path to the pre-trained model ``.nemo`` file or pre-trained model name. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.dataset.data_dir** | string | Path to the data converted to the specified above format. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.punct_head.punct_num_fc_layers** | integer | Number of fully connected layers. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.punct_head.fc_dropout** | float | Activation to use between fully connected layers. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.punct_head.activation** | string | Dropout to apply to the input hidden states. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.punct_head.use_transrormer_init** | bool | Whether to initialize the weights of the classifier head with the same approach used in Transformer. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.capit_head.punct_num_fc_layers** | integer | Number of fully connected layers. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.capit_head.fc_dropout** | float | Dropout to apply to the input hidden states. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.capit_head.activation** | string | Activation function to use between fully connected layers. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.capit_head.use_transrormer_init** | bool | Whether to initialize the weights of the classifier head with the same approach used in Transformer. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.text_file** | string | Name of the text training file located at ``data_dir``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.labels_file** | string | Name of the labels training file located at ``data_dir``, such as ``labels_train.txt``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.num_samples** | integer | Number of samples to use from the training dataset, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.text_file** | string | Name of the text file for evaluation, located at ``data_dir``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.labels_file** | string | Name of the labels dev file located at ``data_dir``, such as ``labels_dev.txt``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.num_samples** | integer | Number of samples to use from the dev set, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
For more information, refer to the :ref:`nlp_model` section.
To train the model from scratch, run:
.. code::
python examples/nlp/token_classification/punctuation_and_capitalization_train.py \
model.dataset.data_dir=<PATH/TO/DATA_DIR> \
trainer.gpus=[0,1] \
optim.name=adam \
optim.lr=0.0001
The above command will start model training on GPUs 0 and 1 with Adam optimizer and learning rate of 0.0001; and the trained model is
stored in the ``nemo_experiments/Punctuation_and_Capitalization`` folder.
To train from the pre-trained model, run:
.. code::
python examples/nlp/token_classification/punctuation_and_capitalization_train.py \
model.dataset.data_dir=<PATH/TO/DATA_DIR> \
pretrained_model=<PATH/TO/SAVE/.nemo>
Required Arguments for Training
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* :code:`model.dataset.data_dir`: Path to the `data_dir` with the pre-processed data files.
.. note::
All parameters defined in the configuration file can be changed with command arguments. For example, the sample config file
mentioned above has :code:`validation_ds.batch_size` set to ``64``. However, if you see that the GPU utilization can be
optimized further by using a larger batch size, you may override to the desired value by adding the field :code:`validation_ds.batch_size=128`
over the command-line. You can repeat this with any of the parameters defined in the sample configuration file.
Inference
---------
An example script on how to run inference on a few examples, can be found at `examples/nlp/token_classification/punctuation_capitalization_evaluate.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/punctuation_capitalization_evaluate.py>`_.
To start inference with a pre-trained model on a few examples, run:
.. code::
python punctuation_capitalization_evaluate.py \
pretrained_model=<PRETRAINED_MODEL>
Model Evaluation
----------------
An example script on how to evaluate the pre-trained model, can be found at `examples/nlp/token_classification/punctuation_capitalization_evaluate.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/punctuation_capitalization_evaluate.py>`_.
To start evaluation of the pre-trained model, run:
.. code::
python punctuation_capitalization_evaluate.py \
model.dataset.data_dir=<PATH/TO/DATA/DIR> \
pretrained_model=punctuation_en_bert \
model.test_ds.text_file=<text_dev.txt> \
model.test_ds.labels_file=<labels_dev.txt>
Required Arguments
^^^^^^^^^^^^^^^^^^
- :code:`pretrained_model`: pretrained Punctuation and Capitalization model from ``list_available_models()`` or path to a ``.nemo``
file. For example: ``punctuation_en_bert`` or ``your_model.nemo``.
- :code:`model.dataset.data_dir`: path to the directory that containes :code:`model.test_ds.text_file` and :code:`model.test_ds.labels_file`
During evaluation of the :code:`test_ds`, the script generates two classification reports: one for capitalization task and another
one for punctuation task. This classification reports include the following metrics:
- :code:`Precision`
- :code:`Recall`
- :code:`F1`
More details about these metrics can be found `here <https://en.wikipedia.org/wiki/Precision_and_recall>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-PUNCT
:keyprefix: nlp-punct-
tts/NeMo/docs/source/nlp/question_answering.rst
+300
View File
@@ -0,0 +1,300 @@
.. _question_answering:
Question Answering model
========================
With Question Answering, or Reading Comprehension, given a question and a passage of content (context) that may contain an answer for
the question, the model predicts the span within the text with a start and end position indicating the answer to the question. For
datasets like SQuAD 2.0, this model supports cases when the answer is not contained in the content.
For every word in the context of a given question, the model is trained to predict:
- The likelihood this word is the start of the span.
- The likelihood this word is the end of the span.
The model chooses the start and end words with maximal probabilities. When the content does not contain the answer, we would like the
start and end span to be set for the first token.
A pretrained BERT encoder with two span prediction heads is used for the prediction start and the end position of the answer. The span
predictions are token classifiers consisting of a single linear layer.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import QAModel
# to get the list of pre-trained models
QAModel.list_available_models()
# Download and load the pre-trained BERT-based model
model = QAModel.from_pretrained("qa_squadv1.1_bertbase")
# try the model on a few examples
model.inference(test_file)
.. note::
We recommend you try Question Answering model in a Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.):
`NeMo/tutorials/nlp/Question_Answering_Squad.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Question_Answering_Squad.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train and evaluate the model can be found here: `NeMo/examples/nlp/question_answering/question_answering_squad.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/question_answering_squad.py>`__.
The default configuration file for the model can be found at: `NeMo/examples/nlp/question_answering/conf/question_answering_squad.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/conf/question_answering_squad_config.yaml>`__.
Available models
^^^^^^^^^^^^^^^^
.. list-table:: *Pretrained Models*
:widths: 5 10
:header-rows: 1
* - Model
- Pretrained Checkpoint
* - qa_squadv1.1_bertbase
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv1_1_bertbase
* - qa_squadv2.0_bertbase
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv2_0_bertbase
* - qa_squadv1.1_bertlarge
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv1_1_bertlarge
* - qa_squadv2.0_bertlarge
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv2_0_bertlarge
* - qa_squadv1.1_megatron_cased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv1_1_megatron_cased
* - qa_squadv2.0_megatron_cased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv2_0_megatron_cased
* - qa_squadv1.1_megatron_uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv1_1_megatron_uncased
* - qa_squadv2.0_megatron_uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:qa_squadv2_0_megatron_uncased
.. _dataset_question_answering:
Data Format
-----------
This model expects the dataset in `SQuAD format`_ (i.e., a JSON file for each dataset split). The code snippet below shows an example
of the training file. Each title has one or multiple paragraph entries, each consisting of the "context" and question-answer entries.
Each question-answer entry has:
- A question
- A globally unique id
- The Boolean flag ``is_impossible``, which shows whether a question is answerable or not:
- if the question is answerable, one answer entry contains the text span and its starting character index in the context
- if the question is not answerable, an empty ``answers`` list is provided
.. _SQuAD format: https://rajpurkar.github.io/SQuAD-explorer/
The evaluation files (for validation and testing) follow the above format, except that it can provide more than one answer to the
same question. The inference file also follows the above format, except that it does not require the ``answers`` and ``is_impossible``
keywords.
The following is an example of the data format (JSON file):
.. code::
{
"data": [
{
"title": "Super_Bowl_50",
"paragraphs": [
{
"context": "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24\u201310 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
"qas": [
{
"question": "Where did Super Bowl 50 take place?",
"is_impossible": "false",
"id": "56be4db0acb8001400a502ee",
"answers": [
{
"answer_start": "403",
"text": "Santa Clara, California"
}
]
},
{
"question": "What was the winning score of the Super Bowl 50?",
"is_impossible": "true",
"id": "56be4db0acb8001400a502ez",
"answers": [
]
}
]
}
]
}
]
}
Dataset Download
----------------
To perform training of the Question Answering model on the SQuAD dataset, you must first download it from `here
<https://rajpurkar.github.io/SQuAD-explorer/>`_ or run:
.. code::
python get_squad.py
There are two versions:
- SQuAD version 1.1, which does not contain questions without the answer and has 100,000+ question-answer pairs on 500+ articles.
- SQuAD version 2.0, which combines the 100,000 questions from SQuAD 1.1 with over 50,000 unanswerable questions. To do well with
SQuAD 2.0, a system must not only answer questions when possible, but also determine when no answer is supported by the paragraph
and abstain from answering.
After downloading the files, you should have a :code:`squad` data folder that contains the following four files for training and
evaluation:
.. code::
.
|--squad
|-- v1.1/train-v1.1.json
|-- v1.1/dev-v1.1.json
|-- v2.0/train-v2.0.json
|-- v2.0/dev-v2.0.json
.. _model_training_question_answering:
Model Training
--------------
In the Question Answering Model, we are training a span prediction head on top of a pre-trained language model, such as
`BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ :cite:`nlp-qa-devlin2018bert`.
Unless the user provides a pre-trained checkpoint for the language model, the language model is initialized with the pre-trained model
from `HuggingFace Transformers <https://github.com/huggingface/transformers>`__.
Example of model configuration file for training the model can be found at: `NeMo/examples/nlp/question_answering/conf/question_answering_squad_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/conf/question_answering_squad_config.yaml>`__.
The specification can be grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the datasets: **model.dataset**, **model.train_ds**, **model.validation_ds**, **model.test_ds**
- Parameters that describe the model: **model**
More details about parameters in the spec file can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **pretrained_model** | string | Pretrained QA model model from ``list_available_models()`` or path to a ``.nemo`` file. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **do_training** | bool | If ``true``, starts training, otherwise, skips training and continues with evaluation/inference. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.dataset.version_2_with_negative** | bool | Set to ``true`` to allow examples without an answer, e.g. for SQuAD v2.0. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.dataset.do_lower_case** | bool | If ``true``, converts text to lower case, only import for inference/evaluation. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.dataset.use_cache** | bool | If ``true``, either loads all preprocessed data from cache or saves preprocessed data for future use. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.file** | string | The training file path. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.num_samples** | integer | The number of samples to use from the training dataset (use ``-1`` to specify all samples). |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.file** | string | The validation file path. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.num_samples** | integer | The number of samples to use from the validation dataset (use ``-1`` to specify all samples). |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **test_ds.file** | string | The test file path (optional). |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **test_ds.num_samples** | integer | The number of samples to use from the test dataset (use ``-1`` to specify all samples). |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
Example of the command for training the model:
.. code::
python question_answering_squad.py \
model.train_ds.file=<PATH_TO_TRAIN_DATA_FILE> \
model.validation_ds.file=<PATH_TO_VALIDATION_DATA_FILE> \
model.dataset.version_2_with_negative=<ALLOW_UNANSWERABLE_SAMPLES> \
model.dataset.do_lower_case=<DO_LOWER_CASE> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.gpus=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>]
.. Note::
The first time you train, it will take an extra 5-10 minutes to process the dataset. For future training runs, it will use the
processed dataset if :code:`model.dataset.use_cache=true`, which is automatically cached in the files in the same directory as
the data.
Required Arguments for Training
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`model.train_ds.file`: path to the training file in JSON format
- :code:`model.validation_ds.file`: path to the validation file in JSON format
Fine-tuning Procedure
^^^^^^^^^^^^^^^^^^^^^
Fine-tuning procedure and logs look similar to what's described in the Model Training section, with the addition of the model
that is initially loaded from a previously trained checkpoint, e.g. by specifying :code:`pretrained_model=<PRETRAINED_MODEL_NAME>`.
Inference
---------
An example script on how to run inference can be found at `examples/nlp/question_answering/question_answering_squad.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/question_answering_squad.py>`_.
To run inference with the pre-trained model, run:
.. code::
python question_answering_squad.py \
pretrained_model=<PRETRAINED_MODEL> \
model.dataset.version_2_with_negative=<ALLOW_UNANSWERABLE_SAMPLES> \
model.dataset.do_lower_case=<DO_LOWER_CASE> \
do_training=false \
model.validation_ds.file=<PATH_TO_INFERENCE_DATA_FILE>
Required Arguments for inference:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`pretrained_model`: pretrained QA Model model from ``list_available_models()`` or path to a ``.nemo`` file
Model Evaluation
----------------
An example script on how to evaluate the pre-trained model, can be found at `examples/nlp/question_answering/question_answering_squad.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/question_answering_squad.py>`_.
To run evaluation of the pre-trained model, run:
.. code::
python question_answering_squad.py \
pretrained_model=<PRETRAINED_MODEL> \
model.dataset.version_2_with_negative=<ALLOW_UNANSWERABLE_SAMPLES> \
model.dataset.do_lower_case=<DO_LOWER_CASE> \
do_training=false \
model.test_ds.file=<PATH_TO_TEST_DATA_FILE>
Required Arguments:
^^^^^^^^^^^^^^^^^^^
- :code:`pretrained_model`: pretrained QA model from `list_available_models()`` or path to a ``.nemo`` file
- :code:`model.test_ds.file`: path to test file
During evaluation of the :code:`test_ds`, the script generates the following metrics:
- :code:`Exact Match (EM)`
- :code:`F1`
More details about these metrics can be found `here <https://en.wikipedia.org/wiki/F-score>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-QA
:keyprefix: nlp-qa-
tts/NeMo/docs/source/nlp/sgd_qa.rst
+17
View File
@@ -0,0 +1,17 @@
.. _sgd_qa:
Dialogue State Tracking - SGD-QA Model
======================================
More details can be found in the paper
`SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services <https://arxiv.org/abs/2105.08049>`__ :cite:`nlp-sgdqa-zhang2021sgdqa`.
An example script on how to train the model can be found here: `NeMo/examples/nlp/dialogue_state_tracking/sgd_qa.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/dialogue_state_tracking/sgd_qa.py>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-SGDQA
:keyprefix: nlp-sgdqa-
tts/NeMo/docs/source/nlp/text_classification.rst
+169
View File
@@ -0,0 +1,169 @@
.. _text_classification:
Text Classification model
=========================
Text Classification is a sequence classification model based on BERT-based encoders. It can be used for a
variety of tasks like text classification, sentiment analysis, domain/intent detection for dialogue systems, etc.
The model takes a text input and predicts a label/class for the whole sequence. Megatron-LM and most of the BERT-based encoders
supported by HuggingFace including BERT, RoBERTa, and DistilBERT.
An example script on how to train the model can be found here: `NeMo/examples/nlp/text_classification/text_classification_with_bert.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/text_classification/text_classification_with_bert.py>`__.
The default configuration file for the model can be found at: `NeMo/examples/nlp/text_classification/conf/text_classification_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/text_classification/conf/text_classification_config.yaml>`__.
There is also a Jupyter notebook which explains how to work with this model. We recommend you try this model in the Jupyter notebook (can run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.):
`NeMo/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb <https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb>`__.
This tutorial shows an example of how run the Text Classification model on a sentiment analysis task. You may connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator) to run the notebook.
Data Format
-----------
The Text Classification model uses a simple text format as the dataset. It requires the data to be stored in TAB separated files
(``.tsv``) with two columns: sentence and label. Each line of the data file contains text sequences, where words are separated with spaces and the label is separated with ``[TAB]``, i.e.:
.. code::
[WORD][SPACE][WORD][SPACE][WORD][TAB][LABEL]
Labels need to be integers starting from ``0``. Some examples taken from the SST2 dataset, which is a two-class dataset for sentiment analysis:
.. code::
saw how bad this movie was 0
lend some dignity to a dumb story 0
the greatest musicians 1
You may need separate files for train, validation, and test with this format.
Dataset Conversion
------------------
If your dataset is stored in another format, you need to convert it to NeMo's format to use this model. There are some conversion
scripts available for the following datasets:
- SST2 :cite:`nlp-textclassify-socher2013`
- IMDB :cite:`nlp-textclassify-maas2011`
- ChemProt :cite:`nlp-textclassify-lim2018chemical`
- THUCnews :cite:`nlp-textclassify-li2007scalable`
You can convert them from their original format to NeMo's format. To convert the original datasets to NeMo's format, use the ``examples/text_classification/data/import_datasets.py`` script:
.. code:: python
python import_datasets.py \
--dataset_name DATASET_NAME \
--target_data_dir TARGET_PATH \
--source_data_dir SOURCE_PATH
It reads the dataset specified by ``DATASET_NAME`` from ``SOURCE_PATH`` and converts it to NeMo's format. It then saves the new
dataset at ``TARGET_PATH``.
Arguments:
- ``dataset_name``: name of the dataset to convert (``sst-2``, ``chemprot``, ``imdb``, and ``thucnews`` are currently supported)
- ``source_data_dir``: directory of your dataset
- ``target_data_dir``: directory to save the converted dataset
After the conversion, the ``TARGET_PATH`` should contain the following files:
.. code::
.
|--TARGET_PATH
|-- train.tsv
|-- dev.tsv
|-- test.tsv
Some datasets do not have the test set or their test set does not have any labels, therefore, the corresponding file may be missing.
Model Training
--------------
You may find an example of a config file to be used for training of the Text Classification model at `NeMo/examples/nlp/text_classification/conf/text_classification_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/text_classification/conf/text_classification_config.yaml>`__.
You can change any of these parameters directly from the config file or update them with the command-line arguments.
The config file of the Text Classification model contains three main sections of ``trainer``, ``exp_manager``, and ``model``. You can
find more details about the ``trainer`` and ``exp_manager`` at :doc:`./nlp_model`. Some sub-sections of the model section including
``tokenizer``, ``language_model``, and ``optim`` are shared among most of the NLP models. The details of these sections can be found
at :doc:`./nlp_model`.
Example of a command for training a Text Classification model on two GPUs for 50 epochs:
.. code::
python examples/nlp/text_classification/text_classification_with_bert.py \
model.training_ds.file_path=<TRAIN_FILE_PATH> \
model.validation_ds.file_path=<VALIDATION_FILE_PATH> \
trainer.max_epochs=50 \
trainer.gpus=[0,1] \
optim.name=adam \
optim.lr=0.0001 \
model.nemo_path=<NEMO_FILE_PATH>
At the start of each training experiment, there is a printed log of the experiment specification including any parameters added or
overridden via the command-line. It also shows additional information, such as which GPUs are available, where logs are saved,
and some samples from the datasets with their corresponding inputs to the model. It also provides some stats on the lengths of
sequences in the dataset.
After each epoch, you should see a summary table of metrics on the validation set which include the following metrics:
- :code:`Precision`
- :code:`Recall`
- :code:`F1`
At the end of training, NeMo saves the last checkpoint at the path specified by ``NEMO_FILE_PATH`` in ``.nemo`` format.
Model Arguments
^^^^^^^^^^^^^^^
The following table lists some of the model's parameters you can use in the config files or set them from the command-line when
training a model:
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Default** | **Description** |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.class_labels.class_labels_file** | string | ``null`` | Path to an optional file containing the labels; each line is the string label corresponding to a label. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.dataset.num_classes** | int | ``Required`` | Number of the categories or classes, ``0`` < ``Label`` < ``num_classes``. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.dataset.do_lower_case** | boolean | ``true`` for uncased models, ``false`` for cased | Specifies if inputs should be made lower case, would be set automatically if pre-trained model is used. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.dataset.max_seq_length** | int | ``256`` | Maximum length of the input sequences. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.dataset.class_balancing** | string | ``null`` | ``null`` or ``weighted_loss``. ``weighted_loss`` enables the weighted class balancing to handle unbalanced classes. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.dataset.use_cache** | boolean | ``false`` | Uses a cache to store the processed dataset, you can use it for large datasets for speed up. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.classifier_head.num_output_layers** | integer | ``2`` | Number of fully connected layers of the classifier on top of the BERT model. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **model.classifier_head.fc_dropout** | float | ``0.1`` | Dropout ratio of the fully connected layers. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.file_path** | string | ``Required`` | Path of the training ``.tsv`` file. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.batch_size** | integer | ``32`` | Data loader's batch size. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.num_workers** | integer | ``2`` | Number of worker threads for data loader. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.shuffle** | boolean | ``true`` (training), ``false`` (test and validation) | Shuffles data for each epoch. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.drop_last** | boolean | ``false`` | Specifies if last batch of data needs to get dropped if it is smaller than batch size. |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.pin_memory** | boolean | ``false`` | Enables pin_memory of PyTorch's data loader to enhance speed |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| **{training,validation,test}_ds.num_samples** | integer | ``-1`` | Number of samples to be used from the dataset; -1 means all samples |
+-----------------------------------------------+-----------------+--------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+
Model Evaluation and Inference
------------------------------
After saving the model in ``.nemo`` format, you can load the model and perform evaluation or inference on the model. You can find
some examples in the example script: `NeMo/examples/nlp/text_classification/text_classification_with_bert.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/text_classification/text_classification_with_bert.py>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-TEXTCLASSIFY
:keyprefix: nlp-textclassify-
tts/NeMo/docs/source/nlp/token_classification.rst
Executable
+219
View File
@@ -0,0 +1,219 @@
.. _token_classification:
Token Classification (Named Entity Recognition) Model
=====================================================
Token Classification model supports named entity recognition (NER) and other token level classification tasks, as long as the data
follows the format specified below.
We're going to use NER task throughout this section. NER, also referred to as entity chunking, identification or extraction, is the
task of detecting and classifying key information (entities) in text. In other words, a NER model takes a piece of text as input and
for each word in the text, the model identifies a category the word belongs to. For example, in a sentence: ``Mary lives in Santa Clara
and works at NVIDIA``, the model should detect that ``Mary`` is a person, ``Santa Clara`` is a location and ``NVIDIA`` is a company.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import TokenClassificationModel
# to get the list of pre-trained models
TokenClassificationModel.list_available_models()
# Download and load the pre-trained BERT-based model
model = TokenClassificationModel.from_pretrained("ner_en_bert")
# try the model on a few examples
model.add_predictions(['we bought four shirts from the nvidia gear store in santa clara.', 'NVIDIA is a company.'])
.. note::
We recommend you try this model in a Jupyter notebook (run on `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_.):
`NeMo/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb>`__.
Connect to an instance with a GPU (**Runtime** -> **Change runtime type** -> select **GPU** for the hardware accelerator).
An example script on how to train the model can be found here: `NeMo/examples/nlp/token_classification/token_classification_train.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_train.py>`__.
An example script on how to run evaluation and inference can be found here: `NeMo/examples/nlp/token_classification/token_classification_evaluate.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_evaluate.py>`__.
The default configuration file for the model can be found here: `NeMo/examples/nlp/token_classification/conf/token_classification_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/token_classification_config.yaml>`__.
.. _dataset_token_classification:
Data Input for Token Classification Model
-----------------------------------------
For pre-training or fine-tuning of the model, the data should be split into 2 files:
- ``text.txt``
- ``labels.txt``
Each line of the ``text.txt`` file contains text sequences, where words are separated with spaces, i.e.: ``[WORD] [SPACE] [WORD] [SPACE] [WORD]``.
The ``labels.txt`` file contains corresponding labels for each word in ``text.txt``, the labels are separated with spaces, i.e.: ``[LABEL] [SPACE] [LABEL] [SPACE] [LABEL]``.
Example of a ``text.txt`` file:
Jennifer is from New York City .
She likes ...
...
Corresponding ``labels.txt`` file:
B-PER O O B-LOC I-LOC I-LOC O
O O ...
...
Dataset Conversion
------------------
To convert an `IOB format <https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)>`__ (short for inside, outside, beginning) data to the format required for training, use
`examples/nlp/token_classification/data/import_from_iob_format.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/import_from_iob_format.py)>`_.
.. code::
# For conversion from IOB format, for example, for CoNLL-2003 dataset:
python import_from_iob_format.py --data_file=<PATH/TO/THE/FILE/IN/IOB/FORMAT>
Convert Dataset Required Arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`--data_file`: path to the file to convert from IOB to NeMo format
After running the above command, the data directory, where the :code:`--data_file` is stored, should contain :code:`text_*.txt` and :code:`labels_*.txt` files.
The default names for the training and evaluation in the :code:`conf/token_classification_config.yaml` are the following:
.. code::
.
|--data_dir
|-- labels_dev.txt
|-- labels_train.txt
|-- text_dev.txt
|-- text_train.txt
Training The Token Classification model
---------------------------------------
In the Token Classification model, we are jointly training a classifier on top of a pre-trained language model, such as
`BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ :cite:`nlp-ner-devlin2018bert`.
Unless the user provides a pre-trained checkpoint for the language model, the language model is initialized with the pre-trained model
from `HuggingFace Transformers <https://github.com/huggingface/transformers>`__.
Example of model configuration file for training the model can be found at: `NeMo/examples/nlp/token_classification/conf/token_classification_config.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/token_classification_config.yaml>`__.
The specification can be roughly grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the datasets: **model.dataset**, **model.train_ds**, **model.validation_ds**
- Parameters that describe the model: **model**
More details about parameters in the spec file can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.dataset.data_dir** | string | Path to the data converted to the specified above format. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.num_fc_layers** | integer | Number of fully connected layers. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.fc_dropout** | float | Dropout to apply to the input hidden states. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.head.activation** | string | Activation to use between fully connected layers. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.punct_head.use_transrormer_init** | bool | Whether to initialize the weights of the classifier head with the same approach used in Transformer. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.text_file** | string | Name of the text training file located at ``data_dir``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.labels_file** | string | Name of the labels training file located at ``data_dir``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **training_ds.num_samples** | integer | Number of samples to use from the training dataset, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.text_file** | string | Name of the text file for evaluation, located at ``data_dir``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.labels_file** | string | Name of the labels dev file located at ``data_dir``. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **validation_ds.num_samples** | integer | Number of samples to use from the dev set, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
For more information, see :ref:`nlp_model`.
Example of the command for training the model:
.. code::
python token_classification_train.py \
model.dataset.data_dir=<PATH_TO_DATA_DIR> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.gpus=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>]
Required Arguments for Training
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`model.dataset.data_dir`: path to the directory with pre-processed data.
.. note::
While the arguments are defined in the spec file, if you want to override these parameter definitions in the spec file and
experiment with them, use the command-line to define the parameter. For example, the sample spec file mentioned above has
:code:`validation_ds.batch_size` set to ``64``. However, if the GPU utilization can be optimized further by
using a larger batch size, override it to the desired value by adding the field :code:`validation_ds.batch_size=128` from
the command-line. You can repeat this with any of the parameters defined in the sample spec file.
Inference
---------
An example script on how to run inference can be found at `examples/nlp/token_classification/token_classification_evaluate.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_evaluate.py>`_.
To run inference with the pre-trained model, run:
.. code::
python token_classification_evaluate.py \
pretrained_model=<PRETRAINED_MODEL>
Required Arguments for Inference
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- :code:`pretrained_model`: pretrained Token Classification model from ``list_available_models()`` or path to a ``.nemo`` file. For example, ``ner_en_bert`` or ``your_model.nemo``
Model Evaluation
----------------
An example script on how to evaluate the pre-trained model can be found at `examples/nlp/token_classification/token_classification_evaluate.py <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_evaluate.py>`_.
To start evaluation of the pre-trained model, run:
.. code::
python token_classification_evaluate.py \
model.dataset.data_dir=<PATH/TO/DATA/DIR> \
pretrained_model=ner_en_bert \
model.test_ds.text_file=<text_*.txt> \
model.test_ds.labels_file=<labels_*.txt> \
model.dataset.max_seq_length=512
Required Arguments
^^^^^^^^^^^^^^^^^^
- :code:`pretrained_model`: pretrained Token Classification model from ``list_available_models()`` or path to a ``.nemo`` file. For example, ``ner_en_bert`` or ``your_model.nemo``
- :code:`model.dataset.data_dir`: path to the directory that containes :code:`model.test_ds.text_file` and :code:`model.test_ds.labels_file`
During evaluation of the :code:`test_ds`, the script generates a classification report that includes the following metrics:
- :code:`Precision`
- :code:`Recall`
- :code:`F1`
For more information, see `here <https://en.wikipedia.org/wiki/Precision_and_recall>`__.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-NER
:keyprefix: nlp-ner-