storytold/storyteller-ml
2
0
Fork 0
mirror of https://github.com/storytold/storyteller-ml.git synced 2026-10-17 00:23:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Introduce code from Rust monorepo. This is a mess

Browse Source
This commit is contained in:
Brandon Thomas
2022-02-28 05:43:54 -05:00
parent 921fb49f9a
commit ba326bfcfd
1058 changed files with 192192 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tts/NeMo/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb
+410
View File
@@ -0,0 +1,410 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"colab_type": "code",
"id": "djqHSONJ20X8",
"outputId": "c8c161ce-6a9a-49a8-9b92-b062d9da025f"
},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]"
]
},
{
"cell_type": "code",
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"id": "JFWG-jYCfvD7",
"colab_type": "code",
"colab": {}
},
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 124
},
"colab_type": "code",
"id": "CmvtH0pxHDQC",
"outputId": "fdaddea0-88fe-4207-ee7e-5a6a8ca360bb"
},
"outputs": [],
"source": [
"import os\n",
"import wget\n",
"from nemo.collections import nlp as nemo_nlp\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "J6AARcXXUEbs"
},
"source": [
"# Language models\n",
"\n",
"Natural Language Processing (NLP) field experienced a huge leap in recent years due to the concept of transfer learning enabled through pretrained language models.\n",
"\n",
"[BERT](https://arxiv.org/abs/1810.04805), [RoBERTa](https://arxiv.org/abs/1907.11692), [Megatron-LM](https://arxiv.org/abs/1909.08053), and many other proposed language models achieve state-of-the-art results on many NLP tasks, such as:\n",
"* question answering\n",
"* sentiment analysis\n",
"* named entity recognition and many others.\n",
"\n",
"In NeMo, most of the NLP models represent a pretrained language model followed by a Token Classification layer or a Sequence Classification layer or a combination of both. By changing the language model, you can improve the performance of your final model on the specific downstream task you are solving.\n",
"\n",
"With NeMo you can use either pretrain a BERT model from your data or use a pretrained language model from [HuggingFace transformers](https://github.com/huggingface/transformers) or [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) libraries.\n",
"\n",
"Let's take a look at the list of available pretrained language models, note the complete list of HuggingFace model could be found at [https://huggingface.co/models](https://huggingface.co/models):\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"colab_type": "code",
"id": "zp7F45bgX7SU",
"outputId": "b87752df-8e50-400b-9e1a-12e05bf175d2"
},
"outputs": [],
"source": [
"nemo_nlp.modules.get_pretrained_lm_models_list()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "OdpxEiirX8F-"
},
"source": [
"NLP models for downstream tasks use `get_lm_model` helper function to easily switch between language models from the list above to another:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"colab_type": "code",
"id": "5zllsvBojxuJ",
"outputId": "1c6e9878-2e82-460a-d890-30073bb11994"
},
"outputs": [],
"source": [
"# use any pretrained model name from the list above\n",
"nemo_nlp.modules.get_lm_model(pretrained_model_name='distilbert-base-uncased')"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "jEgEo0aPj3Ws"
},
"source": [
"All NeMo [NLP models](https://github.com/NVIDIA/NeMo/tree/main/examples/nlp) have an associated config file. As an example, let's examine the config file for the Named Entity Recognition (NER) model (more details about the model and the NER task could be found [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb))."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"colab_type": "code",
"id": "T1gA8PsJ13MJ",
"outputId": "326a3673-a814-422a-bde3-e61ba4f20fc4"
},
"outputs": [],
"source": [
"MODEL_CONFIG = \"token_classification_config.yaml\"\n",
"\n",
"# download the model's configuration file \n",
"if not os.path.exists(MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG)\n",
"else:\n",
" print ('Config file already exists')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"colab_type": "code",
"id": "mX3KmWMvSUQw",
"outputId": "bf9b174f-f3b1-4387-9c66-29170dec48cf"
},
"outputs": [],
"source": [
"# this line will print the entire config of the model\n",
"config = OmegaConf.load(MODEL_CONFIG)\n",
"print(OmegaConf.to_yaml(config))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "PKFbFDwzKUZB"
},
"source": [
"For this tutorial, we are interested in the language_model part of the Named Entity Recognition Model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 104
},
"colab_type": "code",
"id": "v6hlcCYyKFiY",
"outputId": "0fc8a4e4-978b-406f-e9ef-981f527ccedd"
},
"outputs": [],
"source": [
"print(OmegaConf.to_yaml(config.model.language_model))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "xldsDiH9ZY2h"
},
"source": [
"There might be slight differences from one model to another, but most of them have the following important parameters associated with the language model:\n",
"* `pretrained_model_name` - a name of the pretrained model from either HuggingFace or Megatron-LM libraries, for example, bert-base-uncased or megatron-bert-345m-uncased.\n",
"* `lm_checkpoint` - a path to the pretrained model checkpoint if, for example, you trained a BERT model with your data\n",
"* `config_file` - path to the model configuration file\n",
"* `config` or `config_dict` - path to the model configuration dictionary\n",
"\n",
"To modify the default language model, specify the desired language model name with the `model.language_model.pretrained_model_name` argument, like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "5r_1gIXwashY"
},
"outputs": [],
"source": [
"config.model.language_model.pretrained_model_name = 'roberta-base'"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "EVp4zvxPatga"
},
"source": [
"and then start the training as usual (please see [tutorials/nlp](https://github.com/NVIDIA/NeMo/tree/main/tutorials/nlp) for more details about training of a particular model). \n",
"\n",
"You can also provide a pretrained language model checkpoint and a configuration file if available.\n",
"\n",
"Note, that `pretrained_model_name` is used to set up both Language Model and Tokenizer.\n",
"\n",
"All the above holds for both HuggingFace and Megatron-LM pretrained language models. Let's separately examine some specifics of finetuning with Megatron-LM and HuggingFace models."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Wi_LBdSA4U6Q"
},
"source": [
"# Downstream tasks with Megatron and BioMegatron Language Models\n",
"\n",
"[Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. More details could be found at [Megatron-LM github repo](https://github.com/NVIDIA/Megatron-LM).\n",
"\n",
"To see the list of available Megatron-LM models in NeMo, run:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 121
},
"colab_type": "code",
"id": "0o-XPMrIQBmm",
"outputId": "7a7e0419-e9a1-48b3-b6bf-4d2863f69f46"
},
"outputs": [],
"source": [
"nemo_nlp.modules.get_megatron_lm_models_list()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "FM_ei7OSLn1X"
},
"source": [
"If you want to use one of the available Megatron-LM models, specify its name with `model.language_model.pretrained_model_name` argument, for example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "RmcQLoouME8k"
},
"outputs": [],
"source": [
"config.model.language_model.pretrained_model_name = 'megatron-bert-345m-uncased'"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "mla-BZexMM2u"
},
"source": [
"If you have a different checkpoint or a model configuration file, use these general Megatron-LM model names:\n",
"* `megatron-bert-uncased` or \n",
"* `megatron-bert-cased` \n",
"\n",
"and provide associated bert_config and bert_checkpoint files, as follows:\n",
"\n",
"`model.language_model.pretrained_model_name=megatron-bert-uncased \\\n",
"model.language_model.lm_checkpoint=<PATH_TO_CHECKPOINT> \\\n",
"model.language_model.config_file=<PAHT_TO_CONFIG>`\n",
" \n",
" or \n",
" \n",
"`model.language_model.pretrained_model_name=megatron-bert-cased \\\n",
"model.language_model.lm_checkpoint=<PATH_TO_CHECKPOINT> \\\n",
"model.language_model.config_file=<PAHT_TO_CONFIG>`\n",
"\n",
"The general Megatron-LM model names are used to download the correct vocabulary file needed to setup the model correctly. Note, the data preprocessing and model training is done in NeMo. Megatron-LM has its own set of training arguments (including tokenizer) that are ignored during finetuning in NeMo. Please see downstream task [config files and training scripts](https://github.com/NVIDIA/NeMo/tree/main/examples/nlp) for all NeMo supported arguments.\n",
"\n",
"## Download pretrained model\n",
"\n",
"With NeMo, the original and domain-specific Megatron-LM BERT models and model configuration files will be downloaded automatically, but they also could be downloaded with the links below:\n",
"\n",
"[Megatron-LM BERT Uncased 345M (~345M parameters): https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m/files?version=v0.1_uncased)\n",
"\n",
"[Megatron-LM BERT Cased 345M (~345M parameters): https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m/files?version=v0.1_cased)\n",
"\n",
"[BioMegatron-LM BERT Cased 345M (~345M parameters): https://ngc.nvidia.com/catalog/models/nvidia:biomegatron345mcased](https://ngc.nvidia.com/catalog/models/nvidia:biomegatron345mcased)\n",
"\n",
"[BioMegatron-LM BERT Uncased 345M (~345M parameters)](https://ngc.nvidia.com/catalog/models/nvidia:biomegatron345muncased): https://ngc.nvidia.com/catalog/models/nvidia:biomegatron345muncased"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "caafJmVLmEVD"
},
"source": [
"# Using any HuggingFace Pretrained Model\n",
"\n",
"Currently, there are 4 HuggingFace language models that have the most extensive support in [NeMo](https://github.com/NVIDIA/NeMo/tree/main/nemo/collections/nlp/modules/common/huggingface): \n",
"\n",
"* BERT\n",
"* RoBERTa\n",
"* ALBERT\n",
"* DistilBERT\n",
"\n",
"As was mentioned before, just set `model.language_model.pretrained_model_name` to the desired model name in your config and get_lm_model() will take care of the rest.\n",
"\n",
"If you want to use another language model from [https://huggingface.co/models](https://huggingface.co/models), use HuggingFace API directly in NeMo.\n",
"More details on model training could be found at [tutorials](https://github.com/NVIDIA/NeMo/tree/main/tutorials)."
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
tts/NeMo/tutorials/nlp/02_NLP_Tokenizers.ipynb
+616
View File
@@ -0,0 +1,616 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "X87q4jmW4Rmi"
},
"outputs": [],
"source": [
"BRANCH = 'v1.0.2'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "djqHSONJ20X8"
},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "CmvtH0pxHDQC"
},
"outputs": [],
"source": [
"import os\n",
"import wget\n",
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.collections import common as nemo_common\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "J6AARcXXUEbs"
},
"source": [
"# Tokenizers Background\n",
"\n",
"For Natural Language Processing, tokenization is an essential part of data preprocessing. It is the process of splitting a string into a list of tokens. One can think of token as parts like a word is a token in a sentence.\n",
"Depending on the application, different tokenizers are more suitable than others. \n",
"\n",
"\n",
"For example, a WordTokenizer that splits the string on any whitespace, would tokenize the following string \n",
"\n",
"\"My first program, Hello World.\" -> [\"My\", \"first\", \"program,\", \"Hello\", \"World.\"]\n",
"\n",
"To turn the tokens into numerical model input, the standard method is to use a vocabulary and one-hot vectors for [word embeddings](https://en.wikipedia.org/wiki/Word_embedding). If a token appears in the vocabulary, its index is returned, if not the index of the unknown token is returned to mitigate out-of-vocabulary (OOV).\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Fx7kTz00FL9W"
},
"source": [
"# Tokenizers in NeMo\n",
"\n",
"In NeMo, we support the most used tokenization algorithms. We offer a wrapper around [Hugging Faces's AutoTokenizer](https://huggingface.co/transformers/model_doc/auto.html#autotokenizer) - a factory class that gives access to all Hugging Face tokenizers. This includes particularly all BERT-like model tokenizers, such as BertTokenizer, AlbertTokenizer, RobertaTokenizer, GPT2Tokenizer. Apart from that, we also support other tokenizers such as WordTokenizer, CharTokenizer, and [Google's SentencePieceTokenizer](https://github.com/google/sentencepiece). \n",
"\n",
"\n",
"We make sure that all tokenizers are compatible with BERT-like models, e.g. BERT, Roberta, Albert, and Megatron. For that, we provide a high-level user API `get_tokenizer()`, which allows the user to instantiate a tokenizer model with only four input arguments: \n",
"* `tokenizer_name: str`\n",
"* `tokenizer_model: Optional[str] = None`\n",
"* `vocab_file: Optional[str] = None`\n",
"* `special_tokens: Optional[Dict[str, str]] = None`\n",
"\n",
"Hugging Face and Megatron tokenizers (which uses Hugging Face underneath) can be automatically instantiated by only `tokenizer_name`, which downloads the corresponding `vocab_file` from the internet. \n",
"\n",
"For SentencePieceTokenizer, WordTokenizer, and CharTokenizers `tokenizer_model` or/and `vocab_file` can be generated offline in advance using [`scripts/tokenizers/process_asr_text_tokenizer.py`](https://github.com/NVIDIA/NeMo/blob/main/scripts/process_asr_text_tokenizer.py)\n",
"\n",
"The tokenizers in NeMo are designed to be used interchangeably, especially when\n",
"used in combination with a BERT-based model.\n",
"\n",
"Let's take a look at the list of available tokenizers:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "zp7F45bgX7SU"
},
"outputs": [],
"source": [
"nemo_nlp.modules.get_tokenizer_list()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "9O-TJpIgG1Mt"
},
"source": [
"# Hugging Face AutoTokenizer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "5xQdZmj-IuZi"
},
"outputs": [],
"source": [
"# instantiate tokenizer wrapper using pretrained model name only\n",
"tokenizer1 = nemo_nlp.modules.get_tokenizer(tokenizer_name=\"bert-base-cased\")\n",
"\n",
"# the wrapper has a reference to the original HuggingFace tokenizer\n",
"print(tokenizer1.tokenizer)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "_Z7__VbTolBK"
},
"outputs": [],
"source": [
"# check vocabulary (this can be very long)\n",
"print(tokenizer1.tokenizer.vocab)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "z1CRAkRdonV9"
},
"outputs": [],
"source": [
"# show all special tokens if it has any\n",
"print(tokenizer1.tokenizer.all_special_tokens)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "SnE7LetCoCFY"
},
"outputs": [],
"source": [
"# instantiate tokenizer using custom vocabulary\n",
"vocab_file = \"myvocab.txt\"\n",
"vocab = [\"he\", \"llo\", \"world\"]\n",
"with open(vocab_file, 'w') as vocab_fp:\n",
" vocab_fp.write(\"\\n\".join(vocab))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "UEznqEwRo3Jw"
},
"outputs": [],
"source": [
"tokenizer2 = nemo_nlp.modules.get_tokenizer(tokenizer_name=\"bert-base-cased\", vocab_file=vocab_file)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "FsdSZ0pYoySH"
},
"outputs": [],
"source": [
"# Since we did not overwrite special tokens they should be the same as before\n",
"print(tokenizer1.tokenizer.all_special_tokens == tokenizer2.tokenizer.all_special_tokens )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding Special tokens\n",
"\n",
"We do not recommend overwriting special tokens for Hugging Face pretrained models, \n",
"since these are the commonly used default values. \n",
"\n",
"If a user still wants to overwrite the special tokens, specify some of the following keys:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "apQz4QPHoGRW"
},
"outputs": [],
"source": [
"special_tokens_dict = {\"unk_token\": \"<UNK>\", \n",
" \"sep_token\": \"<SEP>\", \n",
" \"pad_token\": \"<PAD>\", \n",
" \"bos_token\": \"<CLS>\", \n",
" \"mask_token\": \"<MASK>\",\n",
" \"eos_token\": \"<SEP>\",\n",
" \"cls_token\": \"<CLS>\"}\n",
"tokenizer3 = nemo_nlp.modules.get_tokenizer(tokenizer_name=\"bert-base-cased\",\n",
" vocab_file=vocab_file,\n",
" special_tokens=special_tokens_dict)\n",
"\n",
"# print newly set special tokens\n",
"print(tokenizer3.tokenizer.all_special_tokens)\n",
"# the special tokens should be different from the previous special tokens\n",
"print(tokenizer3.tokenizer.all_special_tokens != tokenizer1.tokenizer.all_special_tokens )"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab": {},
"colab_type": "code",
"id": "MA1ttDXro64B"
},
"source": [
"Notice, that if you specify tokens that were not previously included in the tokenizer's vocabulary file, new tokens will be added to the vocabulary file. You will see a message like this:"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab": {},
"colab_type": "code",
"id": "MA1ttDXro64B"
},
"source": [
"`['<MASK>', '<CLS>', '<SEP>', '<PAD>', '<SEP>', '<CLS>', '<UNK>'] \n",
" will be added to the vocabulary.\n",
" Please resize your model accordingly`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# A safer way to add special tokens is the following:\n",
"\n",
"# define your model\n",
"pretrained_model_name = 'bert-base-uncased'\n",
"model = nemo_nlp.modules.get_lm_model(pretrained_model_name=pretrained_model_name)\n",
"\n",
"# define pretrained tokenizer\n",
"tokenizer_default = nemo_nlp.modules.get_tokenizer(tokenizer_name=pretrained_model_name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tokenizer_default.text_to_tokens('<MY_NEW_TOKEN> and another word')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see in the above, the tokenizer splits `<MY_NEW_TOKEN>` into subtokens. Let's add this to the special tokens to make sure the tokenizer does not split this into subtokens."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"special_tokens = {'bos_token': '<BOS>',\n",
" 'cls_token': '<CSL>',\n",
" 'additional_special_tokens': ['<MY_NEW_TOKEN>', '<ANOTHER_TOKEN>']}\n",
"tokenizer_default.add_special_tokens(special_tokens_dict=special_tokens)\n",
"\n",
"# resize your model so that the embeddings for newly added tokens are updated during training/finetuning\n",
"model.resize_token_embeddings(tokenizer_default.vocab_size)\n",
"\n",
"# let's make sure the tokenizer doesn't split our special tokens into subtokens\n",
"tokenizer_default.text_to_tokens('<MY_NEW_TOKEN> and another word')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, the model doesn't break down our special token into the subtokens."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "y4Ptfo5dXmk_"
},
"source": [
"## Megatron model tokenizer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "5zllsvBojxuJ"
},
"outputs": [],
"source": [
"# Megatron tokenizers are instances of the Hugging Face BertTokenizer. \n",
"tokenizer4 = nemo_nlp.modules.get_tokenizer(tokenizer_name=\"megatron-bert-cased\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "jEgEo0aPj3Ws"
},
"source": [
"# Train custom tokenizer model and vocabulary from text file "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ykwKmREuPQE-"
},
"source": [
"We use the [`scripts/tokenizers/process_asr_text_tokenizer.py`](https://github.com/NVIDIA/NeMo/blob/main/scripts/process_asr_text_tokenizer.py) script to create a custom tokenizer model with its own vocabulary from an input file"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "OD5FUul3QGhA"
},
"outputs": [],
"source": [
"# download tokenizer script\n",
"script_file = \"process_asr_text_tokenizer.py\"\n",
"\n",
"if not os.path.exists(script_file):\n",
" print('Downloading script file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/scripts/tokenizers/process_asr_text_tokenizer.py')\n",
"else:\n",
" print ('Script already exists')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "T1gA8PsJ13MJ"
},
"outputs": [],
"source": [
"# Let's prepare some small text data for the tokenizer\n",
"data_text = \"NeMo is a toolkit for creating Conversational AI applications. \\\n",
"NeMo toolkit makes it possible for researchers to easily compose complex neural network architectures \\\n",
"for conversational AI using reusable components - Neural Modules. \\\n",
"Neural Modules are conceptual blocks of neural networks that take typed inputs and produce typed outputs. \\\n",
"Such modules typically represent data layers, encoders, decoders, language models, loss functions, or methods of combining activations. \\\n",
"The toolkit comes with extendable collections of pre-built modules and ready-to-use models for automatic speech recognition (ASR), \\\n",
"natural language processing (NLP) and text synthesis (TTS). \\\n",
"Built for speed, NeMo can utilize NVIDIA's Tensor Cores and scale out training to multiple GPUs and multiple nodes.\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "SaqyxVZgpMJk"
},
"outputs": [],
"source": [
"# Write the text data into a file\n",
"data_file=\"data.txt\"\n",
"\n",
"with open(data_file, 'w') as data_fp:\n",
" data_fp.write(data_text)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "iWz31g-opP9z"
},
"outputs": [],
"source": [
"# Some additional parameters for the tokenizer\n",
"# To tokenize at unigram, char or word boundary instead of using bpe, change --spe_type accordingly. \n",
"# More details see https://github.com/google/sentencepiece#train-sentencepiece-model\n",
"\n",
"tokenizer_spe_type = \"bpe\" # <-- Can be `bpe`, `unigram`, `word` or `char`\n",
"vocab_size = 32"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "mX3KmWMvSUQw"
},
"outputs": [],
"source": [
"! python process_asr_text_tokenizer.py --data_file=$data_file --data_root=. --vocab_size=$vocab_size --tokenizer=spe --spe_type=$tokenizer_spe_type"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "v6hlcCYyKFiY"
},
"outputs": [],
"source": [
"# See created tokenizer model and vocabulary\n",
"spe_model_dir=f\"tokenizer_spe_{tokenizer_spe_type}_v{vocab_size}\"\n",
"! ls $spe_model_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "EVp4zvxPatga"
},
"source": [
"# Use custom tokenizer for data preprocessing\n",
"## Example: SentencePiece for BPE"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "0o-XPMrIQBmm"
},
"outputs": [],
"source": [
"# initialize tokenizer with created tokenizer model, which inherently includes the vocabulary and specify optional special tokens\n",
"tokenizer_spe = nemo_nlp.modules.get_tokenizer(tokenizer_name=\"sentencepiece\", tokenizer_model=spe_model_dir+\"/tokenizer.model\", special_tokens=special_tokens_dict)\n",
"\n",
"# specified special tokens are added to the vocabuary\n",
"print(tokenizer_spe.vocab_size)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "FM_ei7OSLn1X"
},
"source": [
"## Example: WordTokenizer from Vocabulary"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "RmcQLoouME8k"
},
"outputs": [],
"source": [
"# If you want to use a simple tokenizer like WordTokenizer without first generating the tokenizer.model first \n",
"# we provide the alternative class WordTokenizer or CharTokenizer that takes a user vocabulary as input\n",
"\n",
"# initialize tokenizer with vocabulary and specify optional special tokens\n",
"tokenizer_word = nemo_nlp.modules.get_tokenizer(tokenizer_name=\"word\", vocab_file=vocab_file, special_tokens=special_tokens_dict)\n",
"\n",
"# specified special tokens are added to the vocabulary\n",
"print(tokenizer_word.vocab_size)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "caafJmVLmEVD"
},
"source": [
"# Using any tokenizer to tokenize text into BERT compatible input\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "ms4DAC4BpvqS"
},
"outputs": [],
"source": [
"text=\"hello world\"\n",
"\n",
"# create tokens\n",
"tokenized = [tokenizer_word.bos_token] + tokenizer_word.text_to_tokens(text) + [tokenizer_word.eos_token]\n",
"print(tokenized)\n",
"\n",
"# turn token into input_ids for a neural model, such as BERTModule\n",
"\n",
"print(tokenizer_word.tokens_to_ids(tokenized))"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "02_NLP_Tokenizers.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}
tts/NeMo/tutorials/nlp/Entity_Linking_Medical.ipynb
+756
View File
@@ -0,0 +1,756 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"\n",
"\n",
"## Install dependencies\n",
"!pip install wget\n",
"!pip install faiss-gpu\n",
"\n",
"## Install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"import faiss\n",
"import torch\n",
"import wget\n",
"import os\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from omegaconf import OmegaConf\n",
"from pytorch_lightning import Trainer\n",
"from IPython.display import display\n",
"from tqdm import tqdm\n",
"\n",
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Entity Linking"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Task Description\n",
"[Entity linking](https://en.wikipedia.org/wiki/Entity_linking) is the process of connecting concepts mentioned in natural language to their canonical forms stored in a knowledge base. For example, say a knowledge base contained the entity 'ID3452 influenza' and we wanted to process some natural language containing the sentence \"The patient has flu like symptoms\". An entity linking model would match the word 'flu' to the knowledge base entity 'ID3452 influenza', allowing for disambiguation and normalization of concepts referenced in text. Entity linking applications range from helping automate data ingestion to assisting in real time dialogue concept normalization. We will be focusing on entity linking in the medical domain for this demo, but the entity linking model, dataset, and training code within NVIDIA NeMo can be applied to other domains like finance and retail.\n",
"\n",
"Within NeMo and this tutorial we use the entity linking approach described in Liu et. al's NAACL 2021 \"[Self-alignment Pre-training for Biomedical Entity Representations](https://arxiv.org/abs/2010.11784v2)\". The main idea behind this approach is to reshape an initial concept embedding space such that synonyms of the same concept are pulled closer together and unrelated concepts are pushed further apart. The concept embeddings from this reshaped space can then be used to build a knowledge base embedding index. This index stores concept IDs mapped to their respective concept embeddings in a format conducive to efficient nearest neighbor search. We can link query concepts to their canonical forms in the knowledge base by performing a nearest neighbor search- matching concept query embeddings to the most similar concepts embeddings in the knowledge base index. \n",
"\n",
"In this tutorial we will be using the [faiss](https://github.com/facebookresearch/faiss) library to build our concept index."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Self Alignment Pretraining\n",
"Self-Alignment pretraining is a second stage pretraining of an existing encoder (called second stage because the encoder model can be further finetuned after this more general pretraining step). The dataset used during training consists of pairs of concept synonyms that map to the same ID. At each training iteration, we only select *hard* examples present in the mini batch to calculate the loss and update the model weights. In this context, a hard example is an example where a concept is closer to an unrelated concept in the mini batch than it is to the synonym concept it is paired with by some margin. I encourage you to take a look at [section 2 of the paper](https://arxiv.org/pdf/2010.11784.pdf) for a more formal and in depth description of how hard examples are selected.\n",
"\n",
"We then use a [metric learning loss](https://openaccess.thecvf.com/content_CVPR_2019/papers/Wang_Multi-Similarity_Loss_With_General_Pair_Weighting_for_Deep_Metric_Learning_CVPR_2019_paper.pdf) calculated from the hard examples selected. This loss helps reshape the embedding space. The concept representation space is rearranged to be more suitable for entity matching via embedding cosine similarity. \n",
"\n",
"Now that we have idea of what's going on, let's get started!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dataset Preprocessing"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"# Download data into project directory\n",
"PROJECT_DIR = \".\" #Change if you don't want the current directory to be the project dir\n",
"DATA_DIR = os.path.join(PROJECT_DIR, \"tiny_example_data\")\n",
"\n",
"if not os.path.isdir(os.path.join(DATA_DIR)):\n",
" wget.download('https://dldata-public.s3.us-east-2.amazonaws.com/tiny_example_data.zip',\n",
" os.path.join(PROJECT_DIR, \"tiny_example_data.zip\"))\n",
"\n",
" !unzip {PROJECT_DIR}/tiny_example_data.zip -d {PROJECT_DIR}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this tutorial we will be using a tiny toy dataset to demonstrate how to use NeMo's entity linking model functionality. The dataset includes synonyms for 12 medical concepts. Entity phrases with the same ID are synonyms for the same concept. For example, \"*chronic kidney failure*\", \"*gradual loss of kidney function*\", and \"*CKD*\" are all synonyms of concept ID 5. Here's the dataset before preprocessing:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" ID CONCEPT\n",
"0 1 Head ache\n",
"1 1 Headache\n",
"2 1 Migraine\n",
"3 1 Pain in the head\n",
"4 1 cephalgia\n",
"5 1 cephalalgia\n",
"6 2 heart attack\n",
"7 2 Myocardial infraction\n",
"8 2 necrosis of heart muscle\n",
"9 2 MI\n",
"10 3 CAD\n",
"11 3 Coronary artery disease\n",
"12 3 atherosclerotic heart disease\n",
"13 3 heart disease\n",
"14 3 damage of major heart blood vessels\n",
"15 4 myocardial ischemia\n",
"16 4 cardiac ischemia\n",
"17 4 reduced ability to pump blood\n",
"18 5 gradual loss of kidney function\n",
"19 5 kidneys cannot filter blood\n",
"20 5 chronic kidney disease\n",
"21 5 chronic kidney failure\n",
"22 5 CKD\n",
"23 6 alchohol intoxication\n",
"24 6 acute alchohol intoxication\n",
"25 6 alchohol poisoning\n",
"26 6 severe drunkenness\n",
"27 6 over consumption of alcohol\n",
"28 7 diabetes mellitus\n",
"29 7 diabetes\n",
"30 7 inability to process glucose\n",
"31 7 unable to take up sugar\n",
"32 7 Type 2 diabetes\n",
"33 8 Hyperinsulinemia\n",
"34 8 High blood sugar\n",
"35 8 abnormally high levels of insulin\n",
"36 9 Dipeptidyl peptidase-4 inhibitor\n",
"37 9 dpp-4 inhibitor\n",
"38 9 alogliptin\n",
"39 9 Nesina\n",
"40 9 Vipidia\n",
"41 10 hypoglycemia\n",
"42 10 low blood sugar\n",
"43 11 anticoagulants\n",
"44 11 blood thinners\n",
"45 11 Apixaban\n",
"46 11 Eliquis\n",
"47 12 Ibuprofen\n",
"48 12 Aspirin\n",
"49 12 over the counter nonsteroidal anti-inflammator...\n",
"50 12 NSAID\n"
]
}
],
"source": [
"raw_data = pd.read_csv(os.path.join(DATA_DIR, \"tiny_example_dev_data.csv\"), names=[\"ID\", \"CONCEPT\"], index_col=False)\n",
"print(raw_data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We've already paired off the concepts for this dataset with the format `ID concept_synonym1 concept_synonym2`. Here are the first ten rows:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" ID CONCEPT_SYN1 CONCEPT_SYN2\n",
"0 1 Pain in the head cephalgia\n",
"1 1 Pain in the head cephalalgia\n",
"2 1 Migraine cephalgia\n",
"3 1 Head ache Pain in the head\n",
"4 1 Head ache Migraine\n",
"5 1 Head ache cephalalgia\n",
"6 1 Headache Migraine\n",
"7 1 Migraine cephalalgia\n",
"8 1 cephalgia cephalalgia\n",
"9 1 Headache Pain in the head\n"
]
}
],
"source": [
"training_data = pd.read_table(os.path.join(DATA_DIR, \"tiny_example_train_pairs.tsv\"), names=[\"ID\", \"CONCEPT_SYN1\", \"CONCEPT_SYN2\"], delimiter='\\t')\n",
"print(training_data.head(10))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use the [Unified Medical Language System (UMLS)](https://www.nlm.nih.gov/research/umls/index.html) dataset for full medical domain entity linking training. The data contains over 9 million entities and is a table of medical concepts with their corresponding concept IDs (CUI). After [requesting a free license and making a UMLS Terminology Services (UTS) account](https://www.nlm.nih.gov/research/umls/index.html), the [entire UMLS dataset](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsknowledgesources.html) can be downloaded from the NIH's website. If you've cloned the NeMo repo you can run the data processing script located in `examples/nlp/entity_linking/data/umls_dataset_processing.py` on the full dataset. This script will take in the initial table of UMLS concepts and produce a .tsv file with each row formatted as `CUI\\tconcept_synonym1\\tconcept_synonym2`. Once the UMLS dataset .RRF file is downloaded, the script can be run from the `examples/nlp/entity_linking` directory like so: \n",
"```\n",
"python data/umls_dataset_processing.py\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Training"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Second stage pretrain a BERT Base encoder on the self-alignment pretraining task (SAP) for improved entity linking. Using a GPU, the model should take 5 minutes or less to train on this example dataset and training progress will be output below the cell."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# Download config\n",
"wget.download(\"https://raw.githubusercontent.com/vadam5/NeMo/main/examples/nlp/entity_linking/conf/tiny_example_entity_linking_config.yaml\",\n",
" os.path.join(PROJECT_DIR, \"tiny_example_entity_linking_config.yaml\"))\n",
"\n",
"# Load in config file\n",
"cfg = OmegaConf.load(os.path.join(PROJECT_DIR, \"tiny_example_entity_linking_config.yaml\"))\n",
"\n",
"# Set config file variables\n",
"cfg.project_dir = PROJECT_DIR\n",
"cfg.model.nemo_path = os.path.join(PROJECT_DIR, \"tiny_example_sap_bert_model.nemo\")\n",
"cfg.model.train_ds.data_file = os.path.join(DATA_DIR, \"tiny_example_train_pairs.tsv\")\n",
"cfg.model.validation_ds.data_file = os.path.join(DATA_DIR, \"tiny_example_validation_pairs.tsv\")"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"# Initialize the trainer and model\n",
"trainer = Trainer(**cfg.trainer)\n",
"exp_manager(trainer, cfg.get(\"exp_manager\", None))\n",
"model = nemo_nlp.models.EntityLinkingModel(cfg=cfg.model, trainer=trainer)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"# Train and save the model\n",
"trainer.fit(model)\n",
"model.save_to(cfg.model.nemo_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can run the script at `examples/nlp/entity_linking/self_alignment_pretraining.py` to train a model on a larger dataset. Run\n",
"\n",
"```\n",
"python self_alignment_pretraining.py project_dir=.\n",
"```\n",
"from the `examples/nlp/entity_linking` directory."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Evaluation\n",
"\n",
"Let's evaluate our freshly trained model and compare its performance with a BERT Base encoder that hasn't undergone self-alignment pretraining. We first need to restore our trained model and load our BERT Base Baseline model."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
"\n",
"# Restore second stage pretrained model\n",
"sap_model_cfg = cfg\n",
"sap_model_cfg.index.index_save_name = os.path.join(PROJECT_DIR, \"tiny_example_entity_linking_index\")\n",
"sap_model_cfg.index.index_ds.data_file = os.path.join(DATA_DIR, \"tiny_example_index_data.tsv\")\n",
"sap_model = nemo_nlp.models.EntityLinkingModel.restore_from(sap_model_cfg.model.nemo_path).to(device)\n",
"\n",
"# Load original model\n",
"base_model_cfg = OmegaConf.load(os.path.join(PROJECT_DIR, \"tiny_example_entity_linking_config.yaml\"))\n",
"\n",
"# Set train/val datasets to None to avoid loading datasets associated with training\n",
"base_model_cfg.model.train_ds = None\n",
"base_model_cfg.model.validation_ds = None\n",
"base_model_cfg.index.index_save_name = os.path.join(PROJECT_DIR, \"base_model_index\")\n",
"base_model_cfg.index.index_ds.data_file = os.path.join(DATA_DIR, \"tiny_example_index_data.tsv\")\n",
"base_model = nemo_nlp.models.EntityLinkingModel(base_model_cfg.model).to(device)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are going evaluate our model on a nearest neighbor task using top 1 and top 5 accuracies as our metric. We will be using a tiny example test knowledge base and test queries. For this evaluation we are going to be comparing every test query with every concept vector in our test set knowledge base. We will rank each item in the knowledge base by its cosine similarity with the test query. We'll then compare the IDs of the predicted most similar test knowledge base concepts with our ground truth query IDs to calculate top 1 and top 5 accuracies. For this metric higher is better."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"# Helper function to get data embeddings\n",
"def get_embeddings(model, dataloader):\n",
" embeddings, cids = [], []\n",
"\n",
" with torch.no_grad():\n",
" for batch in tqdm(dataloader):\n",
" input_ids, token_type_ids, attention_mask, batch_cids = batch\n",
" batch_embeddings = model.forward(input_ids=input_ids.to(device), \n",
" token_type_ids=token_type_ids.to(device), \n",
" attention_mask=attention_mask.to(device))\n",
"\n",
" # Accumulate index embeddings and their corresponding IDs\n",
" embeddings.extend(batch_embeddings.cpu().detach().numpy())\n",
" cids.extend(batch_cids)\n",
" \n",
" return embeddings, cids"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def evaluate(model, test_kb, test_queries, ks):\n",
" # Initialize knowledge base and query data loaders\n",
" test_kb_dataloader = model.setup_dataloader(test_kb, is_index_data=True)\n",
" test_query_dataloader = model.setup_dataloader(test_queries, is_index_data=True)\n",
" \n",
" # Get knowledge base and query embeddings\n",
" test_kb_embs, test_kb_cids = get_embeddings(model, test_kb_dataloader)\n",
" test_query_embs, test_query_cids = get_embeddings(model, test_query_dataloader)\n",
"\n",
" # Calculate the cosine distance between each query and knowledge base concept\n",
" score_matrix = np.matmul(np.array(test_query_embs), np.array(test_kb_embs).T)\n",
" accs = {k : 0 for k in ks}\n",
" \n",
" # Compare the knowledge base IDs of the knowledge base entities with \n",
" # the smallest cosine distance from the query \n",
" for query_idx in tqdm(range(len(test_query_cids))):\n",
" query_emb = test_query_embs[query_idx]\n",
" query_cid = test_query_cids[query_idx]\n",
" query_scores = score_matrix[query_idx]\n",
"\n",
" for k in ks:\n",
" topk_idxs = np.argpartition(query_scores, -k)[-k:]\n",
" topk_cids = [test_kb_cids[idx] for idx in topk_idxs]\n",
" \n",
" # If the correct query ID is amoung the top k closest kb IDs\n",
" # the model correctly linked the entity\n",
" match = int(query_cid in topk_cids)\n",
" accs[k] += match\n",
"\n",
" for k in ks:\n",
" accs[k] /= len(test_query_cids)\n",
" \n",
" return accs"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"# Create configs for our test data\n",
"test_kb = OmegaConf.create({\n",
" \"data_file\": os.path.join(DATA_DIR, \"tiny_example_test_kb.tsv\"),\n",
" \"max_seq_length\": 128,\n",
" \"batch_size\": 10,\n",
" \"shuffle\": False,\n",
"})\n",
"\n",
"test_queries = OmegaConf.create({\n",
" \"data_file\": os.path.join(DATA_DIR, \"tiny_example_test_queries.tsv\"),\n",
" \"max_seq_length\": 128,\n",
" \"batch_size\": 10,\n",
" \"shuffle\": False,\n",
"})\n",
"\n",
"ks = [1, 5]\n",
"\n",
"# Evaluate both models on our test data\n",
"base_accs = evaluate(base_model, test_kb, test_queries, ks)\n",
"base_accs[\"Model\"] = \"BERT Base Baseline\"\n",
"\n",
"sap_accs = evaluate(sap_model, test_kb, test_queries, ks)\n",
"sap_accs[\"Model\"] = \"BERT + SAP\"\n",
"\n",
"print(\"Top 1 and Top 5 Accuracy Comparison:\")\n",
"results_df = pd.DataFrame([base_accs, sap_accs], columns=[\"Model\", 1, 5])\n",
"results_df = results_df.style.set_properties(**{'text-align': 'left', }).set_table_styles([dict(selector='th', props=[('text-align', 'left')])])\n",
"display(results_df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The purpose of this section was to show an example of evaluating your entity linking model. This evaluation set contains very little data, and no serious conclusions should be drawn about model performance. Top 1 accuracy should be between 0.7 and 1.0 for both models and top 5 accuracy should be between 0.8 and 1.0. When evaluating a model trained on a larger dataset, you can use a nearest neighbors index to speed up the evaluation time."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building an Index"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To qualitatively observe the improvement we gain from the second stage pretraining, let's build two indices. One will be built with BERT base embeddings before self-alignment pretraining and one will be built with the model we just trained. Our knowledge base in this tutorial will be in the same domain and have some overlapping concepts as the training set. This data file is formatted as `ID\\tconcept`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `EntityLinkingDataset` class can load the data used for training the entity linking encoder as well as for building the index if the `is_index_data` flag is set to true. "
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"def build_index(cfg, model):\n",
" # Setup index dataset loader\n",
" index_dataloader = model.setup_dataloader(cfg.index.index_ds, is_index_data=True)\n",
" \n",
" # Get index dataset embeddings\n",
" embeddings, _ = get_embeddings(model, index_dataloader)\n",
" \n",
" # Train IVFFlat index using faiss\n",
" embeddings = np.array(embeddings)\n",
" quantizer = faiss.IndexFlatL2(cfg.index.dims)\n",
" index = faiss.IndexIVFFlat(quantizer, cfg.index.dims, cfg.index.nlist)\n",
" index = faiss.index_cpu_to_all_gpus(index)\n",
" index.train(embeddings)\n",
" \n",
" # Add concept embeddings to index\n",
" for i in tqdm(range(0, embeddings.shape[0], cfg.index.index_batch_size)):\n",
" index.add(embeddings[i:i+cfg.index.index_batch_size])\n",
"\n",
" # Save index\n",
" faiss.write_index(faiss.index_gpu_to_cpu(index), cfg.index.index_save_name)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"build_index(sap_model_cfg, sap_model.to(device))\n",
"build_index(base_model_cfg, base_model.to(device))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Entity Linking via Nearest Neighbor Search"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now it's time to query our indices! We are going to query both our index built with embeddings from BERT Base, and our index with embeddings built from the SAP BERT model we trained. Our sample query phrases will be \"*high blood sugar*\" and \"*head pain*\". \n",
"\n",
"To query our indices, we first need to get the embedding of each query from the corresponding encoder model. We can then pass these query embeddings into the faiss index which will perform a nearest neighbor search, using cosine distance to compare the query embedding with embeddings present in the index. Once we get a list of knowledge base index concept IDs most closely matching our query, all that is left to do is map the IDs to a representative string describing the concept. "
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"def query_index(cfg, model, index, queries, id2string):\n",
" # Get query embeddings from our entity linking encoder model\n",
" query_embs = get_query_embedding(queries, model).cpu().detach().numpy()\n",
" \n",
" # Use query embedding to find closest concept embedding in knowledge base\n",
" distances, neighbors = index.search(query_embs, cfg.index.top_n)\n",
" \n",
" # Get the canonical strings corresponding to the IDs of the query's nearest neighbors in the kb \n",
" neighbor_concepts = [[id2string[concept_id] for concept_id in query_neighbor] \\\n",
" for query_neighbor in neighbors]\n",
" \n",
" # Display most similar concepts in the knowledge base. \n",
" for query_idx in range(len(queries)):\n",
" print(f\"\\nThe most similar concepts to {queries[query_idx]} are:\")\n",
" for cid, concept, dist in zip(neighbors[query_idx], neighbor_concepts[query_idx], distances[query_idx]):\n",
" print(cid, concept, 1 - dist)\n",
"\n",
" \n",
"def get_query_embedding(queries, model):\n",
" # Tokenize our queries\n",
" model_input = model.tokenizer(queries,\n",
" add_special_tokens = True,\n",
" padding = True,\n",
" truncation = True,\n",
" max_length = 512,\n",
" return_token_type_ids = True,\n",
" return_attention_mask = True)\n",
" \n",
" # Pass tokenized input into model\n",
" query_emb = model.forward(input_ids=torch.LongTensor(model_input[\"input_ids\"]).to(device),\n",
" token_type_ids=torch.LongTensor(model_input[\"token_type_ids\"]).to(device),\n",
" attention_mask=torch.LongTensor(model_input[\"attention_mask\"]).to(device))\n",
" \n",
" return query_emb"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"# Load indices\n",
"sap_index = faiss.read_index(sap_model_cfg.index.index_save_name)\n",
"base_index = faiss.read_index(base_model_cfg.index.index_save_name)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"# Map concept IDs to one canonical string\n",
"index_data = open(sap_model_cfg.index.index_ds.data_file, \"r\", encoding='utf-8-sig')\n",
"id2string = {}\n",
"\n",
"for line in index_data:\n",
" cid, concept = line.split(\"\\t\")\n",
" id2string[int(cid) - 1] = concept.strip()"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 'Headache',\n",
" 1: 'Myocardial infraction',\n",
" 2: 'Coronary artery disease',\n",
" 3: 'myocardial ischemia',\n",
" 4: 'chronic kidney disease',\n",
" 5: 'alchohol intoxication',\n",
" 6: 'diabetes',\n",
" 7: 'Hyperinsulinemia',\n",
" 8: 'Nesina',\n",
" 9: 'hypoglycemia',\n",
" 10: 'anticoagulants',\n",
" 11: 'Ibuprofen'}"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"id2string"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"BERT Base output before Self Alignment Pretraining:\n",
"\n",
"The most similar concepts to high blood sugar are:\n",
"6 diabetes 0.9095035567879677\n",
"0 Headache 0.9046077281236649\n",
"8 Nesina 0.8512845635414124\n",
"\n",
"The most similar concepts to head pain are:\n",
"1 Myocardial infraction 0.7848672568798065\n",
"4 chronic kidney disease 0.7667323648929596\n",
"3 myocardial ischemia 0.761662557721138\n",
"\n",
"--------------------------------------------------\n",
"\n",
"SAP BERT output after Self Alignment Pretraining:\n",
"\n",
"The most similar concepts to high blood sugar are:\n",
"7 Hyperinsulinemia 0.28035879135131836\n",
"3 myocardial ischemia 0.054734230041503906\n",
"6 diabetes 0.053402602672576904\n",
"\n",
"The most similar concepts to head pain are:\n",
"0 Headache 0.6553303301334381\n",
"6 diabetes 0.19157332181930542\n",
"2 Coronary artery disease 0.12962108850479126\n",
"\n",
"--------------------------------------------------\n",
"\n"
]
}
],
"source": [
"# Some sample queries\n",
"queries = [\"high blood sugar\", \"head pain\"]\n",
"\n",
"# Query BERT Base\n",
"print(\"BERT Base output before Self Alignment Pretraining:\")\n",
"query_index(base_model_cfg, base_model, base_index, queries, id2string)\n",
"print(\"\\n\" + \"-\" * 50 + \"\\n\")\n",
"\n",
"# Query SAP BERT\n",
"print(\"SAP BERT output after Self Alignment Pretraining:\")\n",
"query_index(sap_model_cfg, sap_model, sap_index, queries, id2string)\n",
"print(\"\\n\" + \"-\" * 50 + \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Even after only training on this tiny amount of data, the qualitative performance boost from self-alignment pretraining is visible. The baseline model links \"*high blood sugar*\" to the entity \"*6 diabetes*\" while our SAP BERT model accurately links \"*high blood sugar*\" to \"*Hyperinsulinemia*\". Similarly, \"*head pain*\" and \"*Myocardial infraction*\" are not the same concept, but \"*head pain*\" and \"*Headache*\" are."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For larger knowledge bases keeping the default embedding size might be too large and cause out of memory issues. You can apply PCA or some other dimensionality reduction method to your data to reduce its memory footprint. Code for creating a text file of all the UMLS entities in the correct format needed to build an index and creating a dictionary mapping concept ids to canonical concept strings can be found here `examples/nlp/entity_linking/data/umls_dataset_processing.py`. \n",
"\n",
"The code for extracting knowledge base concept embeddings, training and applying a PCA transformation to the embeddings, building a faiss index and querying the index from the command line is located at `examples/nlp/entity_linking/build_index.py` and `examples/nlp/entity_linking/query_index.py`. \n",
"\n",
"If you've cloned the NeMo repo, both of these steps can be run as follows on the command line from the `examples/nlp/entity_linking/` directory.\n",
"\n",
"```\n",
"python data/umls_dataset_processing.py --index\n",
"python build_index.py --restore\n",
"python query_index.py --restore\n",
"```\n",
"By default the project directory will be \".\" but can be changed by adding the flag `--project_dir=<PATH>` after each of the above commands. Intermediate steps of the index building process are saved. In the occurrence of an error, previously completed steps do not need to be rerun. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Command Recap"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a recap of the commands and steps to repeat this process on the full UMLS dataset. \n",
"\n",
"1) Download the UMLS dataset file `MRCONSO.RRF` from the NIH website and place it in the `examples/nlp/entity_linking/data` directory.\n",
"\n",
"2) Run the following commands from the `examples/nlp/entity_linking` directory\n",
"```\n",
"python data/umls_dataset_processing.py\n",
"python self_alignment_pretraining.py project_dir=. \n",
"python data/umls_dataset_processing.py --index\n",
"python build_index.py --restore\n",
"python query_index.py --restore\n",
"```\n",
"The model will take ~24hrs to train on two GPUs and ~48hrs to train on one GPU. By default the project directory will be \".\" but can be changed by adding the flag `--project_dir=<PATH>` after each of the above commands and changing `project_dir=<PATH>` in the `self_alignment_pretraining.py` command. If you change the project directory, you should also move the `MRCONOSO.RRF` file to a `data` sub directory within the one you've specified. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As mentioned in the introduction, entity linking within NVIDIA NeMo is not limited to the medical domain. The same data processing and training steps can be applied to a variety of domains and use cases. You can edit the datasets used as well as training and loss function hyperparameters within your config file to better suit your domain."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
tts/NeMo/tutorials/nlp/GLUE_Benchmark.ipynb
+565
View File
@@ -0,0 +1,565 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "GLUE_Benchmark.ipynb",
"provenance": [],
"private_outputs": true,
"collapsed_sections": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU",
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"source": [],
"metadata": {
"collapsed": false
}
}
}
},
"cells": [
{
"cell_type": "code",
"metadata": {
"id": "o_0K1lsW1dj9",
"colab_type": "code",
"colab": {}
},
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"id": "JFWG-jYCfvD7",
"colab_type": "code",
"colab": {}
},
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dzqD2WDFOIN-",
"colab_type": "code",
"colab": {}
},
"source": [
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daYw_Xll2ZR9",
"colab_type": "text"
},
"source": [
"In this tutorial, we are going to describe how to finetune a BERT-like model based on [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) on [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://openreview.net/pdf?id=rJ4km2R5t7). \n",
"\n",
"# GLUE tasks\n",
"GLUE Benchmark includes 9 natural language understanding tasks:\n",
"\n",
"## Single-Sentence Tasks\n",
"\n",
"* CoLA - [The Corpus of Linguistic Acceptability](https://arxiv.org/abs/1805.12471) is a set of English sentences from published linguistics literature. The task is to predict whether a given sentence is grammatically correct or not.\n",
"* SST-2 - [The Stanford Sentiment Treebank](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf) consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence: positive or negative.\n",
"\n",
"## Similarity and Paraphrase tasks\n",
"\n",
"* MRPC - [The Microsoft Research Paraphrase Corpus](https://www.aclweb.org/anthology/I05-5002.pdf) is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent.\n",
"* QQP - [The Quora Question Pairs](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) dataset is a collection of question pairs from the community question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent.\n",
"* STS-B - [The Semantic Textual Similarity Benchmark](https://arxiv.org/abs/1708.00055) is a collection of sentence pairs drawn from news headlines, video, and image captions, and natural language inference data. The task is to determine how similar two sentences are.\n",
"\n",
"## Inference Tasks\n",
"\n",
"* MNLI - [The Multi-Genre Natural Language Inference Corpus](https://cims.nyu.edu/~sbowman/multinli/multinli_0.9.pdf) is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The task has the matched (in-domain) and mismatched (cross-domain) sections.\n",
"* QNLI - [The Stanford Question Answering Dataset](https://nlp.stanford.edu/pubs/rajpurkar2016squad.pdf) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question. The task is to determine whether the context sentence contains the answer to the question.\n",
"* RTE The Recognizing Textual Entailment (RTE) datasets come from a series of annual [textual entailment challenges](https://aclweb.org/aclwiki/Recognizing_Textual_Entailment). The task is to determine whether the second sentence is the entailment of the first one or not.\n",
"* WNLI - The Winograd Schema Challenge is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices (Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. 2012).\n",
"\n",
"All tasks are classification tasks, except for the STS-B task which is a regression task. All classification tasks are 2-class problems, except for the MNLI task which has 3-classes.\n",
"\n",
"More details about GLUE benchmark could be found [here](https://gluebenchmark.com/)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnuziSwJ1yEB",
"colab_type": "text"
},
"source": [
"# Datasets\n",
"\n",
"**To proceed further, you need to download the GLUE data.** For example, you can download [this script](https://gist.githubusercontent.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e/raw/17b8dd0d724281ed7c3b2aeeda662b92809aadd5/download_glue_data.py) using `wget` and then execute it by running:\n",
"\n",
"`python download_glue_data.py`\n",
"\n",
"use `--tasks TASK` if datasets for only selected GLUE tasks are needed\n",
"\n",
"After running the above commands, you will have a folder `glue_data` with data folders for every GLUE task. For example, data for MRPC task would be under glue_data/MRPC.\n",
"\n",
"This tutorial and [examples/nlp/glue_benchmark/glue_benchmark.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/glue_benchmark/glue_benchmark.py) work with all GLUE tasks without any modifications. For this tutorial, we are going to use MRPC task.\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "--wJ2891aIIE",
"colab_type": "code",
"colab": {}
},
"source": [
"# supported task names: [\"cola\", \"sst-2\", \"mrpc\", \"sts-b\", \"qqp\", \"mnli\", \"qnli\", \"rte\", \"wnli\"]\n",
"TASK = 'mrpc'\n",
"DATA_DIR = 'glue_data/MRPC'\n",
"WORK_DIR = \"WORK_DIR\"\n",
"MODEL_CONFIG = 'glue_benchmark_config.yaml'"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "qB0oLE4R9EhJ",
"colab_type": "code",
"colab": {}
},
"source": [
"! ls -l $DATA_DIR"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "gMWuU69pbUDe",
"colab_type": "text"
},
"source": [
"For each task, there are 3 files: `train.tsv, dev.tsv, and test.tsv`. Note, MNLI has 2 dev sets: matched and mismatched, evaluation on both dev sets will be done automatically."
]
},
{
"cell_type": "code",
"metadata": {
"id": "6UDPgadLN6SG",
"colab_type": "code",
"colab": {}
},
"source": [
"# let's take a look at the training data \n",
"! head -n 5 {DATA_DIR}/train.tsv"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "_whKCxfTMo6Y",
"colab_type": "text"
},
"source": [
"# Model configuration\n",
"\n",
"Now, let's take a closer look at the model's configuration and learn to train the model.\n",
"\n",
"GLUE model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model followed by a Sequence Regression module (for STS-B task) or Sequence classifier module (for the rest of the tasks).\n",
"\n",
"The model is defined in a config file which declares multiple important sections. They are:\n",
"- **model**: All arguments that are related to the Model - language model, a classifier, optimizer and schedulers, datasets and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning"
]
},
{
"cell_type": "code",
"metadata": {
"id": "T1gA8PsJ13MJ",
"colab_type": "code",
"colab": {}
},
"source": [
"# download the model's configuration file \n",
"config_dir = WORK_DIR + '/configs/'\n",
"os.makedirs(config_dir, exist_ok=True)\n",
"if not os.path.exists(config_dir + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/glue_benchmark/' + MODEL_CONFIG, config_dir)\n",
"else:\n",
" print ('config file is already exists')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "mX3KmWMvSUQw",
"colab_type": "code",
"colab": {}
},
"source": [
"# this line will print the entire config of the model\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)\n",
"print(OmegaConf.to_yaml(config))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZCgWzNBkaQLZ",
"colab_type": "text"
},
"source": [
"# Model Training\n",
"## Setting up Data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called **dataset**, **train_ds** and **validation_ds**. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. \n",
"So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.\n",
"\n",
"Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
"\n",
"Let's now add the data directory path, task name and output directory for saving predictions to the config."
]
},
{
"cell_type": "code",
"metadata": {
"id": "LQHCJN-ZaoLp",
"colab_type": "code",
"colab": {}
},
"source": [
"config.model.task_name = TASK\n",
"config.model.output_dir = WORK_DIR\n",
"config.model.dataset.data_dir = DATA_DIR"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "nB96-3sTc3yk",
"colab_type": "text"
},
"source": [
"## Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
"\n",
"Let's first instantiate a Trainer object"
]
},
{
"cell_type": "code",
"metadata": {
"id": "1tG4FzZ4Ui60",
"colab_type": "code",
"colab": {}
},
"source": [
"print(\"Trainer config - \\n\")\n",
"print(OmegaConf.to_yaml(config.trainer))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "knF6QeQQdMrH",
"colab_type": "code",
"colab": {}
},
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# for mixed precision training, uncomment the line below (precision should be set to 16 and amp_level to O1):\n",
"# config.trainer.amp_level = O1\n",
"\n",
"# remove distributed training flags\n",
"config.trainer.accelerator = None\n",
"\n",
"# setup max number of steps to reduce training time for demonstration purposes of this tutorial\n",
"config.trainer.max_steps = 128\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8IlEMdVxdr6p",
"colab_type": "text"
},
"source": [
"## Setting up a NeMo Experiment\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "8uztqGAmdrYt",
"colab_type": "code",
"colab": {}
},
"source": [
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"exp_dir = str(exp_dir)\n",
"exp_dir"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8tjLhUvL_o7_",
"colab_type": "text"
},
"source": [
"Before initializing the model, we might want to modify some of the model configs. For example, we might want to modify the pretrained BERT model and use [Megatron-LM BERT](https://arxiv.org/abs/1909.08053) or [AlBERT model](https://arxiv.org/abs/1909.11942):"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Xeuc2i7Y_nP5",
"colab_type": "code",
"colab": {}
},
"source": [
"# get the list of supported BERT-like models, for the complete list of HugginFace models, see https://huggingface.co/models\n",
"print(nemo_nlp.modules.get_pretrained_lm_models_list(include_external=True))\n",
"\n",
"# specify BERT-like model, you want to use, for example, \"megatron-bert-345m-uncased\" or 'bert-base-uncased'\n",
"PRETRAINED_BERT_MODEL = \"albert-base-v1\""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "RK2xglXyAUOO",
"colab_type": "code",
"colab": {}
},
"source": [
"# add the specified above model parameters to the config\n",
"config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fzNZNAVRjDD-",
"colab_type": "text"
},
"source": [
"Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
"Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
]
},
{
"cell_type": "code",
"metadata": {
"id": "NgsGLydWo-6-",
"colab_type": "code",
"colab": {}
},
"source": [
"model = nemo_nlp.models.GLUEModel(cfg=config.model, trainer=trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "kQ592Tx4pzyB",
"colab_type": "text"
},
"source": [
"## Monitoring training progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress."
]
},
{
"cell_type": "code",
"metadata": {
"id": "mTJr16_pp0aS",
"colab_type": "code",
"colab": {}
},
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "CFgAlaIdndjW",
"colab_type": "text"
},
"source": [
"Note, it’s recommended to finetune the model on each task separately. Also, based on [GLUE Benchmark FAQ#12](https://gluebenchmark.com/faq), there are might be some differences in dev/test distributions for QQP task and in train/dev for WNLI task."
]
},
{
"cell_type": "code",
"metadata": {
"id": "hUvnSpyjp0Dh",
"colab_type": "code",
"colab": {}
},
"source": [
"# start model training\n",
"trainer.fit(model)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ref1qSonGNhP",
"colab_type": "text"
},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally, you can also train the model with [examples/nlp/glue_benchmark/glue_benchmark.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/glue_benchmark/glue_benchmark.py).\n",
"\n",
"To run training script, use:\n",
"\n",
"`python glue_benchmark.py \\\n",
" model.dataset.data_dir=PATH_TO_DATA_DIR \\\n",
" model.task_name=TASK`\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KVPFofXaoKNE",
"colab_type": "text"
},
"source": [
"Average results after 3 runs:\n",
"\n",
"| Task | Metric | ALBERT-large | ALBERT-xlarge | Megatron-345m | BERT base paper | BERT large paper |\n",
"|-------|--------------------------|--------------|---------------|---------------|-----------------|------------------|\n",
"| CoLA | Matthew's correlation | 54.94 | 61.72 | 64.56 | 52.1 | 60.5 |\n",
"| SST-2 | Accuracy | 92.74 | 91.86 | 95.87 | 93.5 | 94.9 |\n",
"| MRPC | F1/Accuracy | 92.05/88.97 | 91.87/88.61 | 92.36/89.46 | 88.9/- | 89.3/- |\n",
"| STS-B | Person/Spearman corr. | 90.41/90.21 | 90.07/90.10 | 91.51/91.61 | -/85.8 | -/86.5 |\n",
"| QQP | F1/Accuracy | 88.26/91.26 | 88.80/91.65 | 89.18/91.91 | 71.2/- | 72.1/- |\n",
"| MNLI | Matched /Mismatched acc. | 86.69/86.81 | 88.66/88.73 | 89.86/89.81 | 84.6/83.4 | 86.7/85.9 |\n",
"| QNLI | Accuracy | 92.68 | 93.66 | 94.33 | 90.5 | 92.7 |\n",
"| RTE | Accuracy | 80.87 | 82.86 | 83.39 | 66.4 | 70.1 |\n",
"\n",
"WNLI task was excluded from the experiments due to the problematic WNLI set.\n",
"The dev sets were used for evaluation for ALBERT and Megatron models, and the test sets results for [the BERT paper](https://arxiv.org/abs/1810.04805).\n",
"\n",
"Hyperparameters used to get the results from the above table, could be found in the table below. Some tasks could be further finetuned to improve performance numbers, the tables are for a baseline reference only.\n",
"Each cell in the table represents the following parameters:\n",
"Number of GPUs used/ Batch Size/ Learning Rate/ Number of Epochs. For not specified parameters, please refer to the default parameters in the training script.\n",
"\n",
"| Task | ALBERT-large | ALBERT-xlarge | Megatron-345m |\n",
"|-------|--------------|---------------|---------------|\n",
"| CoLA | 1 / 32 / 1e-5 / 3 | 1 / 32 / 1e-5 / 10 | 4 / 16 / 2e-5 / 12 |\n",
"| SST-2 | 4 / 16 / 2e-5 / 5 | 4 / 16 / 2e-5 /12 | 4 / 16 / 2e-5 / 12 |\n",
"| MRPC | 1 / 32 / 1e-5 / 5 | 1 / 16 / 2e-5 / 5 | 1 / 16 / 2e-5 / 10 |\n",
"| STS-B | 1 / 16 / 2e-5 / 5 | 1 / 16 / 4e-5 / 12 | 4 / 16 / 3e-5 / 12 |\n",
"| QQP | 1 / 16 / 2e-5 / 5 | 4 / 16 / 1e-5 / 12 | 4 / 16 / 1e-5 / 12 |\n",
"| MNLI | 4 / 64 / 1e-5 / 5 | 4 / 32 / 1e-5 / 5 | 4 / 32 / 1e-5 / 5 | \n",
"| QNLI | 4 / 16 / 1e-5 / 5 | 4 / 16 / 1e-5 / 5 | 4 / 16 / 2e-5 / 5 | \n",
"| RTE | 1 / 16 / 1e-5 / 5 | 1 / 16 / 1e-5 / 12 | 4 / 16 / 3e-5 / 12 |\n"
]
}
]
}
tts/NeMo/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb
+587
View File
@@ -0,0 +1,587 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager\n",
"from nemo.utils import logging\n",
"\n",
"import os\n",
"import wget\n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Task Description\n",
"**Joint Intent and Slot classification** - is a task of classifying an Intent and detecting all relevant Slots (Entities)\n",
"for this Intent in a query.\n",
"For example, in the query: `What is the weather in Santa Clara tomorrow morning?`, we would like to classify the query\n",
"as a `weather` Intent, and detect `Santa Clara` as a `location` slot and `tomorrow morning` as a `date_time` slot.\n",
"Intents and Slots names are usually task specific and defined as labels in the training data.\n",
"This is a fundamental step that is executed in any task-driven Conversational Assistant.\n",
"\n",
"Our Bert based model implementation enables to train and then detect both of these tasks together.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Dataset and NeMo data format\n",
"\n",
"In this tutorial we are going to use a virtual assistant interaction data set that can be downloaded from here: https://github.com/xliuhw/NLU-Evaluation-Data.\n",
"There are about 10K training and 1K testing queries which cover 64 various Intents and 55 Slots. \n",
"\n",
"To work with NeMo NLP classification model, this dataset should be first converted to the NeMo format, which requires next files:\n",
"- **dict.intents.csv** - list of all intent names in the data. One line per an intent name.\n",
"- **dict.slots.csv** - list of all slot names in the data. One line per a slot name. It is possible to use both: B- I- notations, for separating between first and intermediate tokens for multi token slots. Or just use one slot type for each token of multi token slot. Our recommendation is to use later one, since it is simpler and there is no visible degradation in performance.\n",
"- **train.tsv/test.tsv** - contain original queries, one per line, and intent number separated by tab. For example: `what alarms do i have set right now\t0`. Intent numbers are according to the intent line in the intent dictionary file (dict.intents.csv) starting from 0. First line of these files contains a header line: `sentence \\tab label`.\n",
"- **train_slot.tvs/test_slot.tsv** - contain one line per a query, where instead each token there is a number of the token from the slots dictionary file (dict.slots.csv), starting from 0. Last 'out-of scope' token is usually located in the last line of the dictionary. Example: `54 0 0 54 54 12 12` (numbers separated by space). No header line in these files.\n",
"\n",
"NeMo provides **import_dataset.py** converter for few reference datasets (Assistant / Atis / Snips) which converts them to the NeMo data format for the Intent and Slot classification model. If you have your own annotated dataset in a different format, you will need to write a data converter. Possible recommended format for your own annotation, is to have one text file per all examples of one intent. With one line per query in a form like: `did i set an alarm to [alarm_type : wake up] in the [timeofday : morning]`, using brackets to define slot names. This is very similar to the assistant format from this example and you can use its converter to NeMo format with small changes. \n",
"\n",
"You can run this utility as follows:\n",
"\n",
"**python examples/nlp/intent_slot_classification/data/import_datasets.py --dataset_name=assistant --source_data_dir=source_dir_name --target_data_dir=target_dir_name**\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Download, preprocess and explore the dataset\n",
"## Download the dataset and convert it to the NeMo format"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# you can replace DATA_DIR and NEMO_DIR with your own locations\n",
"DATA_DIR = \".\"\n",
"NEMO_DIR = '.'\n",
"\n",
"# download the converter files from github for the purpose of this tutorial\n",
"wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/intent_slot_classification/data/import_datasets.py', NEMO_DIR)\n",
"wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/intent_slot_classification/data/assistant_utils.py', NEMO_DIR)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# download and unzip the example dataset from github\n",
"print('Downloading dataset...')\n",
"wget.download('https://github.com/xliuhw/NLU-Evaluation-Data/archive/master.zip', DATA_DIR)\n",
"! unzip {DATA_DIR}/NLU-Evaluation-Data-master.zip -d {DATA_DIR}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# convert the dataset to the NeMo format\n",
"!python {NEMO_DIR}/import_datasets.py --dataset_name=assistant --source_data_dir={DATA_DIR}/NLU-Evaluation-Data-master --target_data_dir={DATA_DIR}/nemo_format\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data exploration\n",
"You can see the dataset in both the original and NeMo's formats. We have here 65 different Intents and 55 Slots, which could be typical commands for virtual assistants. Out of scope slot has the name 'O' and is the last in the dictionary of Slots. And we can see examples of queries and also format of training intent and slot files. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# list of queries divided by intent files in the original training dataset\n",
"! ls -l {DATA_DIR}/NLU-Evaluation-Data-master/dataset/trainset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# print all intents from the NeMo format intent dictionary\n",
"!echo 'Intents: ' $(wc -l < {DATA_DIR}/nemo_format/dict.intents.csv)\n",
"! cat {DATA_DIR}/nemo_format/dict.intents.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# print all slots from the NeMo format slot dictionary\n",
"!echo 'Slots: ' $(wc -l < {DATA_DIR}/nemo_format/dict.slots.csv)\n",
"! cat {DATA_DIR}/nemo_format/dict.slots.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# examples from the intent training file\n",
"! head -n 10 {DATA_DIR}/nemo_format/train.tsv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# examples from the slot training file\n",
"! head -n 10 {DATA_DIR}/nemo_format/train_slots.tsv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Training model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model configuration\n",
"\n",
"Our Joint Intent and Slot classification model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model with an Intent and Slot Classification layer on top of it.\n",
"\n",
"All model and training parameters are defined in the **intent_slot_classification_config.yaml** config file. This file is located in the folder **examples/nlp/intent_slot_classification/conf/**. It contains 2 main sections:\n",
"- **model**: All arguments that are related to the Model - language model, token classifier, optimizer and schedulers, datasets and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning\n",
"\n",
"We will download the config file from repository for the purpose of the tutorial. If you have a version of NeMo installed locally, you can use it from the above folder."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# download the model config file from repository for the purpose of this example\n",
"wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/intent_slot_classification/conf/intent_slot_classification_config.yaml', NEMO_DIR)\n",
"\n",
"# print content of the config file\n",
"config_file = \"intent_slot_classification_config.yaml\"\n",
"print(config_file)\n",
"config = OmegaConf.load(config_file)\n",
"print(OmegaConf.to_yaml(config))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up Data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"The converter utility creates both training and evaluation files in the same directory, so we need to specify `model.data_dir` parameter to this directory. Also notice that some config lines, including `model.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
"\n",
"`config.model.intent_loss_weight` parameter - is a balance of training loss between Intent and Slot losses, a number between 0 to 1. Its default value is 0.6 which gives slightly higher priority to the Intent loss and it empirically works quite well. You can experiment with this value if you like.\n",
"Also you can try to change `config.model.class_balancing` parameter to `weighted_loss` and see if you get better accuracy.\n",
"\n",
"Let's now add the data directory path to the config."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"config.model.data_dir = f'{DATA_DIR}/nemo_format'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem. `config.trainer.max_epochs` - param defines number of training epochs. Usually 50-100 epochs or less should be enough to train on your data. Let's instantiate the Trainer object."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# for mixed precision training, uncomment the line below (precision should be set to 16 and amp_level to O1):\n",
"# config.trainer.amp_level = O1\n",
"\n",
"# remove distributed training flags\n",
"config.trainer.accelerator = None\n",
"\n",
"# setup a small number of epochs for demonstration purposes of this tutorial\n",
"config.trainer.max_epochs = 5\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up a NeMo Experiment\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it. Model check points during training will be saved in this directory. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"print(str(exp_dir))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Initializing the model and Training\n",
"\n",
"Initial statistics of the dataset will be displayed at the beginning of the training and then Intent and Slot classification report will be displayed after each training epoch."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# initialize the model\n",
"model = nemo_nlp.models.IntentSlotClassificationModel(config.model, trainer=trainer)\n",
"\n",
"# train\n",
"trainer.fit(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After training for 5 epochs, which should take no more than few minutes, you can expect training precision for this data set to be around these numbers (the accuracy will gradually continue to improve for this dataset up to about 50 epochs of training): \n",
"```\n",
"Intents:\n",
" label precision recall f1 support \n",
" alarm_query (label_id: 0) 94.74 94.74 94.74 19\n",
" alarm_remove (label_id: 1) 100.00 100.00 100.00 11\n",
" alarm_set (label_id: 2) 85.71 94.74 90.00 19\n",
" audio_volume_down (label_id: 3) 0.00 0.00 0.00 8\n",
" audio_volume_mute (label_id: 4) 100.00 86.67 92.86 15\n",
" audio_volume_up (label_id: 5) 56.52 100.00 72.22 13\n",
" calendar_query (label_id: 6) 55.00 57.89 56.41 19\n",
" calendar_remove (label_id: 7) 88.89 84.21 86.49 19\n",
" calendar_set (label_id: 8) 81.25 68.42 74.29 19\n",
" cooking_recipe (label_id: 9) 86.36 100.00 92.68 19\n",
" datetime_convert (label_id: 10) 0.00 0.00 0.00 8\n",
" datetime_query (label_id: 11) 65.52 100.00 79.17 19\n",
" email_addcontact (label_id: 12) 100.00 12.50 22.22 8\n",
" email_query (label_id: 13) 83.33 78.95 81.08 19\n",
" email_querycontact (label_id: 14) 62.50 78.95 69.77 19\n",
" email_sendemail (label_id: 15) 70.83 89.47 79.07 19\n",
" general_affirm (label_id: 16) 95.00 100.00 97.44 19\n",
" general_commandstop (label_id: 17) 100.00 100.00 100.00 19\n",
" general_confirm (label_id: 18) 100.00 100.00 100.00 19\n",
" general_dontcare (label_id: 19) 100.00 100.00 100.00 19\n",
" general_explain (label_id: 20) 100.00 94.74 97.30 19\n",
" general_joke (label_id: 21) 100.00 100.00 100.00 12\n",
" general_negate (label_id: 22) 95.00 100.00 97.44 19\n",
" general_praise (label_id: 23) 100.00 94.74 97.30 19\n",
" general_quirky (label_id: 24) 40.00 10.53 16.67 19\n",
" general_repeat (label_id: 25) 100.00 100.00 100.00 19\n",
" iot_cleaning (label_id: 26) 84.21 100.00 91.43 16\n",
" iot_coffee (label_id: 27) 94.74 94.74 94.74 19\n",
" iot_hue_lightchange (label_id: 28) 94.44 89.47 91.89 19\n",
" iot_hue_lightdim (label_id: 29) 100.00 83.33 90.91 12\n",
" iot_hue_lightoff (label_id: 30) 89.47 89.47 89.47 19\n",
" iot_hue_lighton (label_id: 31) 0.00 0.00 0.00 3\n",
" iot_hue_lightup (label_id: 32) 81.25 92.86 86.67 14\n",
" iot_wemo_off (label_id: 33) 60.00 100.00 75.00 9\n",
" iot_wemo_on (label_id: 34) 100.00 14.29 25.00 7\n",
" lists_createoradd (label_id: 35) 78.95 78.95 78.95 19\n",
" lists_query (label_id: 36) 78.95 78.95 78.95 19\n",
" lists_remove (label_id: 37) 90.00 94.74 92.31 19\n",
" music_likeness (label_id: 38) 70.59 66.67 68.57 18\n",
" music_query (label_id: 39) 77.78 73.68 75.68 19\n",
" music_settings (label_id: 40) 0.00 0.00 0.00 7\n",
" news_query (label_id: 41) 77.78 73.68 75.68 19\n",
" play_audiobook (label_id: 42) 90.00 94.74 92.31 19\n",
" play_game (label_id: 43) 80.00 84.21 82.05 19\n",
" play_music (label_id: 44) 53.85 73.68 62.22 19\n",
" play_podcasts (label_id: 45) 89.47 89.47 89.47 19\n",
" play_radio (label_id: 46) 93.75 78.95 85.71 19\n",
" qa_currency (label_id: 47) 95.00 100.00 97.44 19\n",
" qa_definition (label_id: 48) 85.00 89.47 87.18 19\n",
" qa_factoid (label_id: 49) 45.16 73.68 56.00 19\n",
" qa_maths (label_id: 50) 100.00 100.00 100.00 14\n",
" qa_stock (label_id: 51) 95.00 100.00 97.44 19\n",
" recommendation_events (label_id: 52) 94.44 89.47 91.89 19\n",
" recommendation_locations (label_id: 53) 94.74 94.74 94.74 19\n",
" recommendation_movies (label_id: 54) 100.00 100.00 100.00 10\n",
" social_post (label_id: 55) 90.00 94.74 92.31 19\n",
" social_query (label_id: 56) 94.74 100.00 97.30 18\n",
" takeaway_order (label_id: 57) 93.75 78.95 85.71 19\n",
" takeaway_query (label_id: 58) 85.71 94.74 90.00 19\n",
" transport_query (label_id: 59) 83.33 78.95 81.08 19\n",
" transport_taxi (label_id: 60) 100.00 100.00 100.00 18\n",
" transport_ticket (label_id: 61) 89.47 89.47 89.47 19\n",
" transport_traffic (label_id: 62) 100.00 100.00 100.00 19\n",
" weather_query (label_id: 63) 100.00 89.47 94.44 19\n",
" -------------------\n",
" micro avg 85.04 85.04 85.04 1076\n",
" macro avg 81.13 80.81 79.36 1076\n",
" weighted avg 84.10 85.04 83.54 1076\n",
" \n",
"Slots:\n",
" label precision recall f1 support \n",
" alarm_type (label_id: 0) 0.00 0.00 0.00 0\n",
" app_name (label_id: 1) 0.00 0.00 0.00 6\n",
" artist_name (label_id: 2) 0.00 0.00 0.00 21\n",
" audiobook_author (label_id: 3) 0.00 0.00 0.00 1\n",
" audiobook_name (label_id: 4) 0.00 0.00 0.00 18\n",
" business_name (label_id: 5) 60.00 56.60 58.25 53\n",
" business_type (label_id: 6) 0.00 0.00 0.00 24\n",
" change_amount (label_id: 7) 0.00 0.00 0.00 25\n",
" coffee_type (label_id: 8) 0.00 0.00 0.00 4\n",
" color_type (label_id: 9) 0.00 0.00 0.00 12\n",
" cooking_type (label_id: 10) 0.00 0.00 0.00 0\n",
" currency_name (label_id: 11) 84.09 75.51 79.57 49\n",
" date (label_id: 12) 57.95 91.07 70.83 112\n",
" definition_word (label_id: 13) 0.00 0.00 0.00 20\n",
" device_type (label_id: 14) 74.55 51.25 60.74 80\n",
" drink_type (label_id: 15) 0.00 0.00 0.00 0\n",
" email_address (label_id: 16) 0.00 0.00 0.00 14\n",
" email_folder (label_id: 17) 0.00 0.00 0.00 1\n",
" event_name (label_id: 18) 100.00 13.24 23.38 68\n",
" food_type (label_id: 19) 51.72 69.77 59.41 43\n",
" game_name (label_id: 20) 60.00 14.29 23.08 21\n",
" game_type (label_id: 21) 0.00 0.00 0.00 0\n",
" general_frequency (label_id: 22) 0.00 0.00 0.00 9\n",
" house_place (label_id: 23) 93.33 42.42 58.33 33\n",
" ingredient (label_id: 24) 0.00 0.00 0.00 6\n",
" joke_type (label_id: 25) 0.00 0.00 0.00 4\n",
" list_name (label_id: 26) 0.00 0.00 0.00 21\n",
" meal_type (label_id: 27) 0.00 0.00 0.00 0\n",
" media_type (label_id: 28) 0.00 0.00 0.00 37\n",
" movie_name (label_id: 29) 0.00 0.00 0.00 0\n",
" movie_type (label_id: 30) 0.00 0.00 0.00 0\n",
" music_album (label_id: 31) 0.00 0.00 0.00 0\n",
" music_descriptor (label_id: 32) 0.00 0.00 0.00 3\n",
" music_genre (label_id: 33) 0.00 0.00 0.00 9\n",
" news_topic (label_id: 34) 0.00 0.00 0.00 17\n",
" order_type (label_id: 35) 0.00 0.00 0.00 17\n",
" person (label_id: 36) 44.86 92.31 60.38 52\n",
" personal_info (label_id: 37) 0.00 0.00 0.00 20\n",
" place_name (label_id: 38) 71.25 77.03 74.03 148\n",
" player_setting (label_id: 39) 0.00 0.00 0.00 1\n",
" playlist_name (label_id: 40) 0.00 0.00 0.00 1\n",
" podcast_descriptor (label_id: 41) 0.00 0.00 0.00 13\n",
" podcast_name (label_id: 42) 0.00 0.00 0.00 4\n",
" radio_name (label_id: 43) 66.67 10.53 18.18 38\n",
" relation (label_id: 44) 0.00 0.00 0.00 17\n",
" song_name (label_id: 45) 0.00 0.00 0.00 22\n",
" time (label_id: 46) 70.27 78.20 74.02 133\n",
" time_zone (label_id: 47) 0.00 0.00 0.00 9\n",
" timeofday (label_id: 48) 0.00 0.00 0.00 28\n",
" transport_agency (label_id: 49) 0.00 0.00 0.00 9\n",
" transport_descriptor (label_id: 50) 0.00 0.00 0.00 0\n",
" transport_name (label_id: 51) 0.00 0.00 0.00 4\n",
" transport_type (label_id: 52) 78.38 82.86 80.56 35\n",
" weather_descriptor (label_id: 53) 0.00 0.00 0.00 17\n",
" O (label_id: 54) 92.42 98.80 95.50 5920\n",
" -------------------\n",
" micro avg 89.10 89.10 89.10 7199\n",
" macro avg 21.86 18.56 18.18 7199\n",
" weighted avg 84.42 89.10 86.01 7199\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluation\n",
"To see how the model performs, we can evaluate the performance of the trained model on a test data file. Here we would load the best checkpoint (the one with the lowest validation loss) and create a model (eval_model) from the checkpoint. We will use the same trainer for testing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# extract the path of the best checkpoint from the training, you may update it to any other saved checkpoint file\n",
"checkpoint_path = trainer.checkpoint_callback.best_model_path\n",
"\n",
"# load the model from this checkpoint\n",
"eval_model = nemo_nlp.models.IntentSlotClassificationModel.load_from_checkpoint(checkpoint_path=checkpoint_path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# we will setup testing data reusing the same config (test section)\n",
"eval_model.setup_test_data(test_data_config=config.model.test_ds)\n",
"\n",
"# run the evaluation on the test dataset\n",
"trainer.test(model=eval_model, ckpt_path=None, verbose=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Inference from Examples\n",
"Next step to see how the trained model will classify Intents and Slots for given queries from this domain. To improve the predictions you may need to train the model for more than 5 epochs.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"queries = [\n",
" 'set alarm for seven thirty am',\n",
" 'lower volume by fifty percent',\n",
" 'what is my schedule for tomorrow',\n",
"]\n",
"\n",
"pred_intents, pred_slots = eval_model.predict_from_examples(queries, config.model.test_ds)\n",
"\n",
"logging.info('The prediction results of some sample queries with the trained model:')\n",
"for query, intent, slots in zip(queries, pred_intents, pred_slots):\n",
" logging.info(f'Query : {query}')\n",
" logging.info(f'Predicted Intent: {intent}')\n",
" logging.info(f'Predicted Slots: {slots}')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally (eg. cloned from the Github), you can also train the model with the example script: `examples/nlp/intent_slot_classification/intent_slot_classification.py.`\n",
"This script contains an example on how to train, evaluate and perform inference with the IntentSlotClassificationModel.\n",
"\n",
"To run a training script, use:\n",
"\n",
"`cd examples/nlp/intent_slot_classification`\n",
"\n",
"`python intent_slot_classification.py model.data_dir=PATH_TO_DATA_DIR`\n",
"\n",
"By default, this script uses examples/nlp/intent_slot_classification/conf/intent_slot_classification_config.py config file, and you may update all the params inside of this config file or alternatively providing them in the command line."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
tts/NeMo/tutorials/nlp/Neural_Machine_Translation-Text2Sparql.ipynb
+2260
View File
File diff suppressed because one or more lines are too long
tts/NeMo/tutorials/nlp/Punctuation_and_Capitalization.ipynb
+735
View File
@@ -0,0 +1,735 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Punctuation_and_Capitalization.ipynb",
"provenance": [],
"private_outputs": true,
"collapsed_sections": [
"daYw_Xll2ZR9"
]
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU",
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"source": [],
"metadata": {
"collapsed": false
}
}
}
},
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"BRANCH = 'v1.0.2'"
]
},
{
"cell_type": "code",
"metadata": {
"id": "o_0K1lsW1dj9",
"colab_type": "code",
"colab": {}
},
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"id": "uglDB-pVh__t",
"colab_type": "code",
"colab": {}
},
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dzqD2WDFOIN-",
"colab_type": "code",
"colab": {}
},
"source": [
"from nemo.utils.exp_manager import exp_manager\n",
"from nemo.collections import nlp as nemo_nlp\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daYw_Xll2ZR9",
"colab_type": "text"
},
"source": [
"# Task Description\n",
"Automatic Speech Recognition (ASR) systems typically generate text with no punctuation and capitalization of the words. \n",
"This tutorial explains how to implement a model in NeMo that will predict punctuation and capitalization for each word in a sentence to make ASR output more readable and to boost performance of the named entity recognition, machine translation or text-to-speech models. \n",
"We'll show how to train a model for this task using a pre-trained BERT model. \n",
"For every word in our training dataset we’re going to predict:\n",
"\n",
"- punctuation mark that should follow the word and\n",
"- whether the word should be capitalized"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnuziSwJ1yEB",
"colab_type": "text"
},
"source": [
"# Dataset\n",
"This model can work with any dataset as long as it follows the format specified below. \n",
"The training and evaluation data is divided into *2 files: text.txt and labels.txt*. \n",
"Each line of the **text.txt** file contains text sequences, where words are separated with spaces: [WORD] [SPACE] [WORD] [SPACE] [WORD], for example:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TXFORGBv2Jqu",
"colab_type": "text"
},
"source": [
"\n",
"\n",
"```\n",
"when is the next flight to new york\n",
"the next flight is ...\n",
"...\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Y7ci55rM2QH8",
"colab_type": "text"
},
"source": [
"The **labels.txt** file contains corresponding labels for each word in text.txt, the labels are separated with spaces. Each label in labels.txt file consists of 2 symbols:\n",
"\n",
"- the first symbol of the label indicates what punctuation mark should follow the word (where O means no punctuation needed);\n",
"- the second symbol determines if a word needs to be capitalized or not (where U indicates that the word should be upper cased, and O - no capitalization needed.)\n",
"\n",
"In this tutorial, we are considering only commas, periods, and question marks the rest punctuation marks were removed. To use more punctuation marks, update the dataset to include desired labels, no changes to the model needed. \n",
"\n",
"Each line of the **labels.txt** should follow the format: \n",
"[LABEL] [SPACE] [LABEL] [SPACE] [LABEL] (for labels.txt). \n",
"For example, labels for the above text.txt file should be:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-94C1-864EW1",
"colab_type": "text"
},
"source": [
"\n",
"\n",
"```\n",
"OU OO OO OO OO OO OU ?U\n",
"OU OO OO OO ...\n",
"...\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VsEmwIPO4L4V",
"colab_type": "text"
},
"source": [
"The complete list of all possible labels for this task used in this tutorial is: `OO, ,O, .O, ?O, OU, ,U, .U, ?U.`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SL58EWkd2ZVb",
"colab_type": "text"
},
"source": [
"## Download and preprocess the data¶"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "THi6s1Qx2G1k",
"colab_type": "text"
},
"source": [
"In this notebook we are going to use a subset of English examples from the [Tatoeba collection of sentences](https://tatoeba.org/eng) this script will download and preprocess the Tatoeba data [NeMo/examples/nlp/token_classification/get_tatoeba_data.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/get_tatoeba_data.py). Note, for further experiments with the model, set NUM_SAMPLES=-1 and consider including other datasets to improve model performance. \n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "n8HZrDmr12_-",
"colab_type": "code",
"colab": {}
},
"source": [
"DATA_DIR = \"PATH_TO_DATA\"\n",
"WORK_DIR = \"PATH_TO_CHECKPOINTS_AND_LOGS\"\n",
"MODEL_CONFIG = \"punctuation_capitalization_config.yaml\"\n",
"\n",
"# model parameters\n",
"BATCH_SIZE = 128\n",
"MAX_SEQ_LENGTH = 64\n",
"LEARNING_RATE = 0.00002\n",
"NUM_SAMPLES = 10000"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "UOeeeCGqI-9c",
"colab_type": "code",
"colab": {}
},
"source": [
"## download get_tatoeba_data.py script to download and preprocess the Tatoeba data\n",
"os.makedirs(WORK_DIR, exist_ok=True)\n",
"if not os.path.exists(WORK_DIR + '/get_tatoeba_data.py'):\n",
" print('Downloading get_tatoeba_data.py...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/data/get_tatoeba_data.py', WORK_DIR)\n",
"else:\n",
" print ('get_tatoeba_data.py is already exists')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "H0ulD2TL13DR",
"colab_type": "code",
"colab": {}
},
"source": [
"# download and preprocess the data\n",
"# --clean_dir flag deletes raw Tataoeba data, remove the flag to avoid multiple data downloads if you want to experiment with the dataset size\n",
"! python $WORK_DIR/get_tatoeba_data.py --data_dir $DATA_DIR --num_sample $NUM_SAMPLES --clean_dir"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "pctMm2vsPlPT",
"colab_type": "text"
},
"source": [
"after execution of the above cell, your data folder will contain the following 4 files needed for training (raw Tatoeba data could be present if `--clean_dir` was not used):\n",
"- labels_dev.txt\n",
"- labels_train.txt\n",
"- text_dev.txt\n",
"- text_train.txt\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "KKwxXXezPvXF",
"colab_type": "code",
"colab": {}
},
"source": [
"! ls -l {DATA_DIR}"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "6UDPgadLN6SG",
"colab_type": "code",
"colab": {}
},
"source": [
"# let's take a look at the data \n",
"print('Text:')\n",
"! head -n 5 {DATA_DIR}/text_train.txt\n",
"\n",
"print('\\nLabels:')\n",
"! head -n 5 {DATA_DIR}/labels_train.txt"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daludzzL2Jba",
"colab_type": "text"
},
"source": [
"# Model Configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_whKCxfTMo6Y",
"colab_type": "text"
},
"source": [
"In the Punctuation and Capitalization Model, we are jointly training two token-level classifiers on top of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model: \n",
"- one classifier to predict punctuation and\n",
"- the other one - capitalization.\n",
"\n",
"The model is defined in a config file which declares multiple important sections. They are:\n",
"- **model**: All arguments that are related to the Model - language model, token classifiers, optimizer and schedulers, dataset and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning"
]
},
{
"cell_type": "code",
"metadata": {
"id": "T1gA8PsJ13MJ",
"colab_type": "code",
"colab": {}
},
"source": [
"# download the model's configuration file \n",
"config_dir = WORK_DIR + '/configs/'\n",
"os.makedirs(config_dir, exist_ok=True)\n",
"if not os.path.exists(config_dir + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG, config_dir)\n",
"else:\n",
" print ('config file is already exists')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "mX3KmWMvSUQw",
"colab_type": "code",
"colab": {}
},
"source": [
"# this line will print the entire config of the model\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)\n",
"print(OmegaConf.to_yaml(config))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZCgWzNBkaQLZ",
"colab_type": "text"
},
"source": [
"# Setting up Data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"If both training and evaluation files are located in the same directory, simply specify `model.dataset.data_dir`, like we are going to do below.\n",
"However, if your evaluation files are located in a different directory, or you want to use multiple datasets for evaluation, specify paths to the directory(ies) with evaluation file(s) in the following way:\n",
"\n",
"`model.validation_ds.ds_item=[PATH_TO_DEV1,PATH_TO_DEV2]` (Note no space between the paths and square brackets).\n",
"\n",
"Also notice that some configs, including `model.dataset.data_dir`, have `???` in place of paths, this values are required to be specified by the user.\n",
"\n",
"Let's now add the data directory path to the config."
]
},
{
"cell_type": "code",
"metadata": {
"id": "LQHCJN-ZaoLp",
"colab_type": "code",
"colab": {}
},
"source": [
"# in this tutorial train and dev data is located in the same folder, so it is enought to add the path of the data directory to our config\n",
"config.model.dataset.data_dir = DATA_DIR"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "nB96-3sTc3yk",
"colab_type": "text"
},
"source": [
"# Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem!\n",
"\n",
"Let's first instantiate a Trainer object!"
]
},
{
"cell_type": "code",
"metadata": {
"id": "1tG4FzZ4Ui60",
"colab_type": "code",
"colab": {}
},
"source": [
"print(\"Trainer config - \\n\")\n",
"print(OmegaConf.to_yaml(config.trainer))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "knF6QeQQdMrH",
"colab_type": "code",
"colab": {}
},
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# For mixed precision training, use precision=16 and amp_level=O1\n",
"\n",
"# Reduces maximum number of epochs to 1 for a quick training\n",
"config.trainer.max_epochs = 1\n",
"\n",
"# Remove distributed training flags\n",
"config.trainer.accelerator = None\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8IlEMdVxdr6p",
"colab_type": "text"
},
"source": [
"# Setting up a NeMo Experiment¶\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it!"
]
},
{
"cell_type": "code",
"metadata": {
"id": "8uztqGAmdrYt",
"colab_type": "code",
"colab": {}
},
"source": [
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"exp_dir = str(exp_dir)\n",
"exp_dir"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "6FI_nQsJo_11",
"colab_type": "text"
},
"source": [
"# Model Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8tjLhUvL_o7_",
"colab_type": "text"
},
"source": [
"Before initializing the model, we might want to modify some of the model configs. For example, we might want to modify the pretrained BERT model."
]
},
{
"cell_type": "code",
"metadata": {
"id": "Xeuc2i7Y_nP5",
"colab_type": "code",
"colab": {}
},
"source": [
"# complete list of supported BERT-like models\n",
"nemo_nlp.modules.get_pretrained_lm_models_list()\n",
"\n",
"PRETRAINED_BERT_MODEL = \"bert-base-uncased\""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "RK2xglXyAUOO",
"colab_type": "code",
"colab": {}
},
"source": [
"# add the specified above model parameters to the config\n",
"config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL\n",
"config.model.train_ds.batch_size = BATCH_SIZE\n",
"config.model.validation_ds.batch_size = BATCH_SIZE\n",
"config.model.optim.lr = LEARNING_RATE\n",
"config.model.train_ds.num_samples = NUM_SAMPLES\n",
"config.model.validation_ds.num_samples = NUM_SAMPLES\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": false,
"id": "gYKcziSsiAAd",
"colab_type": "text"
},
"source": [
"Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
"Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
]
},
{
"cell_type": "code",
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"id": "Yk2hJssviAAe",
"colab_type": "code",
"colab": {}
},
"source": [
"# initialize the model\n",
"# during this stage, the dataset and data loaders we'll be prepared for training and evaluation\n",
"config.trainer.max_epochs = 3\n",
"model = nemo_nlp.models.PunctuationCapitalizationModel(cfg=config.model, trainer=trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "kQ592Tx4pzyB",
"colab_type": "text"
},
"source": [
"## Monitoring training progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress."
]
},
{
"cell_type": "code",
"metadata": {
"id": "mTJr16_pp0aS",
"colab_type": "code",
"colab": {}
},
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "hUvnSpyjp0Dh",
"colab_type": "code",
"colab": {}
},
"source": [
"# start the training\n",
"trainer.fit(model)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPdzJVAgSFaJ",
"colab_type": "text"
},
"source": [
"# Inference\n",
"\n",
"To see how the model performs, let’s run inference on a few examples."
]
},
{
"cell_type": "code",
"metadata": {
"id": "DQhsamclRtxJ",
"colab_type": "code",
"colab": {}
},
"source": [
"# define the list of queiries for inference\n",
"queries = [\n",
" 'we bought four shirts and one mug from the nvidia gear store in santa clara',\n",
" 'what can i do for you today',\n",
" 'how are you',\n",
" 'how is the weather in',\n",
" ]\n",
"inference_results = model.add_punctuation_capitalization(queries)\n",
"print()\n",
"\n",
"for query, result in zip(queries, inference_results):\n",
" print(f'Query : {query}')\n",
" print(f'Combined: {result.strip()}\\n')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ref1qSonGNhP",
"colab_type": "text"
},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally, you can also train the model with [nlp/token_classification/punctuation_capitalization_train.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/punctuation_capitalization_train.py).\n",
"\n",
"To run training script, use:\n",
"\n",
"`python punctuation_and_capitalization_train.py model.dataset.data_dir=PATH_TO_DATA_DIR`\n",
"\n",
"Set NUM_SAMPLES=-1 and consider including other datasets to improve the performance of the model.\n",
"\n",
"# Finetuning model with your data\n",
"\n",
"When we were training the model from scratch, the datasets were prepared for training during the model initialization. When we are using a pretrained Punctuation and Capitalization model, before training, we need to setup training and evaluation data."
]
},
{
"cell_type": "code",
"metadata": {
"id": "4X1BahRlkaNf",
"colab_type": "code",
"colab": {}
},
"source": [
"# let's reload our pretrained model\n",
"pretrained_model = nemo_nlp.models.PunctuationCapitalizationModel.from_pretrained('punctuation_en_distilbert')\n",
"\n",
"# setup train and validation Pytorch DataLoaders\n",
"pretrained_model.update_data_dir(DATA_DIR)\n",
"pretrained_model.setup_training_data()\n",
"pretrained_model.setup_validation_data()\n",
"\n",
"# and now we can create a PyTorch Lightning trainer and call `fit` again\n",
"# for this tutorial we are setting fast_dev_run to True, and the trainer will run 1 training batch and 1 validation batch\n",
"# for actual model training, disable the flag\n",
"fast_dev_run = True\n",
"trainer = pl.Trainer(gpus=1, fast_dev_run=fast_dev_run)\n",
"trainer.fit(pretrained_model)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "l7A5FeiTl6Zd",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": null,
"outputs": []
}
]
}
tts/NeMo/tutorials/nlp/Question_Answering_Squad.ipynb
Executable
+724
View File
@@ -0,0 +1,724 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"accelerator": "GPU",
"colab": {
"name": "Question_Answering_Squad.ipynb",
"provenance": [],
"private_outputs": true,
"collapsed_sections": [
"daYw_Xll2ZR9"
],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"cells": [
{
"cell_type": "code",
"metadata": {
"id": "uRLPr0TnIAHO"
},
"source": [
"BRANCH = 'v1.0.2'"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "o_0K1lsW1dj9"
},
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dzqD2WDFOIN-"
},
"source": [
"from nemo.utils.exp_manager import exp_manager\n",
"from nemo.collections import nlp as nemo_nlp\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daYw_Xll2ZR9"
},
"source": [
"# Task Description\n",
"Given a question and a context both in natural language, predict the span within the context with a start and end position which indicates the answer to the question.\n",
"For every word in our training dataset we’re going to predict:\n",
"- likelihood this word is the start of the span \n",
"- likelihood this word is the end of the span \n",
"\n",
"We are using a pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) encoder with 2 span prediction heads for prediction start and end position of the answer. The span predictions are token classifiers consisting of a single linear layer. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnuziSwJ1yEB"
},
"source": [
"# Dataset\n",
"This model expects the dataset to be in [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, e.g. a JSON file for each dataset split. \n",
"In the following we will show example for a training file. Each title has one or multiple paragraph entries, each consisting of the text - \"context\", and question-answer entries. Each question-answer entry has:\n",
"* a question\n",
"* a globally unique id\n",
"* a boolean flag \"is_impossible\" which shows if the question is answerable or not\n",
"* in case the question is answerable one answer entry, which contains the text span and its starting character index in the context. If not answerable, the \"answers\" list is empty\n",
"\n",
"The evaluation files (for validation and testing) follow the above format except for it can provide more than one answer to the same question. \n",
"The inference file follows the above format except for it does not require the \"answers\" and \"is_impossible\" keywords.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TXFORGBv2Jqu"
},
"source": [
"\n",
"\n",
"```\n",
"{\n",
" \"data\": [\n",
" {\n",
" \"title\": \"Super_Bowl_50\", \n",
" \"paragraphs\": [\n",
" {\n",
" \"context\": \"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24\\u201310 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \\\"golden anniversary\\\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \\\"Super Bowl L\\\"), so that the logo could prominently feature the Arabic numerals 50.\", \n",
" \"qas\": [\n",
" {\n",
" \"question\": \"Where did Super Bowl 50 take place?\", \n",
" \"is_impossible\": \"false\", \n",
" \"id\": \"56be4db0acb8001400a502ee\", \n",
" \"answers\": [\n",
" {\n",
" \"answer_start\": \"403\", \n",
" \"text\": \"Santa Clara, California\"\n",
" }\n",
" ]\n",
" },\n",
" {\n",
" \"question\": \"What was the winning score of the Super Bowl 50?\", \n",
" \"is_impossible\": \"true\", \n",
" \"id\": \"56be4db0acb8001400a502ez\", \n",
" \"answers\": [\n",
" ]\n",
" }\n",
" ]\n",
" }\n",
" ]\n",
" }\n",
" ]\n",
"}\n",
"...\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SL58EWkd2ZVb"
},
"source": [
"## Download the data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "THi6s1Qx2G1k"
},
"source": [
"In this notebook we are going download the [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) dataset to showcase how to do training and inference. There are two datasets, SQuAD1.0 and SQuAD2.0. SQuAD 1.1, the previous version of the SQuAD dataset, contains 100,000+ question-answer pairs on 500+ articles. SQuAD2.0 dataset combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. \n",
"\n",
"\n",
"To download both datasets, we use [NeMo/examples/nlp/question_answering/get_squad.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/get_squad.py). \n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "tv3qXTTR_hBk"
},
"source": [
"# set the following paths\n",
"DATA_DIR = \"PATH_TO_DATA\"\n",
"WORK_DIR = \"PATH_TO_CHECKPOINTS_AND_LOGS\""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "qcz3Djem_hBn"
},
"source": [
"## download get_squad.py script to download and preprocess the SQuAD data\n",
"os.makedirs(WORK_DIR, exist_ok=True)\n",
"if not os.path.exists(WORK_DIR + '/get_squad.py'):\n",
" print('Downloading get_squad.py...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/question_answering/get_squad.py', WORK_DIR)\n",
"else:\n",
" print ('get_squad.py already exists')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "mpzsC41t_hBq"
},
"source": [
"# download and preprocess the data\n",
"! python $WORK_DIR/get_squad.py --destDir $DATA_DIR"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "m_HLLl6t_hBs"
},
"source": [
"after execution of the above cell, your data folder will contain a subfolder \"squad\" the following 4 files for training and evaluation\n",
"- v1.1/train-v1.1.json\n",
"- v1.1/dev-v1.1.json\n",
"- v2.0/train-v2.0.json\n",
"- v2.0/dev-v2.0.json"
]
},
{
"cell_type": "code",
"metadata": {
"id": "qYHcfxPL_hBt"
},
"source": [
"! ls -LR {DATA_DIR}/squad"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "bdpikZVreLlI"
},
"source": [
"## Data preprocessing\n",
"\n",
"The input into the model is the concatenation of two tokenized sequences:\n",
"\" [CLS] query [SEP] context [SEP]\".\n",
"This is the tokenization used for BERT, i.e. [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) Tokenizer, which uses the [Google's BERT vocabulary](https://github.com/google-research/bert). This tokenizer is configured with `model.tokenizer.tokenizer_name=bert-base-uncased` and is automatically instantiated using [Huggingface](https://huggingface.co/)'s API. \n",
"The benefit of this tokenizer is that this is compatible with a pretrained BERT model, from which we can finetune instead of training the question answering model from scratch. However, we also support other tokenizers, such as `model.tokenizer.tokenizer_name=sentencepiece`. Unlike the BERT WordPiece tokenizer, the [SentencePiece](https://github.com/google/sentencepiece) tokenizer model needs to be first created from a text file.\n",
"See [02_NLP_Tokenizers.ipynb](https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/02_NLP_Tokenizers.ipynb) for more details on how to use NeMo Tokenizers."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0q7Y7nyW_hBv"
},
"source": [
"# Data and Model Parameters\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B0b0Tn8M_hBv"
},
"source": [
"Note, this is only an example to showcase usage and is not optimized for accuracy. In the following, we will download and adjust the model configuration to create a toy example, where we only use a small fraction of the original dataset. \n",
"\n",
"In order to train the full SQuAD model, leave the model parameters from the configuration file unchanged. This sets NUM_SAMPLES=-1 to use the entire dataset, which will slow down performance significantly. We recommend to use bash script and multi-GPU to accelerate this. \n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "n8HZrDmr12_-"
},
"source": [
"# This is the model configuration file that we will download, do not change this\n",
"MODEL_CONFIG = \"question_answering_squad_config.yaml\"\n",
"\n",
"# model parameters, play with these\n",
"BATCH_SIZE = 12\n",
"MAX_SEQ_LENGTH = 384\n",
"# specify BERT-like model, you want to use\n",
"PRETRAINED_BERT_MODEL = \"bert-base-uncased\"\n",
"TOKENIZER_NAME = \"bert-base-uncased\" # tokenizer name\n",
"\n",
"# Number of data examples used for training, validation, test and inference\n",
"TRAIN_NUM_SAMPLES = VAL_NUM_SAMPLES = TEST_NUM_SAMPLES = 5000 \n",
"INFER_NUM_SAMPLES = 5\n",
"\n",
"TRAIN_FILE = f\"{DATA_DIR}/squad/v1.1/train-v1.1.json\"\n",
"VAL_FILE = f\"{DATA_DIR}/squad/v1.1/dev-v1.1.json\"\n",
"TEST_FILE = f\"{DATA_DIR}/squad/v1.1/dev-v1.1.json\"\n",
"INFER_FILE = f\"{DATA_DIR}/squad/v1.1/dev-v1.1.json\"\n",
"\n",
"INFER_PREDICTION_OUTPUT_FILE = \"output_prediction.json\"\n",
"INFER_NBEST_OUTPUT_FILE = \"output_nbest.json\"\n",
"\n",
"# training parameters\n",
"LEARNING_RATE = 0.00003\n",
"\n",
"# number of epochs\n",
"MAX_EPOCHS = 1"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daludzzL2Jba"
},
"source": [
"# Model Configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_whKCxfTMo6Y"
},
"source": [
"The model is defined in a config file which declares multiple important sections. They are:\n",
"- **model**: All arguments that will relate to the Model - language model, span prediction, optimizer and schedulers, datasets and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning"
]
},
{
"cell_type": "code",
"metadata": {
"id": "T1gA8PsJ13MJ"
},
"source": [
"# download the model's default configuration file \n",
"config_dir = WORK_DIR + '/configs/'\n",
"os.makedirs(config_dir, exist_ok=True)\n",
"if not os.path.exists(config_dir + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/question_answering/conf/{MODEL_CONFIG}', config_dir)\n",
"else:\n",
" print ('config file is already exists')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "mX3KmWMvSUQw"
},
"source": [
"# this line will print the entire default config of the model\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)\n",
"print(OmegaConf.to_yaml(config))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZCgWzNBkaQLZ"
},
"source": [
"## Setting up data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds, test_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"Specify data paths using `model.train_ds.file`, `model.valuation_ds.file` and `model.test_ds.file`.\n",
"\n",
"Let's now add the data paths to the config."
]
},
{
"cell_type": "code",
"metadata": {
"id": "LQHCJN-ZaoLp"
},
"source": [
"config.model.train_ds.file = TRAIN_FILE\n",
"config.model.validation_ds.file = VAL_FILE\n",
"config.model.test_ds.file = TEST_FILE\n",
"\n",
"config.model.train_ds.num_samples = TRAIN_NUM_SAMPLES\n",
"config.model.validation_ds.num_samples = VAL_NUM_SAMPLES\n",
"config.model.test_ds.num_samples = TEST_NUM_SAMPLES\n",
"\n",
"config.model.tokenizer.tokenizer_name = TOKENIZER_NAME"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "nB96-3sTc3yk"
},
"source": [
"# Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem!\n",
"\n",
"Let's first instantiate a Trainer object!"
]
},
{
"cell_type": "code",
"metadata": {
"id": "knF6QeQQdMrH"
},
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# For mixed precision training, use precision=16 and amp_level=O1\n",
"\n",
"config.trainer.max_epochs = MAX_EPOCHS\n",
"\n",
"# Remove distributed training flags if only running on a single GPU or CPU\n",
"config.trainer.accelerator = None\n",
"\n",
"print(\"Trainer config - \\n\")\n",
"print(OmegaConf.to_yaml(config.trainer))\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8IlEMdVxdr6p"
},
"source": [
"# Setting up a NeMo Experiment¶\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it!"
]
},
{
"cell_type": "code",
"metadata": {
"id": "8uztqGAmdrYt"
},
"source": [
"config.exp_manager.exp_dir = WORK_DIR\n",
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"exp_dir = str(exp_dir)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "D4jy28fbjekD"
},
"source": [
"# Using an Out-Of-Box Model"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Ins2ZzJckKKo"
},
"source": [
"# list available pretrained models\n",
"nemo_nlp.models.QAModel.list_available_models()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "iFnzHvkVk-S5"
},
"source": [
"# load pretained model\n",
"pretrained_model_name=\"qa_squadv1.1_bertbase\"\n",
"model = nemo_nlp.models.QAModel.from_pretrained(model_name=pretrained_model_name)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "6FI_nQsJo_11"
},
"source": [
"# Model Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8tjLhUvL_o7_"
},
"source": [
"Before initializing the model, we might want to modify some of the model configs."
]
},
{
"cell_type": "code",
"metadata": {
"id": "Xeuc2i7Y_nP5"
},
"source": [
"# complete list of supported BERT-like models\n",
"nemo_nlp.modules.get_pretrained_lm_models_list()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "RK2xglXyAUOO"
},
"source": [
"# add the specified above model parameters to the config\n",
"config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL\n",
"config.model.train_ds.batch_size = BATCH_SIZE\n",
"config.model.validation_ds.batch_size = BATCH_SIZE\n",
"config.model.test_ds.batch_size = BATCH_SIZE\n",
"config.model.optim.lr = LEARNING_RATE\n",
"\n",
"print(\"Updated model config - \\n\")\n",
"print(OmegaConf.to_yaml(config.model))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "NgsGLydWo-6-"
},
"source": [
"# initialize the model\n",
"# dataset we'll be prepared for training and evaluation during\n",
"model = nemo_nlp.models.QAModel(cfg=config.model, trainer=trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "kQ592Tx4pzyB"
},
"source": [
"## Monitoring Training Progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress."
]
},
{
"cell_type": "code",
"metadata": {
"id": "mTJr16_pp0aS"
},
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "hUvnSpyjp0Dh"
},
"source": [
"# start the training\n",
"trainer.fit(model)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "JxBiIKMlH8yv"
},
"source": [
"After training for 1 epoch, exact match on the evaluation data should be around 59.2%, F1 around 70.2%."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ynCLBmAWFVsM"
},
"source": [
"# Evaluation\n",
"\n",
"To see how the model performs, let’s run evaluation on the test dataset."
]
},
{
"cell_type": "code",
"metadata": {
"id": "XBMCoXAKFtSd"
},
"source": [
"model.setup_test_data(test_data_config=config.model.test_ds)\n",
"trainer.test(model)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPdzJVAgSFaJ"
},
"source": [
"# Inference\n",
"\n",
"To use the model for creating predictions, let’s run inference on the unlabeled inference dataset."
]
},
{
"cell_type": "code",
"metadata": {
"id": "DQhsamclRtxJ"
},
"source": [
"# # store test prediction under the experiment output folder\n",
"output_prediction_file = f\"{exp_dir}/{INFER_PREDICTION_OUTPUT_FILE}\"\n",
"output_nbest_file = f\"{exp_dir}/{INFER_NBEST_OUTPUT_FILE}\"\n",
"all_preds, all_nbests = model.inference(file=INFER_FILE, batch_size=5, num_samples=INFER_NUM_SAMPLES, output_nbest_file=output_nbest_file, output_prediction_file=output_prediction_file)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "sQpRIOaM_hCQ"
},
"source": [
"for _, item in all_preds.items():\n",
" print(f\"question: {item[0]} answer: {item[1]}\")\n",
"#The prediction file contains the predicted answer to each question id for the first TEST_NUM_SAMPLES.\n",
"! python -m json.tool $exp_dir/$INFER_PREDICTION_OUTPUT_FILE"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ref1qSonGNhP"
},
"source": [
"If you have NeMo installed locally, you can also train the model with \n",
"[NeMo/examples/nlp/question_answering/get_squad.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/question_answering/question_answering_squad.py).\n",
"\n",
"To run training script, use:\n",
"\n",
"`python question_answering_squad.py model.train_ds.file=TRAIN_FILE model.validation_ds.file=VAL_FILE model.test_ds.file=TEST_FILE`\n",
"\n",
"To improve the performance of the model, train with multi-GPU and a global batch size of 24. So if you use 8 GPUs with `trainer.gpus=8`, set `model.train_ds.batch_size=3`"
]
}
]
}
tts/NeMo/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb
+630
View File
@@ -0,0 +1,630 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "o_0K1lsW1dj9"
},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"scrolled": true
},
"outputs": [],
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "dzqD2WDFOIN-"
},
"outputs": [],
"source": [
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "daYw_Xll2ZR9"
},
"source": [
"In this tutorial, we are going to describe how to finetune BioMegatron - a [BERT](https://arxiv.org/abs/1810.04805)-like [Megatron-LM](https://arxiv.org/pdf/1909.08053.pdf) model pre-trained on large biomedical text corpus ([PubMed](https://pubmed.ncbi.nlm.nih.gov/) abstracts and full-text commercial use collection) - on [RE: Text mining chemical-protein interactions (CHEMPROT)](https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vi/track-5/).\n",
"\n",
"The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large.\n",
"There are some alternatives of BioMegatron, most notably [BioBERT](https://arxiv.org/abs/1901.08746). Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.\n",
"\n",
"A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Task Description\n",
"**Relation Extraction (RE)** can be regarded as a type of sentence classification.\n",
"\n",
"The task is to classify the relation of a [GENE] and [CHEMICAL] in a sentence, for example like the following:\n",
"```html\n",
"14967461.T1.T22\t<@CHEMICAL$> inhibitors currently under investigation include the small molecules <@GENE$> (Iressa, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux).\t<CPR:4>\n",
"14967461.T2.T22\t<@CHEMICAL$> inhibitors currently under investigation include the small molecules gefitinib (<@GENE$>, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux).\t<CPR:4>\n",
"```\n",
"to one of the following class:\n",
"\n",
"| Relation Class | Relations |\n",
"| ----------- | ----------- |\n",
"| CPR:3 | Upregulator and activator |\n",
"| CPR:4 | Downregulator and inhibitor |\n",
"| CPR:5 | Agonist |\n",
"| CPR:6 | Antagonist |\n",
"| CPR:9 | Substrate and product of |"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ZnuziSwJ1yEB"
},
"source": [
"# Datasets\n",
"\n",
"Details of ChemProt Relation Extraction task and the original data can be found on the [BioCreative VI website](https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vi/track-5/)\n",
"\n",
"ChemProt dataset pre-processed for easier consumption can be downloaded from [here](https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip) or [here](https://github.com/ncbi-nlp/BLUE_Benchmark/releases/download/0.1/bert_data.zip)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "--wJ2891aIIE"
},
"outputs": [],
"source": [
"TASK = 'ChemProt'\n",
"DATA_DIR = os.path.join(os.getcwd(), 'DATA_DIR')\n",
"RE_DATA_DIR = os.path.join(DATA_DIR, 'RE')\n",
"WORK_DIR = os.path.join(os.getcwd(), 'WORK_DIR')\n",
"MODEL_CONFIG = 'text_classification_config.yaml'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"os.makedirs(DATA_DIR, exist_ok=True)\n",
"os.makedirs(os.path.join(DATA_DIR, 'RE'), exist_ok=True)\n",
"os.makedirs(WORK_DIR, exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# download the dataset\n",
"wget.download('https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip?raw=true',\n",
" os.path.join(DATA_DIR, 'data_re.zip'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!unzip -o {DATA_DIR}/data_re.zip -d {RE_DATA_DIR}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "qB0oLE4R9EhJ"
},
"outputs": [],
"source": [
"! ls -l $RE_DATA_DIR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pre-process dataset\n",
"Let's convert the dataset into the format that is compatible for [NeMo text-classification module](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/text_classification/text_classification_with_bert.py)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/data/import_datasets.py')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! python import_datasets.py --dataset_name=chemprot --source_data_dir={RE_DATA_DIR} --target_data_dir={RE_DATA_DIR}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# let's take a look at the training data \n",
"! head -n 5 {RE_DATA_DIR}/train.tsv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# let's check the label mapping\n",
"! cat {RE_DATA_DIR}/label_mapping.tsv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is not necessary to have the mapping exactly like this - it can be different.\n",
"We use the same [mapping used by BioBERT](https://github.com/dmis-lab/biobert/blob/master/run_re.py#L438) so that comparison can be more straightforward."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "_whKCxfTMo6Y"
},
"source": [
"# Model configuration\n",
"\n",
"Now, let's take a closer look at the model's configuration and learn to train the model.\n",
"\n",
"The model is defined in a config file which declares multiple important sections. They are:\n",
"- **model**: All arguments that are related to the Model - language model, a classifier, optimizer and schedulers, datasets and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "T1gA8PsJ13MJ"
},
"outputs": [],
"source": [
"# download the model's configuration file \n",
"config_dir = WORK_DIR + '/configs/'\n",
"os.makedirs(config_dir, exist_ok=True)\n",
"if not os.path.exists(config_dir + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/conf/' + MODEL_CONFIG, config_dir)\n",
"else:\n",
" print ('config file is already exists')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "mX3KmWMvSUQw"
},
"outputs": [],
"source": [
"# this line will print the entire config of the model\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"config.model.train_ds.file_path = os.path.join(RE_DATA_DIR, 'train.tsv')\n",
"config.model.validation_ds.file_path = os.path.join(RE_DATA_DIR, 'dev.tsv')\n",
"config.model.task_name = 'chemprot'\n",
"# Note: these are small batch-sizes - increase as appropriate to available GPU capacity\n",
"config.model.train_ds.batch_size=8\n",
"config.model.validation_ds.batch_size=8\n",
"config.model.dataset.num_classes=6"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(OmegaConf.to_yaml(config))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ZCgWzNBkaQLZ"
},
"source": [
"# Model Training\n",
"## Setting up Data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called **dataset**, **train_ds** and **validation_ds**. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. \n",
"So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.\n",
"\n",
"Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
"\n",
"Let's now add the data directory path, task name and output directory for saving predictions to the config."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "LQHCJN-ZaoLp"
},
"outputs": [],
"source": [
"config.model.task_name = TASK\n",
"config.model.output_dir = WORK_DIR\n",
"config.model.dataset.data_dir = RE_DATA_DIR"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "nB96-3sTc3yk"
},
"source": [
"## Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
"\n",
"Let's first instantiate a Trainer object"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "1tG4FzZ4Ui60"
},
"outputs": [],
"source": [
"print(\"Trainer config - \\n\")\n",
"print(OmegaConf.to_yaml(config.trainer))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "knF6QeQQdMrH"
},
"outputs": [],
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"\n",
"# for PyTorch Native AMP set precision=16\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# remove distributed training flags\n",
"config.trainer.accelerator = 'DDP'\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "8IlEMdVxdr6p"
},
"source": [
"## Setting up a NeMo Experiment\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "8uztqGAmdrYt"
},
"outputs": [],
"source": [
"config.exp_manager.exp_dir = WORK_DIR\n",
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"exp_dir = str(exp_dir)\n",
"exp_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "8tjLhUvL_o7_"
},
"source": [
"Before initializing the model, we might want to modify some of the model configs. Here we are modifying it to use BioMegatron, [Megatron-LM BERT](https://arxiv.org/abs/1909.08053) pre-trained on [PubMed](https://pubmed.ncbi.nlm.nih.gov/) biomedical text corpus."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Xeuc2i7Y_nP5"
},
"outputs": [],
"source": [
"# complete list of supported BERT-like models\n",
"print(nemo_nlp.modules.get_pretrained_lm_models_list())\n",
"\n",
"# specify BERT-like model, you want to use, for example, \"megatron-bert-345m-uncased\" or 'bert-base-uncased'\n",
"PRETRAINED_BERT_MODEL = \"biomegatron-bert-345m-uncased\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "RK2xglXyAUOO"
},
"outputs": [],
"source": [
"# add the specified above model parameters to the config\n",
"config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "fzNZNAVRjDD-"
},
"source": [
"Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
"Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "NgsGLydWo-6-"
},
"outputs": [],
"source": [
"model = nemo_nlp.models.TextClassificationModel(cfg=config.model, trainer=trainer)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "kQ592Tx4pzyB"
},
"source": [
"## Monitoring training progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress.\n",
"If you're not using Colab, refer to [https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks](https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks) if you're facing issues with running the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "mTJr16_pp0aS"
},
"outputs": [],
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "hUvnSpyjp0Dh"
},
"outputs": [],
"source": [
"# start model training\n",
"trainer.fit(model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ref1qSonGNhP"
},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally, you can also train the model with `examples/nlp/text_classification/text_classification_with_bert.py.`\n",
"\n",
"To run training script, use:\n",
"\n",
"`python text_classification_with_bert.py \\\n",
" model.dataset.data_dir=PATH_TO_DATA_DIR \\\n",
" model.task_name=TASK`\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The training could take several minutes and the results should look something like:\n",
"\n",
"```\n",
"precision recall f1-score support\n",
" \n",
" 0 0.7328 0.8348 0.7805 115\n",
" 1 0.9402 0.9291 0.9346 7950\n",
" 2 0.8311 0.9146 0.8708 199\n",
" 3 0.6400 0.6302 0.6351 457\n",
" 4 0.8002 0.8317 0.8156 1093\n",
" 5 0.7228 0.7518 0.7370 548\n",
" \n",
" accuracy 0.8949 10362\n",
" macro avg 0.7778 0.8153 0.7956 10362\n",
" weighted avg 0.8963 0.8949 0.8954 10362\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "Relation_Extraction-BioMegatron.ipynb",
"private_outputs": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}
tts/NeMo/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb
+828
View File
@@ -0,0 +1,828 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "o_0K1lsW1dj9"
},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "95FHWXOVpUFp",
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dzqD2WDFOIN-"
},
"outputs": [],
"source": [
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "daYw_Xll2ZR9"
},
"source": [
"# Task Description\n",
"**Sentiment Analysis** is the task of detecting the sentiment in text. We model this problem as a simple form of a text classification problem. For example `Gollum's performance is incredible!` has a positive sentiment while `It's neither as romantic nor as thrilling as it should be.` has a negative sentiment.\n",
"."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnuziSwJ1yEB"
},
"source": [
"# Dataset\n",
"\n",
"In this tutorial we going to use [The Stanford Sentiment Treebank (SST-2)](https://nlp.stanford.edu/sentiment/index.html) corpus for sentiment analysis. This version of the dataset contains a collection of sentences with binary labels of positive and negative. It is a standard benchmark for sentence classification and is part of the GLUE Benchmark: https://gluebenchmark.com/tasks. Please download and unzip the SST-2 dataset from GLUE. It should contain three files of train.tsv, dev.tsv, and test.tsv which can be used for training, validation, and test respectively.\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qzcZ3nb_-SVT"
},
"source": [
"# NeMo Text Classification Data Format\n",
"\n",
"[TextClassificationModel](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/text_classification/text_classification_model.py) in NeMo supports text classification problems such as sentiment analysis or domain/intent detection for dialogue systems, as long as the data follows the format specified below. \n",
"\n",
"TextClassificationModel requires the data to be stored in TAB separated files (.tsv) with two columns of sentence and label. Each line of the data file contains text sequences, where words are separated with spaces and label separated with [TAB], i.e.: \n",
"\n",
"```\n",
"[WORD][SPACE][WORD][SPACE][WORD][TAB][LABEL]\n",
"```\n",
"\n",
"For example:\n",
"```\n",
"hide new secretions from the parental units[TAB]0\n",
"\n",
"that loves its characters and communicates something rather beautiful about human nature[TAB]1\n",
"...\n",
"```\n",
"\n",
"\n",
"If your dataset is stored in another format, you need to convert it to this format to use the TextClassificationModel. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SL58EWkd2ZVb"
},
"source": [
"## Download and Preprocess the Data\n",
"\n",
"First, you need to download the zipped file of the SST-2 dataset from the GLUE Benchmark website: https://gluebenchmark.com/tasks, and put it in the current folder. Then the following script would extract it into the data path specified by `DATA_DIR`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "n8HZrDmr12_-"
},
"outputs": [],
"source": [
"DATA_DIR = \"DATA_DIR\"\n",
"WORK_DIR = \"WORK_DIR\"\n",
"os.environ['DATA_DIR'] = DATA_DIR\n",
"\n",
"os.makedirs(WORK_DIR, exist_ok=True)\n",
"os.makedirs(DATA_DIR, exist_ok=True)\n",
"\n",
"! unzip -o SST-2.zip -d {DATA_DIR}\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "U8Ty5_S7Ye8h"
},
"source": [
"Now, the data folder should contain the following files:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L8vsyh3JZH26"
},
"source": [
"\n",
"\n",
"* train.tsv\n",
"* dev.tsv\n",
"* test.tsv\n",
"\n",
"\n",
"The format of `train.tsv` and `dev.tsv` is close to NeMo's format except to have an extra header line at the beginning of the files. We would remove these extra lines. But `test.tsv` has different format and labels are missing for this part of the data.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qB0oLE4R9EhJ"
},
"outputs": [],
"source": [
"! sed 1d {DATA_DIR}/SST-2/train.tsv > {DATA_DIR}/SST-2/train_nemo_format.tsv\n",
"! sed 1d {DATA_DIR}/SST-2/dev.tsv > {DATA_DIR}/SST-2/dev_nemo_format.tsv\n",
"! ls -l {DATA_DIR}/SST-2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6UDPgadLN6SG"
},
"outputs": [],
"source": [
"# let's take a look at the data \n",
"print('Contents (first 5 lines) of train.tsv:')\n",
"! head -n 5 {DATA_DIR}/SST-2/train_nemo_format.tsv\n",
"\n",
"print('\\nContents (first 5 lines) of test.tsv:')\n",
"! head -n 5 {DATA_DIR}/SST-2/test.tsv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "daludzzL2Jba"
},
"source": [
"# Model Configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_whKCxfTMo6Y"
},
"source": [
"Now, let's take a closer look at the model's configuration and learn to train the model from scratch and finetune the pretrained model.\n",
"\n",
"Our text classification model uses a pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model (or other BERT-like models) followed by a classification layer on the output of the first token ([CLS]).\n",
"\n",
"The model is defined in a config file which declares multiple important sections. The most important ones are:\n",
"- **model**: All arguments that are related to the Model - language model, tokenizer, head classifier, optimizer, schedulers, and datasets/data loaders.\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning including number of epochs, number of GPUs, precision level, etc.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "T1gA8PsJ13MJ"
},
"outputs": [],
"source": [
"# download the model's configuration file \n",
"MODEL_CONFIG = \"text_classification_config.yaml\"\n",
"CONFIG_DIR = WORK_DIR + '/configs/'\n",
"\n",
"os.makedirs(CONFIG_DIR, exist_ok=True)\n",
"if not os.path.exists(CONFIG_DIR + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/conf/' + MODEL_CONFIG, CONFIG_DIR)\n",
" print('Config file downloaded!')\n",
"else:\n",
" print ('config file already exists')\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mX3KmWMvSUQw"
},
"source": [
"## this line will print the entire config of the model\n",
"print(OmegaConf.to_yaml(config))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZCgWzNBkaQLZ"
},
"source": [
"# Model Training From Scratch\n",
"## Setting up data within the config\n",
"\n",
"We first need to set the num_classes in the config file which specifies the number of classes in the dataset. For SST-2, we have just two classes (0-positive and 1-negative). So we set the num_classes to 2. The model supports more than 2 classes too.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jFSMiWtlkaC5"
},
"outputs": [],
"source": [
"config.model.dataset.num_classes=2"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xCkc7QGikqPh"
},
"source": [
"\n",
"Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"Notice that some config lines, including `model.dataset.classes_num`, have `???` as their value, this means that values for these fields are required to be to be specified by the user. We need to specify and set the `model.train_ds.file_name`, `model.validation_ds.file_name`, and `model.test_ds.file_name` in the config file to the paths of the train, validation, and test files if they exist. We may do it by updating the config file or by setting them from the command line. \n",
"\n",
"Let's now set the train and validation paths in the config."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LQHCJN-ZaoLp"
},
"outputs": [],
"source": [
"config.model.train_ds.file_path = os.path.join(DATA_DIR, 'SST-2/train_nemo_format.tsv')\n",
"config.model.validation_ds.file_path = os.path.join(DATA_DIR, 'SST-2/dev_nemo_format.tsv')\n",
"# Name of the .nemo file where trained model will be saved.\n",
"config.save_to = 'trained-model.nemo'\n",
"config.export_to = 'trained-model.onnx'\n",
"\n",
"print(\"Train dataloader's config: \\n\")\n",
"# OmegaConf.to_yaml() is used to create a proper format for printing the train dataloader's config\n",
"# You may change other params like batch size or the number of samples to be considered (-1 means all the samples)\n",
"print(OmegaConf.to_yaml(config.model.train_ds))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nB96-3sTc3yk"
},
"source": [
"## Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning (PT) modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
"\n",
"Let's first instantiate a PT Trainer object by using the trainer section of the config."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1tG4FzZ4Ui60"
},
"outputs": [],
"source": [
"print(\"Trainer config - \\n\")\n",
"# OmegaConf.to_yaml() is used to create a proper format for printing the trainer config\n",
"print(OmegaConf.to_yaml(config.trainer))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XMVVs0INi5zj"
},
"source": [
"First you need to create a PT trainer with the params stored in the trainer's config. You may set the number of steps for training with max_steps or number of epochs with max_epochs in the trainer's config."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "knF6QeQQdMrH"
},
"outputs": [],
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"config.trainer.gpus = 1 if torch.cuda.is_available() else 0\n",
"\n",
"\n",
"# for mixed precision training, uncomment the lines below (precision should be set to 16 and amp_level to O1):\n",
"# config.trainer.precision = 16\n",
"# config.trainer.amp_level = O1\n",
"\n",
"# disable distributed training when using Colab to prevent the errors\n",
"config.trainer.accelerator = None\n",
"\n",
"# setup max number of steps to reduce training time for demonstration purposes of this tutorial\n",
"# Training stops when max_step or max_epochs is reached (earliest)\n",
"config.trainer.max_epochs = 5\n",
"\n",
"# instantiates a PT Trainer object by using the trainer section of the config\n",
"trainer = pl.Trainer(**config.trainer)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8IlEMdVxdr6p"
},
"source": [
"## Setting up the NeMo Experiment¶\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Kl5IdnV3O8y"
},
"source": [
"NeMo has an experiment manager that handles the logging and saving checkpoints for us, so let's setup it. We need the PT trainer and the exp_manager config:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8uztqGAmdrYt"
},
"outputs": [],
"source": [
"# The experiment manager of a trainer object can not be set twice. We repeat the trainer creation code again here to prevent getting error when this cell is executed more than once. \n",
"trainer = pl.Trainer(**config.trainer)\n",
"\n",
"# exp_dir specifies the path to store the the checkpoints and also the logs, it's default is \"./nemo_experiments\"\n",
"# You may set it by uncommentig the following line\n",
"# config.exp_manager.exp_dir = 'LOG_CHECKPOINT_DIR'\n",
"\n",
"# OmegaConf.to_yaml() is used to create a proper format for printing the trainer config\n",
"print(OmegaConf.to_yaml(config.exp_manager))\n",
"\n",
"exp_dir = exp_manager(trainer, config.exp_manager)\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"print(exp_dir)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8tjLhUvL_o7_"
},
"source": [
"Before initializing the model, we might want to modify some of the model configs. For example, we might want to modify the pretrained BERT model to another model. The default model is `bert-base-uncased`. We support a variety of models including all the models available in HuggingFace, and Megatron."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xeuc2i7Y_nP5"
},
"outputs": [],
"source": [
"# complete list of supported BERT-like models\n",
"print(nemo_nlp.modules.get_pretrained_lm_models_list())\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RK2xglXyAUOO"
},
"outputs": [],
"source": [
"# specify the BERT-like model, you want to use\n",
"# set the `model.language_modelpretrained_model_name' parameter in the config to the model you want to use\n",
"config.model.language_model.pretrained_model_name = \"bert-base-uncased\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fzNZNAVRjDD-"
},
"source": [
"Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders will also be prepared for the training and validation.\n",
"\n",
"Also, the pretrained BERT model will be automatically downloaded. Note it can take up to a few minutes depending on the size of the chosen BERT model for the first time you create the model. If your dataset is large, it also may take some time to read and process all the datasets. \n",
"\n",
"Now we can create the model with the model config and the trainer object like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NgsGLydWo-6-"
},
"outputs": [],
"source": [
"model = nemo_nlp.models.TextClassificationModel(cfg=config.model, trainer=trainer)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kQ592Tx4pzyB"
},
"source": [
"## Monitoring Training Progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mTJr16_pp0aS"
},
"outputs": [],
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MW_JVIi5z68e"
},
"source": [
"## Training\n",
"\n",
"You may start the training by using the trainer.fit() method. The number of steps/epochs of the training are specified already in the config of the trainer and you may update them before creating the trainer."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hUvnSpyjp0Dh"
},
"outputs": [],
"source": [
"# start model training\n",
"trainer.fit(model)\n",
"model.save_to(config.save_to)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPdzJVAgSFaJ"
},
"source": [
"# Evaluation\n",
"\n",
"To see how the model performs, we can run evaluate and test the performance of the trained model on a data file. Here we would load the best checkpoint (the one with the lowest validation loss) and create a model (eval_model) from the checkpoint. We would also create a new trainer (eval_trainer) to show how it is done when training is done and you have just the checkpoints. If you want to perform the evaluation in the same script as the training's script, you may still use the same model and trainer you used for training."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "92PB0iTqNnW-"
},
"outputs": [],
"source": [
"# extract the path of the best checkpoint from the training, you may update it to any checkpoint\n",
"checkpoint_path = trainer.checkpoint_callback.best_model_path\n",
"# Create an evaluation model and load the checkpoint\n",
"eval_model = nemo_nlp.models.TextClassificationModel.load_from_checkpoint(checkpoint_path=checkpoint_path)\n",
"\n",
"# create a dataloader config for evaluation, the same data file provided in validation_ds is used here\n",
"# file_path can get updated with any file\n",
"eval_config = OmegaConf.create({'file_path': config.model.validation_ds.file_path, 'batch_size': 64, 'shuffle': False, 'num_samples': -1})\n",
"eval_model.setup_test_data(test_data_config=eval_config)\n",
"#eval_dataloader = eval_model._create_dataloader_from_config(cfg=eval_config, mode='test')\n",
"\n",
"# a new trainer is created to show how to evaluate a checkpoint from an already trained model\n",
"# create a copy of the trainer config and update it to be used for final evaluation\n",
"eval_trainer_cfg = config.trainer.copy()\n",
"eval_trainer_cfg.gpus = 1 if torch.cuda.is_available() else 0 # it is safer to perform evaluation on single GPU as PT is buggy with the last batch on multi-GPUs\n",
"eval_trainer_cfg.accelerator = None # 'ddp' is buggy with test process in the current PT, it looks like it has been fixed in the latest master\n",
"eval_trainer = pl.Trainer(**eval_trainer_cfg)\n",
"\n",
"eval_trainer.test(model=eval_model, verbose=False) # test_dataloaders=eval_dataloader,\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Tit5kG4Z5SXu"
},
"source": [
"# Inference\n",
"\n",
"You may create a model from a saved checkpoint and use the model.infer() method to perform inference on a list of queries. There is no need of any trainer for inference."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BKe5Jn4u9xng"
},
"outputs": [],
"source": [
"# extract the path of the best checkpoint from the training, you may update it to any other checkpoint file\n",
"checkpoint_path = trainer.checkpoint_callback.best_model_path\n",
"# Create an evaluation model and load the checkpoint\n",
"infer_model = nemo_nlp.models.TextClassificationModel.load_from_checkpoint(checkpoint_path=checkpoint_path)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y8SFxPJd-hkH"
},
"source": [
"To see how the model performs, let’s get model's predictions for a few examples:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DQhsamclRtxJ"
},
"outputs": [],
"source": [
"# move the model to the desired device for inference\n",
"# we move the model to \"cuda\" if available otherwise \"cpu\" would be used\n",
"if torch.cuda.is_available():\n",
" infer_model.to(\"cuda\")\n",
"else:\n",
" infer_model.to(\"cpu\")\n",
"\n",
"# define the list of queries for inference\n",
"queries = ['by the end of no such thing the audience , like beatrice , has a watchful affection for the monster .', \n",
" 'director rob marshall went out gunning to make a great one .', \n",
" 'uneasy mishmash of styles and genres .']\n",
" \n",
"# max_seq_length=512 is the maximum length BERT supports. \n",
"results = infer_model.classifytext(queries=queries, batch_size=3, max_seq_length=512)\n",
"\n",
"print('The prediction results of some sample queries with the trained model:')\n",
"for query, result in zip(queries, results):\n",
" print(f'Query : {query}')\n",
" print(f'Predicted label: {result}')\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ref1qSonGNhP"
},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally (eg. cloned from the Github), you can also train the model with `examples/nlp/text_classification/text_classifciation_with_bert.py`. This script contains an example on how to train, evaluate and perform inference with the TextClassificationModel.\n",
"\n",
"For example the following would train a model for 50 epochs in 2 GPUs on a classification task with 2 classes:\n",
"\n",
"```\n",
"# python text_classification_with_bert.py \n",
" model.dataset.num_classes=2\n",
" model.train_ds=PATH_TO_TRAIN_FILE \n",
" model.validation_ds=PATH_TO_VAL_FILE \n",
" trainer.max_epochs=50\n",
" trainer.gpus=2\n",
"```\n",
"\n",
"This script would also reload the best checkpoint after the training is done and does evaluation on the dev set. Then perform inference on some sample queries. \n",
"\n",
"\n",
"By default, this script uses `examples/nlp/text_classification/conf/text_classifciation_config.py` config file, and you may update all the params in the config file from the command line. You may also use another config file like this:\n",
"\n",
"```\n",
"# python text_classification_with_bert.py --config-name==PATH_TO_CONFIG_FILE\n",
" model.dataset.num_classes=2\n",
" model.train_ds=PATH_TO_TRAIN_FILE \n",
" model.validation_ds=PATH_TO_VAL_FILE \n",
" trainer.max_epochs=50\n",
" trainer.gpus=2\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Deployment\n",
"\n",
"You can also deploy a model to an inference engine (like TensorRT or ONNXRuntime) using ONNX exporter.\n",
"If you don't have one, let's install it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!mkdir -p ort\n",
"%cd ort\n",
"!git clean -xfd\n",
"!git clone --depth 1 --branch v1.5.2 https://github.com/microsoft/onnxruntime.git .\n",
"!./build.sh --skip_tests --config Release --build_shared_lib --parallel --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu --build_wheel\n",
"!pip uninstall -y onnxruntime\n",
"!pip uninstall -y onnxruntime-gpu\n",
"!pip install --upgrade --force-reinstall ./build/Linux/Release/dist/onnxruntime*.whl\n",
"%cd .."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then export"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.export(config.export_to)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And run some queries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import torch\n",
"from nemo.utils import logging\n",
"from nemo.collections.nlp.parts.utils_funcs import tensor2list\n",
"from nemo.collections.nlp.models.text_classification import TextClassificationModel\n",
"from nemo.collections.nlp.data.text_classification import TextClassificationDataset\n",
"\n",
"import onnxruntime\n",
"\n",
"def to_numpy(tensor):\n",
" return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()\n",
"\n",
"def postprocessing(results, labels):\n",
" return [labels[str(r)] for r in results]\n",
"\n",
"def create_infer_dataloader(model, queries):\n",
" batch_size = len(queries)\n",
" dataset = TextClassificationDataset(tokenizer=model.tokenizer, queries=queries, max_seq_length=512)\n",
" return torch.utils.data.DataLoader(\n",
" dataset=dataset,\n",
" batch_size=batch_size,\n",
" shuffle=False,\n",
" num_workers=2,\n",
" pin_memory=True,\n",
" drop_last=False,\n",
" collate_fn=dataset.collate_fn,\n",
" )\n",
"\n",
"\n",
"queries = [\"by the end of no such thing the audience , like beatrice , has a watchful affection for the monster .\",\n",
"\"director rob marshall went out gunning to make a great one .\",\n",
"\"uneasy mishmash of styles and genres .\",\n",
"\"I love exotic science fiction / fantasy movies but this one was very unpleasant to watch . Suggestions and images of child abuse , mutilated bodies (live or dead) , other gruesome scenes , plot holes , boring acting made this a regretable experience , The basic idea of entering another person's mind is not even new to the movies or TV (An Outer Limits episode was better at exploring this idea) . i gave it 4 / 10 since some special effects were nice .\"]\n",
"\n",
"model.eval()\n",
"infer_datalayer = create_infer_dataloader(model, queries)\n",
"\n",
"ort_session = onnxruntime.InferenceSession(config.export_to)\n",
"\n",
"for batch in infer_datalayer:\n",
" input_ids, input_type_ids, input_mask, subtokens_mask = batch\n",
" ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(input_ids),\n",
" ort_session.get_inputs()[1].name: to_numpy(input_mask),\n",
" ort_session.get_inputs()[2].name: to_numpy(input_type_ids),}\n",
" ologits = ort_session.run(None, ort_inputs)\n",
" alogits = np.asarray(ologits)\n",
" logits = torch.from_numpy(alogits[0])\n",
" preds = tensor2list(torch.argmax(logits, dim = -1))\n",
" processed_results = postprocessing(preds, {\"0\": \"negative\", \"1\": \"positive\"})\n",
"\n",
" logging.info('The prediction results of some sample queries with the trained model:')\n",
" for query, result in zip(queries, processed_results):\n",
" logging.info(f'Query : {query}')\n",
" logging.info(f'Predicted label: {result}')\n",
" break"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "Text_Classification_Sentiment_Analysis.ipynb",
"private_outputs": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}
tts/NeMo/tutorials/nlp/Token_Classification-BioMegatron.ipynb
+648
View File
@@ -0,0 +1,648 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "medieval-neighbor",
"metadata": {},
"outputs": [],
"source": [
"BRANCH = 'v1.0.2'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "developmental-gibraltar",
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "second-lucas",
"metadata": {},
"outputs": [],
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "challenging-pioneer",
"metadata": {},
"outputs": [],
"source": [
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "markdown",
"id": "employed-ethiopia",
"metadata": {},
"source": [
"In this tutorial, we are going to describe how to finetune BioMegatron - a [BERT](https://arxiv.org/abs/1810.04805)-like [Megatron-LM](https://arxiv.org/pdf/1909.08053.pdf) model pre-trained on large biomedical text corpus ([PubMed](https://pubmed.ncbi.nlm.nih.gov/) abstracts and full-text commercial use collection) - on the [NCBI Disease Dataset](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3951655/) for Named Entity Recognition.\n",
"\n",
"The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large.\n",
"There are some alternatives of BioMegatron, most notably [BioBERT](https://arxiv.org/abs/1901.08746). Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.\n",
"\n",
"A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb).\n",
"\n",
"# Task Description\n",
"**Named entity recognition (NER)**, also referred to as entity chunking, identification or extraction, is the task of detecting and classifying key information (entities) in text.\n",
"\n",
"For instance, **given sentences from medical abstracts, what diseases are mentioned?**<br>\n",
"In this case, our data input is sentences from the abstracts, and our labels are the precise locations of the named disease entities. Take a look at the information provided for the dataset.\n",
"\n",
"For more details and general examples on Named Entity Recognition, please refer to the [Token Classification and Named Entity Recognition tutorial notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb).\n",
"\n",
"# Dataset\n",
"\n",
"The [NCBI-disease corpus](https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/) is a set of 793 PubMed abstracts, annotated by 14 annotators. The annotations take the form of HTML-style tags inserted into the abstract text using the clearly defined rules. The annotations identify named diseases, and can be used to fine-tune a language model to identify disease mentions in future abstracts, *whether those diseases were part of the original training set or not*.\n",
"\n",
"Here's an example of what an annotated abstract from the corpus looks like:\n",
"\n",
"```html\n",
"10021369\tIdentification of APC2, a homologue of the <category=\"Modifier\">adenomatous polyposis coli tumour</category> suppressor .\tThe <category=\"Modifier\">adenomatous polyposis coli ( APC ) tumour</category>-suppressor protein controls the Wnt signalling pathway by forming a complex with glycogen synthase kinase 3beta ( GSK-3beta ) , axin / conductin and betacatenin . Complex formation induces the rapid degradation of betacatenin . In <category=\"Modifier\">colon carcinoma</category> cells , loss of APC leads to the accumulation of betacatenin in the nucleus , where it binds to and activates the Tcf-4 transcription factor ( reviewed in [ 1 ] [ 2 ] ) . Here , we report the identification and genomic structure of APC homologues . Mammalian APC2 , which closely resembles APC in overall domain structure , was functionally analyzed and shown to contain two SAMP domains , both of which are required for binding to conductin . Like APC , APC2 regulates the formation of active betacatenin-Tcf complexes , as demonstrated using transient transcriptional activation assays in APC - / - <category=\"Modifier\">colon carcinoma</category> cells . Human APC2 maps to chromosome 19p13 . 3 . APC and APC2 may therefore have comparable functions in development and <category=\"SpecificDisease\">cancer</category> .\n",
"```\n",
"\n",
"In this example, we see the following tags within the abstract:\n",
"```html\n",
"<category=\"Modifier\">adenomatous polyposis coli tumour</category>\n",
"<category=\"Modifier\">adenomatous polyposis coli ( APC ) tumour</category>\n",
"<category=\"Modifier\">colon carcinoma</category>\n",
"<category=\"Modifier\">colon carcinoma</category>\n",
"<category=\"SpecificDisease\">cancer</category>\n",
"```\n",
"\n",
"For our purposes, we will consider any identified category (such as \"Modifier\", \"Specific Disease\", and a few others) to generally be a \"disease\".\n",
"\n",
"Let's download the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "federal-beads",
"metadata": {},
"outputs": [],
"source": [
"DATA_DIR = \"DATA_DIR\"\n",
"os.makedirs(DATA_DIR, exist_ok=True)\n",
"os.makedirs(os.path.join(DATA_DIR, 'NER'), exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "relevant-juvenile",
"metadata": {},
"outputs": [],
"source": [
"print('Downloading NCBI data...')\n",
"wget.download('https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/NCBI_corpus.zip', DATA_DIR)\n",
"! unzip -o {DATA_DIR}/NCBI_corpus.zip -d {DATA_DIR}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "radical-castle",
"metadata": {},
"outputs": [],
"source": [
"# If you want to see more examples, you can explore the text of the corpus using the file browser to the left, or open files directly, for example typing a command like the following in a code-cell:\n",
"\n",
"! head -1 $DATA_DIR/NCBI_corpus_testing.txt"
]
},
{
"cell_type": "markdown",
"id": "specified-maine",
"metadata": {},
"source": [
"We have two datasets derived from this corpus: a text classification dataset and a named entity recognition (NER) dataset. The text classification dataset labels the abstracts among three broad disease groupings. We'll use this simple split to demonstrate the NLP text classification task. The NER dataset labels individual words as diseases. This dataset will be used for the NLP NER task. "
]
},
{
"cell_type": "markdown",
"id": "affected-numbers",
"metadata": {},
"source": [
"## Pre-process dataset\n",
"A pre-processed NCBI-disease dataset for NER can be found [here](https://github.com/spyysalo/ncbi-disease/tree/master/conll) or [here](https://github.com/dmis-lab/biobert#datasets).<br>\n",
"We download the files under {DATA_DIR/NER} directory."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "present-interference",
"metadata": {},
"outputs": [],
"source": [
"NER_DATA_DIR = f'{DATA_DIR}/NER'\n",
"wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/train.tsv', NER_DATA_DIR)\n",
"wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/devel.tsv', NER_DATA_DIR)\n",
"wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/test.tsv', NER_DATA_DIR)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "identical-figure",
"metadata": {},
"outputs": [],
"source": [
"!ls -lh $NER_DATA_DIR"
]
},
{
"cell_type": "markdown",
"id": "powerful-donor",
"metadata": {},
"source": [
"Convert these to a format that is compatible with [NeMo Token Classification module](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_train.py), using the [conversion script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/import_from_iob_format.py)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "utility-wesley",
"metadata": {},
"outputs": [],
"source": [
"! mv $NER_DATA_DIR/devel.tsv $NER_DATA_DIR/dev.tsv"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "suited-jenny",
"metadata": {},
"outputs": [],
"source": [
"wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/data/import_from_iob_format.py')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "sensitive-victoria",
"metadata": {},
"outputs": [],
"source": [
"! python import_from_iob_format.py --data_file=$NER_DATA_DIR/train.tsv\n",
"! python import_from_iob_format.py --data_file=$NER_DATA_DIR/dev.tsv\n",
"! python import_from_iob_format.py --data_file=$NER_DATA_DIR/test.tsv"
]
},
{
"cell_type": "markdown",
"id": "graphic-debate",
"metadata": {},
"source": [
"The NER task requires two files: the text sentences, and the labels. Run the next two cells to see a sample of the two files."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "sound-surgeon",
"metadata": {},
"outputs": [],
"source": [
"!head $NER_DATA_DIR/text_train.txt"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "spectacular-strain",
"metadata": {},
"outputs": [],
"source": [
"!head $NER_DATA_DIR/labels_train.txt"
]
},
{
"cell_type": "markdown",
"id": "moved-principal",
"metadata": {},
"source": [
"### IOB Tagging\n",
"We can see that the abstract has been broken into sentences. Each sentence is then further parsed into words with labels that correspond to the original HTML-style tags in the corpus. \n",
"\n",
"The sentences and labels in the NER dataset map to each other with _inside, outside, beginning (IOB)_ tagging. Anything separated by white space is a word, including punctuation. For the first sentence we have the following mapping:\n",
"\n",
"```text\n",
"Identification of APC2 , a homologue of the adenomatous polyposis coli tumour suppressor .\n",
"O O O O O O O O B I I I O O \n",
"```\n",
"\n",
"Recall the original corpus tags:\n",
"```html\n",
"Identification of APC2, a homologue of the <category=\"Modifier\">adenomatous polyposis coli tumour</category> suppressor .\n",
"```\n",
"The beginning word of the tagged text, \"adenomatous\", is now IOB-tagged with a <span style=\"font-family:verdana;font-size:110%;\">B</span> (beginning) tag, the other parts of the disease, \"polyposis coli tumour\" tagged with <span style=\"font-family:verdana;font-size:110%;\">I</span> (inside) tags, and everything else tagged as <span style=\"font-family:verdana;font-size:110%;\">O</span> (outside).\n",
"\n",
"# Model configuration\n",
"\n",
"Our Named Entity Recognition model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model followed by a Token Classification layer.\n",
"\n",
"The model is defined in a config file which declares multiple important sections. They are:\n",
"- **model**: All arguments that are related to the Model - language model, token classifier, optimizer and schedulers, datasets and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "speaking-grant",
"metadata": {},
"outputs": [],
"source": [
"WORK_DIR = \"WORK_DIR\"\n",
"os.makedirs(WORK_DIR, exist_ok=True)\n",
"MODEL_CONFIG = \"token_classification_config.yaml\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "demanding-ballet",
"metadata": {},
"outputs": [],
"source": [
"# download the model's configuration file \n",
"config_dir = WORK_DIR + '/configs/'\n",
"os.makedirs(config_dir, exist_ok=True)\n",
"if not os.path.exists(config_dir + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG, config_dir)\n",
"else:\n",
" print ('config file is already exists')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "criminal-outdoors",
"metadata": {},
"outputs": [],
"source": [
"# this line will print the entire config of the model\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)\n",
"# Note: these are small batch-sizes - increase as appropriate to available GPU capacity\n",
"config.model.train_ds.batch_size=8\n",
"config.model.validation_ds.batch_size=8"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "divine-belly",
"metadata": {},
"outputs": [],
"source": [
"print(OmegaConf.to_yaml(config))"
]
},
{
"cell_type": "markdown",
"id": "dedicated-effort",
"metadata": {},
"source": [
"# Model Training\n",
"## Setting up Data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. \n",
"So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.\n",
"\n",
"Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
"\n",
"Let's now add the data directory path to the config."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "informed-purse",
"metadata": {},
"outputs": [],
"source": [
"# in this tutorial train and dev datasets are located in the same folder, so it is enought to add the path of the data directory to the config\n",
"config.model.dataset.data_dir = os.path.join(DATA_DIR, 'NER')\n",
"\n",
"# if you want to decrease the size of your datasets, uncomment the lines below:\n",
"# NUM_SAMPLES = 1000\n",
"# config.model.train_ds.num_samples = NUM_SAMPLES\n",
"# config.model.validation_ds.num_samples = NUM_SAMPLES"
]
},
{
"cell_type": "markdown",
"id": "changed-mauritius",
"metadata": {},
"source": [
"## Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
"\n",
"Let's first instantiate a Trainer object"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "computational-battlefield",
"metadata": {},
"outputs": [],
"source": [
"print(\"Trainer config - \\n\")\n",
"print(OmegaConf.to_yaml(config.trainer))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "unique-genre",
"metadata": {},
"outputs": [],
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"\n",
"# for PyTorch Native AMP set precision=16\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# remove distributed training flags\n",
"config.trainer.accelerator = 'ddp'\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
]
},
{
"cell_type": "markdown",
"id": "overall-literature",
"metadata": {},
"source": [
"## Setting up a NeMo Experiment\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "mathematical-portable",
"metadata": {},
"outputs": [],
"source": [
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"exp_dir = str(exp_dir)\n",
"exp_dir"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "built-scratch",
"metadata": {},
"outputs": [],
"source": [
"# complete list of supported BERT-like models\n",
"print(nemo_nlp.modules.get_pretrained_lm_models_list())\n",
"\n",
"# specify BERT-like model, you want to use\n",
"PRETRAINED_BERT_MODEL = \"biomegatron-bert-345m-cased\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "compact-horse",
"metadata": {},
"outputs": [],
"source": [
"# add the specified above model parameters to the config\n",
"config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL"
]
},
{
"cell_type": "markdown",
"id": "seeing-geometry",
"metadata": {},
"source": [
"Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
"Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "indoor-france",
"metadata": {},
"outputs": [],
"source": [
"model_ner = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)"
]
},
{
"cell_type": "markdown",
"id": "genuine-pipeline",
"metadata": {},
"source": [
"## Monitoring training progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress.\n",
"If you're not using Colab, refer to [https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks](https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks) if you're facing issues with running the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "changed-expense",
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "applied-quality",
"metadata": {},
"outputs": [],
"source": [
"# start model training\n",
"trainer.fit(model_ner)"
]
},
{
"cell_type": "markdown",
"id": "cooperative-michael",
"metadata": {},
"source": [
"# Inference\n",
"\n",
"To see how the model performs, we can run generate prediction similar to the way we did it earlier"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "classical-scientist",
"metadata": {},
"outputs": [],
"source": [
"# let's first create a subset of our dev data\n",
"! head -n 100 $NER_DATA_DIR/text_dev.txt > $NER_DATA_DIR/sample_text_dev.txt\n",
"! head -n 100 $NER_DATA_DIR/labels_dev.txt > $NER_DATA_DIR/sample_labels_dev.txt"
]
},
{
"cell_type": "markdown",
"id": "adult-ranking",
"metadata": {},
"source": [
"Now, let's generate predictions for the provided text file.\n",
"If labels file is also specified, the model will evaluate the predictions and plot confusion matrix. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "twenty-abortion",
"metadata": {},
"outputs": [],
"source": [
"model_ner.evaluate_from_file(\n",
" text_file=os.path.join(NER_DATA_DIR, 'sample_text_dev.txt'),\n",
" labels_file=os.path.join(NER_DATA_DIR, 'sample_labels_dev.txt'),\n",
" output_dir=exp_dir,\n",
" add_confusion_matrix=False,\n",
" normalize_confusion_matrix=True,\n",
" batch_size=1\n",
")\n",
"# Please check matplotlib version if encountering any error plotting confusion matrix:\n",
"# https://stackoverflow.com/questions/63212347/importerror-cannot-import-name-png-from-matplotlib"
]
},
{
"cell_type": "markdown",
"id": "connected-typing",
"metadata": {},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally, you can also train the model with `nlp/token_classification/token_classification_train.py.`\n",
"\n",
"To run training script, use:\n",
"\n",
"`python token_classification_train.py model.dataset.data_dir=PATH_TO_DATA_DIR PRETRAINED_BERT_MODEL=biomegatron-bert-345m-cased`"
]
},
{
"cell_type": "markdown",
"id": "legitimate-electric",
"metadata": {},
"source": [
"The training could take several minutes and the result should look something like\n",
"```\n",
"[NeMo I 2020-05-22 17:13:48 token_classification_callback:82] Accuracy: 0.9882348032875798\n",
"[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 weighted: 98.82\n",
"[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 macro: 93.74\n",
"[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 micro: 98.82\n",
"[NeMo I 2020-05-22 17:13:49 token_classification_callback:89] precision recall f1-score support\n",
" \n",
" O (label id: 0) 0.9938 0.9957 0.9947 22092\n",
" B (label id: 1) 0.8843 0.9034 0.8938 787\n",
" I (label id: 2) 0.9505 0.8982 0.9236 1090\n",
" \n",
" accuracy 0.9882 23969\n",
" macro avg 0.9429 0.9324 0.9374 23969\n",
" weighted avg 0.9882 0.9882 0.9882 23969\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "unique-fiber",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
tts/NeMo/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb
+849
View File
@@ -0,0 +1,849 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Token_Classification_Named_Entity_Recognition.ipynb",
"provenance": [],
"private_outputs": true,
"collapsed_sections": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU",
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"source": [],
"metadata": {
"collapsed": false
}
}
}
},
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"BRANCH = 'v1.0.2'"
]
},
{
"cell_type": "code",
"metadata": {
"id": "o_0K1lsW1dj9",
"colab_type": "code",
"colab": {}
},
"source": [
"\"\"\"\n",
"You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"# If you're using Google Colab and not running locally, run this cell\n",
"\n",
"# install NeMo\n",
"BRANCH = 'v1.0.2'\n!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"id": "pC0slAc0h9zN",
"colab_type": "code",
"colab": {}
},
"source": [
"# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
"# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
"\n",
"! pip install ipywidgets\n",
"! jupyter nbextension enable --py widgetsnbextension\n",
"\n",
"# Please restart the kernel after running this cell"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dzqD2WDFOIN-",
"colab_type": "code",
"colab": {}
},
"source": [
"from nemo.collections import nlp as nemo_nlp\n",
"from nemo.utils.exp_manager import exp_manager\n",
"\n",
"import os\n",
"import wget \n",
"import torch\n",
"import pytorch_lightning as pl\n",
"from omegaconf import OmegaConf"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daYw_Xll2ZR9",
"colab_type": "text"
},
"source": [
"# Task Description\n",
"**Named entity recognition (NER)**, also referred to as entity chunking, identification or extraction, is the task of detecting and classifying key information (entities) in text.\n",
"For example, in a sentence: `Mary lives in Santa Clara and works at NVIDIA`, we should detect that `Mary` is a person, `Santa Clara` is a location and `NVIDIA` is a company."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnuziSwJ1yEB",
"colab_type": "text"
},
"source": [
"# Dataset\n",
"\n",
"In this tutorial we going to use [GMB(Groningen Meaning Bank)](http://www.let.rug.nl/bjerva/gmb/about.php) corpus for entity recognition. \n",
"\n",
"GMB is a fairly large corpus with a lot of annotations. Note, that GMB is not completely human annotated and it’s not considered 100% correct. \n",
"The data is labeled using the [IOB format](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)) (short for inside, outside, beginning). \n",
"\n",
"The following classes appear in the dataset:\n",
"* LOC = Geographical Entity\n",
"* ORG = Organization\n",
"* PER = Person\n",
"* GPE = Geopolitical Entity\n",
"* TIME = Time indicator\n",
"* ART = Artifact\n",
"* EVE = Event\n",
"* NAT = Natural Phenomenon\n",
"\n",
"For this tutorial, classes ART, EVE, and NAT were combined into a MISC class due to small number of examples for these classes.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qzcZ3nb_-SVT",
"colab_type": "text"
},
"source": [
"# NeMo Token Classification Data Format\n",
"\n",
"[TokenClassification Model](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/token_classification/token_classification_model.py) in NeMo supports NER and other token level classification tasks, as long as the data follows the format specified below. \n",
"\n",
"Token Classification Model requires the data to be split into 2 files: \n",
"* text.txt and \n",
"* labels.txt. \n",
"\n",
"Each line of the **text.txt** file contains text sequences, where words are separated with spaces, i.e.: \n",
"[WORD] [SPACE] [WORD] [SPACE] [WORD].\n",
"\n",
"The **labels.txt** file contains corresponding labels for each word in text.txt, the labels are separated with spaces, i.e.:\n",
"[LABEL] [SPACE] [LABEL] [SPACE] [LABEL].\n",
"\n",
"Example of a text.txt file:\n",
"```\n",
"Jennifer is from New York City .\n",
"She likes ...\n",
"...\n",
"```\n",
"Corresponding labels.txt file:\n",
"```\n",
"B-PER O O B-LOC I-LOC I-LOC O\n",
"O O ...\n",
"...\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VsEmwIPO4L4V",
"colab_type": "text"
},
"source": [
"To convert an IOB format data to the format required for training, run [examples/nlp/token_classification/data/import_from_iob_format.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/import_from_iob_format.py) on your train and dev files, as follows:\n",
"\n",
"\n",
"\n",
"\n",
"```\n",
"python examples/nlp/token_classification/data/import_from_iob_format.py --data_file PATH_TO_IOB_FORMAT_DATAFILE\n",
"```\n",
"\n",
"For this tutorial, we are going to use the preprocessed GMB dataset.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SL58EWkd2ZVb",
"colab_type": "text"
},
"source": [
"## Download and preprocess the data¶"
]
},
{
"cell_type": "code",
"metadata": {
"id": "n8HZrDmr12_-",
"colab_type": "code",
"colab": {}
},
"source": [
"DATA_DIR = \"DATA_DIR\"\n",
"WORK_DIR = \"WORK_DIR\"\n",
"MODEL_CONFIG = \"token_classification_config.yaml\""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "jrx2ZXHrCHb_",
"colab_type": "code",
"colab": {}
},
"source": [
"# download preprocessed data\n",
"os.makedirs(WORK_DIR, exist_ok=True)\n",
"os.makedirs(DATA_DIR, exist_ok=True)\n",
"print('Downloading GMB data...')\n",
"wget.download('https://dldata-public.s3.us-east-2.amazonaws.com/gmb_v_2.2.0_clean.zip', DATA_DIR)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "NhUzIeF0Yg0l",
"colab_type": "text"
},
"source": [
"Let's extract files from the .zip file:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Y01BdjPRW-7B",
"colab_type": "code",
"colab": {}
},
"source": [
"! unzip {DATA_DIR}/gmb_v_2.2.0_clean.zip -d {DATA_DIR}\n",
"DATA_DIR = os.path.join(DATA_DIR, 'gmb_v_2.2.0_clean')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "U8Ty5_S7Ye8h",
"colab_type": "text"
},
"source": [
"Now, the data folder should contain 4 files:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L8vsyh3JZH26",
"colab_type": "text"
},
"source": [
"\n",
"\n",
"* labels_dev.txt\n",
"* labels_train.txt\n",
"* text_dev.txt\n",
"* text_train.txt\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "qB0oLE4R9EhJ",
"colab_type": "code",
"colab": {}
},
"source": [
"! ls -l {DATA_DIR}"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "6UDPgadLN6SG",
"colab_type": "code",
"colab": {}
},
"source": [
"# let's take a look at the data \n",
"print('Text:')\n",
"! head -n 5 {DATA_DIR}/text_train.txt\n",
"\n",
"print('\\nLabels:')\n",
"! head -n 5 {DATA_DIR}/labels_train.txt"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "daludzzL2Jba",
"colab_type": "text"
},
"source": [
"# Model Configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Tit5kG4Z5SXu",
"colab_type": "text"
},
"source": [
"# Using an Out-of-the-Box Model\n",
"\n",
"To use a pretrained NER model, run:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "BKe5Jn4u9xng",
"colab_type": "code",
"colab": {}
},
"source": [
"# this line will download pre-trained NER model from NVIDIA's NGC cloud and instantiate it for you\n",
"pretrained_ner_model = nemo_nlp.models.TokenClassificationModel.from_pretrained(model_name=\"ner_en_bert\") "
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "y8SFxPJd-hkH"
},
"source": [
"To see how the model performs, let’s get model's predictions for a few examples:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "DQhsamclRtxJ",
"colab_type": "code",
"colab": {}
},
"source": [
"# define the list of queries for inference\n",
"queries = [\n",
" 'we bought four shirts from the nvidia gear store in santa clara.',\n",
" 'Nvidia is a company.',\n",
" 'The Adventures of Tom Sawyer by Mark Twain is an 1876 novel about a young boy growing '\n",
" + 'up along the Mississippi River.',\n",
"]\n",
"results = pretrained_ner_model.add_predictions(queries)\n",
"\n",
"for query, result in zip(queries, results):\n",
" print()\n",
" print(f'Query : {query}')\n",
" print(f'Result: {result.strip()}\\n')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "_whKCxfTMo6Y",
"colab_type": "text"
},
"source": [
"Now, let's take a closer look at the model's configuration and learn to train the model from scratch and finetune the pretrained model.\n",
"\n",
"# Model configuration\n",
"\n",
"Our Named Entity Recognition model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model followed by a Token Classification layer.\n",
"\n",
"The model is defined in a config file which declares multiple important sections. They are:\n",
"- **model**: All arguments that are related to the Model - language model, token classifier, optimizer and schedulers, datasets and any other related information\n",
"\n",
"- **trainer**: Any argument to be passed to PyTorch Lightning"
]
},
{
"cell_type": "code",
"metadata": {
"id": "T1gA8PsJ13MJ",
"colab_type": "code",
"colab": {}
},
"source": [
"# download the model's configuration file \n",
"config_dir = WORK_DIR + '/configs/'\n",
"os.makedirs(config_dir, exist_ok=True)\n",
"if not os.path.exists(config_dir + MODEL_CONFIG):\n",
" print('Downloading config file...')\n",
" wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG, config_dir)\n",
"else:\n",
" print ('config file is already exists')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "mX3KmWMvSUQw",
"colab_type": "code",
"colab": {}
},
"source": [
"# this line will print the entire config of the model\n",
"config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
"print(config_path)\n",
"config = OmegaConf.load(config_path)\n",
"print(OmegaConf.to_yaml(config))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZCgWzNBkaQLZ",
"colab_type": "text"
},
"source": [
"# Model Training From Scratch\n",
"## Setting up Data within the config\n",
"\n",
"Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
"\n",
"We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. \n",
"So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.\n",
"\n",
"Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
"\n",
"Let's now add the data directory path to the config."
]
},
{
"cell_type": "code",
"metadata": {
"id": "LQHCJN-ZaoLp",
"colab_type": "code",
"colab": {}
},
"source": [
"# in this tutorial train and dev datasets are located in the same folder, so it is enought to add the path of the data directory to the config\n",
"config.model.dataset.data_dir = DATA_DIR\n",
"\n",
"# if you want to use the full dataset, set NUM_SAMPLES to -1\n",
"NUM_SAMPLES = 1000\n",
"config.model.train_ds.num_samples = NUM_SAMPLES\n",
"config.model.validation_ds.num_samples = NUM_SAMPLES"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "nB96-3sTc3yk",
"colab_type": "text"
},
"source": [
"## Building the PyTorch Lightning Trainer\n",
"\n",
"NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
"\n",
"Let's first instantiate a Trainer object"
]
},
{
"cell_type": "code",
"metadata": {
"id": "1tG4FzZ4Ui60",
"colab_type": "code",
"colab": {}
},
"source": [
"print(\"Trainer config - \\n\")\n",
"print(OmegaConf.to_yaml(config.trainer))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "knF6QeQQdMrH",
"colab_type": "code",
"colab": {}
},
"source": [
"# lets modify some trainer configs\n",
"# checks if we have GPU available and uses it\n",
"cuda = 1 if torch.cuda.is_available() else 0\n",
"config.trainer.gpus = cuda\n",
"\n",
"config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
"\n",
"# for mixed precision training, uncomment the line below (precision should be set to 16 and amp_level to O1):\n",
"# config.trainer.amp_level = O1\n",
"\n",
"# remove distributed training flags\n",
"config.trainer.accelerator = None\n",
"\n",
"# setup max number of steps to reduce training time for demonstration purposes of this tutorial\n",
"config.trainer.max_steps = 32\n",
"\n",
"trainer = pl.Trainer(**config.trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8IlEMdVxdr6p",
"colab_type": "text"
},
"source": [
"## Setting up a NeMo Experiment¶\n",
"\n",
"NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "8uztqGAmdrYt",
"colab_type": "code",
"colab": {}
},
"source": [
"exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
"\n",
"# the exp_dir provides a path to the current experiment for easy access\n",
"exp_dir = str(exp_dir)\n",
"exp_dir"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8tjLhUvL_o7_",
"colab_type": "text"
},
"source": [
"Before initializing the model, we might want to modify some of the model configs. For example, we might want to modify the pretrained BERT model:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Xeuc2i7Y_nP5",
"colab_type": "code",
"colab": {}
},
"source": [
"# get the list of supported BERT-like models, for the complete list of HugginFace models, see https://huggingface.co/models\n",
"print(nemo_nlp.modules.get_pretrained_lm_models_list(include_external=True))\n",
"\n",
"# specify BERT-like model, you want to use\n",
"PRETRAINED_BERT_MODEL = \"bert-base-uncased\""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "RK2xglXyAUOO",
"colab_type": "code",
"colab": {}
},
"source": [
"# add the specified above model parameters to the config\n",
"config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fzNZNAVRjDD-",
"colab_type": "text"
},
"source": [
"Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
"Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
]
},
{
"cell_type": "code",
"metadata": {
"id": "NgsGLydWo-6-",
"colab_type": "code",
"colab": {}
},
"source": [
"model_from_scratch = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "kQ592Tx4pzyB",
"colab_type": "text"
},
"source": [
"## Monitoring training progress\n",
"Optionally, you can create a Tensorboard visualization to monitor training progress."
]
},
{
"cell_type": "code",
"metadata": {
"id": "mTJr16_pp0aS",
"colab_type": "code",
"colab": {}
},
"source": [
"try:\n",
" from google import colab\n",
" COLAB_ENV = True\n",
"except (ImportError, ModuleNotFoundError):\n",
" COLAB_ENV = False\n",
"\n",
"# Load the TensorBoard notebook extension\n",
"if COLAB_ENV:\n",
" %load_ext tensorboard\n",
" %tensorboard --logdir {exp_dir}\n",
"else:\n",
" print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "hUvnSpyjp0Dh",
"colab_type": "code",
"colab": {}
},
"source": [
"# start model training\n",
"trainer.fit(model_from_scratch)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "JxBiIKMlH8yv",
"colab_type": "text"
},
"source": [
"After training for 5 epochs, with the default config and NUM_SAMPLES = -1 (i.e.all data is used), your model performance should look similar to this: \n",
"```\n",
" label precision recall f1 support \n",
" O (label_id: 0) 99.14 99.19 99.17 131141\n",
" B-GPE (label_id: 1) 95.86 94.03 94.93 2362\n",
" B-LOC (label_id: 2) 83.99 90.31 87.04 5346\n",
" B-MISC (label_id: 3) 39.82 34.62 37.04 130\n",
" B-ORG (label_id: 4) 78.33 67.82 72.70 2980\n",
" B-PER (label_id: 5) 84.36 84.32 84.34 2577\n",
" B-TIME (label_id: 6) 91.94 91.23 91.58 2975\n",
" I-GPE (label_id: 7) 88.89 34.78 50.00 23\n",
" I-LOC (label_id: 8) 77.18 79.13 78.14 1030\n",
" I-MISC (label_id: 9) 28.57 24.00 26.09 75\n",
" I-ORG (label_id: 10) 78.67 75.67 77.14 2384\n",
" I-PER (label_id: 11) 86.69 90.17 88.40 2687\n",
" I-TIME (label_id: 12) 83.21 83.48 83.34 938\n",
" -------------------\n",
" micro avg 96.95 96.95 96.95 154648\n",
" macro avg 78.20 72.98 74.61 154648\n",
" weighted avg 96.92 96.95 96.92 154648\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPdzJVAgSFaJ",
"colab_type": "text"
},
"source": [
"# Inference\n",
"\n",
"To see how the model performs, we can run generate prediction similar to the way we did it earlier"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QaW0A1OOwefR",
"colab_type": "text"
},
"source": [
"## Generate Predictions\n",
"\n",
"To see how the model performs, we can generate prediction the same way we did it earlier or we can use our model to generate predictions for a dataset from a file, for example, to perform final evaluation or to do error analysis.\n",
"Below, we are using a subset of dev set, but it could be any text file as long as it follows the data format described above.\n",
"Labels_file is optional here, and if provided will be used to get metrics."
]
},
{
"cell_type": "code",
"metadata": {
"id": "92PB0iTqNnW-",
"colab_type": "code",
"colab": {}
},
"source": [
"# let's first create a subset of our dev data\n",
"! head -n 100 {DATA_DIR}/text_dev.txt > {DATA_DIR}/sample_text_dev.txt\n",
"! head -n 100 {DATA_DIR}/labels_dev.txt > {DATA_DIR}/sample_labels_dev.txt"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "vXnx2tKoOohy",
"colab_type": "text"
},
"source": [
"Now, let's generate predictions for the provided text file.\n",
"If labels file is also specified, the model will evaluate the predictions and plot confusion matrix. "
]
},
{
"cell_type": "code",
"metadata": {
"id": "sglcZV1bwsv0",
"colab_type": "code",
"colab": {}
},
"source": [
"model_from_scratch.evaluate_from_file(\n",
" text_file=os.path.join(DATA_DIR, 'sample_text_dev.txt'),\n",
" labels_file=os.path.join(DATA_DIR, 'sample_labels_dev.txt'),\n",
" output_dir=exp_dir,\n",
")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ref1qSonGNhP",
"colab_type": "text"
},
"source": [
"## Training Script\n",
"\n",
"If you have NeMo installed locally, you can also train the model with [nlp/token_classification/token_classification_train.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_train.py).\n",
"\n",
"To run training script, use:\n",
"\n",
"`python token_classification_train.py model.dataset.data_dir=PATH_TO_DATA_DIR`\n",
"\n",
"# Finetuning model with your data\n",
"\n",
"When we were training from scratch, the datasets were prepared for training during the model initialization. When we are using a pretrained NER model, before training, we need to setup training and evaluation data.\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "yu9fZc2vPQfw",
"colab_type": "code",
"colab": {}
},
"source": [
"# let's reload our pretrained NER model\n",
"pretrained_ner_model = nemo_nlp.models.TokenClassificationModel.from_pretrained('ner_en_bert')\n",
"\n",
"# then we need to setup the data dir to get class weights statistics\n",
"pretrained_ner_model.update_data_dir(DATA_DIR)\n",
"\n",
"# setup train and validation Pytorch DataLoaders\n",
"pretrained_ner_model.setup_training_data()\n",
"pretrained_ner_model.setup_validation_data()\n",
"\n",
"# then we're setting up loss, use class_balancing='weighted_loss' if you want to add class weights to the CrossEntropyLoss\n",
"pretrained_ner_model.setup_loss()\n",
"\n",
"# and now we can create a PyTorch Lightning trainer and call `fit` again\n",
"# for this tutorial we are setting fast_dev_run to True, and the trainer will run 1 training batch and 1 validation batch\n",
"# for actual model training, disable the flag\n",
"fast_dev_run = True\n",
"trainer = pl.Trainer(gpus=1, fast_dev_run=fast_dev_run)\n",
"trainer.fit(pretrained_ner_model)"
],
"execution_count": null,
"outputs": []
},
{
"source": [
"# Labeling your own data\n",
"\n",
"If you have raw data, NeMo recommends using the Datasaur labeling platform to apply labels to data. Datasaur was designed specifically for labeling text data and supports basic NLP labeling tasks such as Named Entity Recognition and text classification through advanced NLP tasks such as dependency parsing and coreference resolution. You can sign up for Datasaur for free at https://datasaur.ai/sign-up/. Once you upload a file, you can choose from multiple NLP project types and use the Datasaur interface to label the data. After labeling, you can export the labeled data using the conll_2003 format, which integrates directly with NeMo. A video walkthrough can be found here: https://www.youtube.com/watch?v=I9WVmnnSciE.\n"
],
"cell_type": "markdown",
"metadata": {}
}
]
}