storyteller-ml/tts/NeMo/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o_0K1lsW1dj9"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
    "\n",
    "Instructions for setting up Colab are as follows:\n",
    "1. Open a new Python 3 notebook.\n",
    "2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
    "3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
    "4. Run this cell to set up dependencies.\n",
    "\"\"\"\n",
    "# If you're using Google Colab and not running locally, run this cell\n",
    "\n",
    "# install NeMo\n",
    "BRANCH = 'v1.0.2'\n",
    "!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
    "# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
    "\n",
    "! pip install ipywidgets\n",
    "! jupyter nbextension enable --py widgetsnbextension\n",
    "\n",
    "# Please restart the kernel after running this cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dzqD2WDFOIN-"
   },
   "outputs": [],
   "source": [
    "from nemo.collections import nlp as nemo_nlp\n",
    "from nemo.utils.exp_manager import exp_manager\n",
    "\n",
    "import os\n",
    "import wget \n",
    "import torch\n",
    "import pytorch_lightning as pl\n",
    "from omegaconf import OmegaConf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "daYw_Xll2ZR9"
   },
   "source": [
    "In this tutorial, we are going to describe how to finetune BioMegatron - a [BERT](https://arxiv.org/abs/1810.04805)-like [Megatron-LM](https://arxiv.org/pdf/1909.08053.pdf) model pre-trained on large biomedical text corpus ([PubMed](https://pubmed.ncbi.nlm.nih.gov/) abstracts and full-text commercial use collection) - on [RE: Text mining chemical-protein interactions (CHEMPROT)](https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vi/track-5/).\n",
    "\n",
    "The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large.\n",
    "There are some alternatives of BioMegatron, most notably [BioBERT](https://arxiv.org/abs/1901.08746). Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.\n",
    "\n",
    "A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task Description\n",
    "**Relation Extraction (RE)** can be regarded as a type of sentence classification.\n",
    "\n",
    "The task is to classify the relation of a [GENE] and [CHEMICAL] in a sentence, for example like the following:\n",
    "```html\n",
    "14967461.T1.T22\t<@CHEMICAL$> inhibitors currently under investigation include the small molecules <@GENE$> (Iressa, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux).\t<CPR:4>\n",
    "14967461.T2.T22\t<@CHEMICAL$> inhibitors currently under investigation include the small molecules gefitinib (<@GENE$>, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux).\t<CPR:4>\n",
    "```\n",
    "to one of the following class:\n",
    "\n",
    "| Relation Class      | Relations |\n",
    "| ----------- | ----------- |\n",
    "| CPR:3      |  Upregulator and activator       |\n",
    "| CPR:4   | Downregulator and inhibitor         |\n",
    "| CPR:5 | Agonist |\n",
    "| CPR:6 | Antagonist |\n",
    "| CPR:9 | Substrate and product of |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZnuziSwJ1yEB"
   },
   "source": [
    "# Datasets\n",
    "\n",
    "Details of ChemProt Relation Extraction task and the original data can be found on the [BioCreative VI website](https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vi/track-5/)\n",
    "\n",
    "ChemProt dataset pre-processed for easier consumption can be downloaded from [here](https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip) or [here](https://github.com/ncbi-nlp/BLUE_Benchmark/releases/download/0.1/bert_data.zip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "--wJ2891aIIE"
   },
   "outputs": [],
   "source": [
    "TASK = 'ChemProt'\n",
    "DATA_DIR = os.path.join(os.getcwd(), 'DATA_DIR')\n",
    "RE_DATA_DIR = os.path.join(DATA_DIR, 'RE')\n",
    "WORK_DIR = os.path.join(os.getcwd(), 'WORK_DIR')\n",
    "MODEL_CONFIG = 'text_classification_config.yaml'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "os.makedirs(os.path.join(DATA_DIR, 'RE'), exist_ok=True)\n",
    "os.makedirs(WORK_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the dataset\n",
    "wget.download('https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip?raw=true',\n",
    "              os.path.join(DATA_DIR, 'data_re.zip'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip -o {DATA_DIR}/data_re.zip -d {RE_DATA_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qB0oLE4R9EhJ"
   },
   "outputs": [],
   "source": [
    "! ls -l $RE_DATA_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-process dataset\n",
    "Let's convert the dataset into the format that is compatible for [NeMo text-classification module](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/text_classification/text_classification_with_bert.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/data/import_datasets.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python import_datasets.py --dataset_name=chemprot --source_data_dir={RE_DATA_DIR} --target_data_dir={RE_DATA_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's take a look at the training data \n",
    "! head -n 5 {RE_DATA_DIR}/train.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's check the label mapping\n",
    "! cat {RE_DATA_DIR}/label_mapping.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is not necessary to have the mapping exactly like this - it can be different.\n",
    "We use the same [mapping used by BioBERT](https://github.com/dmis-lab/biobert/blob/master/run_re.py#L438) so that comparison can be more straightforward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_whKCxfTMo6Y"
   },
   "source": [
    "# Model configuration\n",
    "\n",
    "Now, let's take a closer look at the model's configuration and learn to train the model.\n",
    "\n",
    "The model is defined in a config file which declares multiple important sections. They are:\n",
    "- **model**: All arguments that are related to the Model - language model, a classifier, optimizer and schedulers, datasets and any other related information\n",
    "\n",
    "- **trainer**: Any argument to be passed to PyTorch Lightning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "T1gA8PsJ13MJ"
   },
   "outputs": [],
   "source": [
    "# download the model's configuration file \n",
    "config_dir = WORK_DIR + '/configs/'\n",
    "os.makedirs(config_dir, exist_ok=True)\n",
    "if not os.path.exists(config_dir + MODEL_CONFIG):\n",
    "    print('Downloading config file...')\n",
    "    wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/conf/' + MODEL_CONFIG, config_dir)\n",
    "else:\n",
    "    print ('config file is already exists')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mX3KmWMvSUQw"
   },
   "outputs": [],
   "source": [
    "# this line will print the entire config of the model\n",
    "config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
    "print(config_path)\n",
    "config = OmegaConf.load(config_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config.model.train_ds.file_path = os.path.join(RE_DATA_DIR, 'train.tsv')\n",
    "config.model.validation_ds.file_path = os.path.join(RE_DATA_DIR, 'dev.tsv')\n",
    "config.model.task_name = 'chemprot'\n",
    "# Note: these are small batch-sizes - increase as appropriate to available GPU capacity\n",
    "config.model.train_ds.batch_size=8\n",
    "config.model.validation_ds.batch_size=8\n",
    "config.model.dataset.num_classes=6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(OmegaConf.to_yaml(config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZCgWzNBkaQLZ"
   },
   "source": [
    "# Model Training\n",
    "## Setting up Data within the config\n",
    "\n",
    "Among other things, the config file contains dictionaries called **dataset**, **train_ds** and **validation_ds**. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
    "\n",
    "We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. \n",
    "So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.\n",
    "\n",
    "Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
    "\n",
    "Let's now add the data directory path, task name and output directory for saving predictions to the config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LQHCJN-ZaoLp"
   },
   "outputs": [],
   "source": [
    "config.model.task_name = TASK\n",
    "config.model.output_dir = WORK_DIR\n",
    "config.model.dataset.data_dir = RE_DATA_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nB96-3sTc3yk"
   },
   "source": [
    "## Building the PyTorch Lightning Trainer\n",
    "\n",
    "NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
    "\n",
    "Let's first instantiate a Trainer object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1tG4FzZ4Ui60"
   },
   "outputs": [],
   "source": [
    "print(\"Trainer config - \\n\")\n",
    "print(OmegaConf.to_yaml(config.trainer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "knF6QeQQdMrH"
   },
   "outputs": [],
   "source": [
    "# lets modify some trainer configs\n",
    "# checks if we have GPU available and uses it\n",
    "cuda = 1 if torch.cuda.is_available() else 0\n",
    "config.trainer.gpus = cuda\n",
    "\n",
    "# for PyTorch Native AMP set precision=16\n",
    "config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
    "\n",
    "# remove distributed training flags\n",
    "config.trainer.accelerator = 'DDP'\n",
    "\n",
    "trainer = pl.Trainer(**config.trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8IlEMdVxdr6p"
   },
   "source": [
    "## Setting up a NeMo Experiment\n",
    "\n",
    "NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8uztqGAmdrYt"
   },
   "outputs": [],
   "source": [
    "config.exp_manager.exp_dir = WORK_DIR\n",
    "exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
    "\n",
    "# the exp_dir provides a path to the current experiment for easy access\n",
    "exp_dir = str(exp_dir)\n",
    "exp_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8tjLhUvL_o7_"
   },
   "source": [
    "Before initializing the model, we might want to modify some of the model configs. Here we are modifying it to use BioMegatron, [Megatron-LM BERT](https://arxiv.org/abs/1909.08053) pre-trained on [PubMed](https://pubmed.ncbi.nlm.nih.gov/) biomedical text corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Xeuc2i7Y_nP5"
   },
   "outputs": [],
   "source": [
    "# complete list of supported BERT-like models\n",
    "print(nemo_nlp.modules.get_pretrained_lm_models_list())\n",
    "\n",
    "# specify BERT-like model, you want to use, for example, \"megatron-bert-345m-uncased\" or 'bert-base-uncased'\n",
    "PRETRAINED_BERT_MODEL = \"biomegatron-bert-345m-uncased\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RK2xglXyAUOO"
   },
   "outputs": [],
   "source": [
    "# add the specified above model parameters to the config\n",
    "config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fzNZNAVRjDD-"
   },
   "source": [
    "Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
    "Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NgsGLydWo-6-"
   },
   "outputs": [],
   "source": [
    "model = nemo_nlp.models.TextClassificationModel(cfg=config.model, trainer=trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kQ592Tx4pzyB"
   },
   "source": [
    "## Monitoring training progress\n",
    "Optionally, you can create a Tensorboard visualization to monitor training progress.\n",
    "If you're not using Colab, refer to [https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks](https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks) if you're facing issues with running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mTJr16_pp0aS"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    from google import colab\n",
    "    COLAB_ENV = True\n",
    "except (ImportError, ModuleNotFoundError):\n",
    "    COLAB_ENV = False\n",
    "\n",
    "# Load the TensorBoard notebook extension\n",
    "if COLAB_ENV:\n",
    "    %load_ext tensorboard\n",
    "    %tensorboard --logdir {exp_dir}\n",
    "else:\n",
    "    print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hUvnSpyjp0Dh"
   },
   "outputs": [],
   "source": [
    "# start model training\n",
    "trainer.fit(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ref1qSonGNhP"
   },
   "source": [
    "## Training Script\n",
    "\n",
    "If you have NeMo installed locally, you can also train the model with `examples/nlp/text_classification/text_classification_with_bert.py.`\n",
    "\n",
    "To run training script, use:\n",
    "\n",
    "`python text_classification_with_bert.py \\\n",
    " model.dataset.data_dir=PATH_TO_DATA_DIR \\\n",
    " model.task_name=TASK`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training could take several minutes and the results should look something like:\n",
    "\n",
    "```\n",
    "precision    recall  f1-score   support\n",
    "    \n",
    "               0     0.7328    0.8348    0.7805       115\n",
    "               1     0.9402    0.9291    0.9346      7950\n",
    "               2     0.8311    0.9146    0.8708       199\n",
    "               3     0.6400    0.6302    0.6351       457\n",
    "               4     0.8002    0.8317    0.8156      1093\n",
    "               5     0.7228    0.7518    0.7370       548\n",
    "    \n",
    "        accuracy                         0.8949     10362\n",
    "       macro avg     0.7778    0.8153    0.7956     10362\n",
    "    weighted avg     0.8963    0.8949    0.8954     10362\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Relation_Extraction-BioMegatron.ipynb",
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}