artcraft/models/tts/NeMo/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb

In [ ]:

"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
"""
# If you're using Google Colab and not running locally, run this cell

# install NeMo
BRANCH = 'v1.0.2'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]

In [ ]:

# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:
# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'

! pip install ipywidgets
! jupyter nbextension enable --py widgetsnbextension

# Please restart the kernel after running this cell

In [ ]:

from nemo.collections import nlp as nemo_nlp
from nemo.utils.exp_manager import exp_manager

import os
import wget 
import torch
import pytorch_lightning as pl
from omegaconf import OmegaConf

In this tutorial, we are going to describe how to finetune BioMegatron - a BERT-like Megatron-LM model pre-trained on large biomedical text corpus (PubMed abstracts and full-text commercial use collection) - on RE: Text mining chemical-protein interactions (CHEMPROT).

The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large. There are some alternatives of BioMegatron, most notably BioBERT. Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.

A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found here.

Task Description

Relation Extraction (RE) can be regarded as a type of sentence classification.

The task is to classify the relation of a [GENE] and [CHEMICAL] in a sentence, for example like the following:

14967461.T1.T22	<@CHEMICAL$> inhibitors currently under investigation include the small molecules <@GENE$> (Iressa, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux).	<CPR:4>
14967461.T2.T22	<@CHEMICAL$> inhibitors currently under investigation include the small molecules gefitinib (<@GENE$>, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux).	<CPR:4>

to one of the following class:

Relation Class	Relations
CPR:3	Upregulator and activator
CPR:4	Downregulator and inhibitor
CPR:5	Agonist
CPR:6	Antagonist
CPR:9	Substrate and product of

Datasets

Details of ChemProt Relation Extraction task and the original data can be found on the BioCreative VI website

ChemProt dataset pre-processed for easier consumption can be downloaded from here or here

In [ ]:

TASK = 'ChemProt'
DATA_DIR = os.path.join(os.getcwd(), 'DATA_DIR')
RE_DATA_DIR = os.path.join(DATA_DIR, 'RE')
WORK_DIR = os.path.join(os.getcwd(), 'WORK_DIR')
MODEL_CONFIG = 'text_classification_config.yaml'

In [ ]:

os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(os.path.join(DATA_DIR, 'RE'), exist_ok=True)
os.makedirs(WORK_DIR, exist_ok=True)

In [ ]:

# download the dataset
wget.download('https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip?raw=true',
              os.path.join(DATA_DIR, 'data_re.zip'))

In [ ]:

!unzip -o {DATA_DIR}/data_re.zip -d {RE_DATA_DIR}

In [ ]:

! ls -l $RE_DATA_DIR

Pre-process dataset

Let's convert the dataset into the format that is compatible for NeMo text-classification module.

In [ ]:

wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/data/import_datasets.py')

In [ ]:

! python import_datasets.py --dataset_name=chemprot --source_data_dir={RE_DATA_DIR} --target_data_dir={RE_DATA_DIR}

In [ ]:

# let's take a look at the training data 
! head -n 5 {RE_DATA_DIR}/train.tsv

In [ ]:

# let's check the label mapping
! cat {RE_DATA_DIR}/label_mapping.tsv

It is not necessary to have the mapping exactly like this - it can be different. We use the same mapping used by BioBERT so that comparison can be more straightforward.

Model configuration

Now, let's take a closer look at the model's configuration and learn to train the model.

The model is defined in a config file which declares multiple important sections. They are:

• model: All arguments that are related to the Model - language model, a classifier, optimizer and schedulers, datasets and any other related information

• trainer: Any argument to be passed to PyTorch Lightning

In [ ]:

# download the model's configuration file 
config_dir = WORK_DIR + '/configs/'
os.makedirs(config_dir, exist_ok=True)
if not os.path.exists(config_dir + MODEL_CONFIG):
    print('Downloading config file...')
    wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/text_classification/conf/' + MODEL_CONFIG, config_dir)
else:
    print ('config file is already exists')

In [ ]:

# this line will print the entire config of the model
config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'
print(config_path)
config = OmegaConf.load(config_path)

In [ ]:

config.model.train_ds.file_path = os.path.join(RE_DATA_DIR, 'train.tsv')
config.model.validation_ds.file_path = os.path.join(RE_DATA_DIR, 'dev.tsv')
config.model.task_name = 'chemprot'
# Note: these are small batch-sizes - increase as appropriate to available GPU capacity
config.model.train_ds.batch_size=8
config.model.validation_ds.batch_size=8
config.model.dataset.num_classes=6

In [ ]:

print(OmegaConf.to_yaml(config))

Model Training

Setting up Data within the config

Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.

We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. So, to start model training, we simply need to specify model.dataset.data_dir, like we are going to do below.

Also notice that some config lines, including model.dataset.data_dir, have ??? in place of paths, this means that values for these fields are required to be specified by the user.

Let's now add the data directory path, task name and output directory for saving predictions to the config.

In [ ]:

config.model.task_name = TASK
config.model.output_dir = WORK_DIR
config.model.dataset.data_dir = RE_DATA_DIR

Building the PyTorch Lightning Trainer

NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.

Let's first instantiate a Trainer object

In [ ]:

print("Trainer config - \n")
print(OmegaConf.to_yaml(config.trainer))

In [ ]:

# lets modify some trainer configs
# checks if we have GPU available and uses it
cuda = 1 if torch.cuda.is_available() else 0
config.trainer.gpus = cuda

# for PyTorch Native AMP set precision=16
config.trainer.precision = 16 if torch.cuda.is_available() else 32

# remove distributed training flags
config.trainer.accelerator = 'DDP'

trainer = pl.Trainer(**config.trainer)

Setting up a NeMo Experiment

NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:

In [ ]:

config.exp_manager.exp_dir = WORK_DIR
exp_dir = exp_manager(trainer, config.get("exp_manager", None))

# the exp_dir provides a path to the current experiment for easy access
exp_dir = str(exp_dir)
exp_dir

Before initializing the model, we might want to modify some of the model configs. Here we are modifying it to use BioMegatron, Megatron-LM BERT pre-trained on PubMed biomedical text corpus.

In [ ]:

# complete list of supported BERT-like models
print(nemo_nlp.modules.get_pretrained_lm_models_list())

# specify BERT-like model, you want to use, for example, "megatron-bert-345m-uncased" or 'bert-base-uncased'
PRETRAINED_BERT_MODEL = "biomegatron-bert-345m-uncased"

In [ ]:

# add the specified above model parameters to the config
config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL

Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation. Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model.

In [ ]:

model = nemo_nlp.models.TextClassificationModel(cfg=config.model, trainer=trainer)

Monitoring training progress

Optionally, you can create a Tensorboard visualization to monitor training progress. If you're not using Colab, refer to https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks if you're facing issues with running the cell below.

In [ ]:

try:
    from google import colab
    COLAB_ENV = True
except (ImportError, ModuleNotFoundError):
    COLAB_ENV = False

# Load the TensorBoard notebook extension
if COLAB_ENV:
    %load_ext tensorboard
    %tensorboard --logdir {exp_dir}
else:
    print("To use tensorboard, please use this notebook in a Google Colab environment.")

In [ ]:

# start model training
trainer.fit(model)

Training Script

If you have NeMo installed locally, you can also train the model with examples/nlp/text_classification/text_classification_with_bert.py.

To run training script, use:

python text_classification_with_bert.py \ model.dataset.data_dir=PATH_TO_DATA_DIR \ model.task_name=TASK

The training could take several minutes and the results should look something like:

precision    recall  f1-score   support
    
               0     0.7328    0.8348    0.7805       115
               1     0.9402    0.9291    0.9346      7950
               2     0.8311    0.9146    0.8708       199
               3     0.6400    0.6302    0.6351       457
               4     0.8002    0.8317    0.8156      1093
               5     0.7228    0.7518    0.7370       548
    
        accuracy                         0.8949     10362
       macro avg     0.7778    0.8153    0.7956     10362
    weighted avg     0.8963    0.8949    0.8954     10362

In [ ]: