artcraft/models/tts/NeMo/tutorials/nlp/Token_Classification-BioMegatron.ipynb

In [ ]:

BRANCH = 'v1.0.2'

In [ ]:

"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
"""
# If you're using Google Colab and not running locally, run this cell

# install NeMo
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]

In [ ]:

# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:
# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'

! pip install ipywidgets
! jupyter nbextension enable --py widgetsnbextension

# Please restart the kernel after running this cell

In [ ]:

from nemo.collections import nlp as nemo_nlp
from nemo.utils.exp_manager import exp_manager

import os
import wget 
import torch
import pytorch_lightning as pl
from omegaconf import OmegaConf

In this tutorial, we are going to describe how to finetune BioMegatron - a BERT-like Megatron-LM model pre-trained on large biomedical text corpus (PubMed abstracts and full-text commercial use collection) - on the NCBI Disease Dataset for Named Entity Recognition.

The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large. There are some alternatives of BioMegatron, most notably BioBERT. Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.

A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found here.

Task Description

Named entity recognition (NER), also referred to as entity chunking, identification or extraction, is the task of detecting and classifying key information (entities) in text.

For instance, given sentences from medical abstracts, what diseases are mentioned?
In this case, our data input is sentences from the abstracts, and our labels are the precise locations of the named disease entities. Take a look at the information provided for the dataset.

For more details and general examples on Named Entity Recognition, please refer to the Token Classification and Named Entity Recognition tutorial notebook.

Dataset

The NCBI-disease corpus is a set of 793 PubMed abstracts, annotated by 14 annotators. The annotations take the form of HTML-style tags inserted into the abstract text using the clearly defined rules. The annotations identify named diseases, and can be used to fine-tune a language model to identify disease mentions in future abstracts, whether those diseases were part of the original training set or not.

Here's an example of what an annotated abstract from the corpus looks like:

10021369	Identification of APC2, a homologue of the <category="Modifier">adenomatous polyposis coli tumour</category> suppressor .	The <category="Modifier">adenomatous polyposis coli ( APC ) tumour</category>-suppressor protein controls the Wnt signalling pathway by forming a complex with glycogen synthase kinase 3beta ( GSK-3beta ) , axin / conductin and betacatenin . Complex formation induces the rapid degradation of betacatenin . In <category="Modifier">colon carcinoma</category> cells , loss of APC leads to the accumulation of betacatenin in the nucleus , where it binds to and activates the Tcf-4 transcription factor ( reviewed in [ 1 ] [ 2 ] ) . Here , we report the identification and genomic structure of APC homologues . Mammalian APC2 , which closely resembles APC in overall domain structure , was functionally analyzed and shown to contain two SAMP domains , both of which are required for binding to conductin . Like APC , APC2 regulates the formation of active betacatenin-Tcf complexes , as demonstrated using transient transcriptional activation assays in APC - / - <category="Modifier">colon carcinoma</category> cells . Human APC2 maps to chromosome 19p13 . 3 . APC and APC2 may therefore have comparable functions in development and <category="SpecificDisease">cancer</category> .

In this example, we see the following tags within the abstract:

<category="Modifier">adenomatous polyposis coli tumour</category>
<category="Modifier">adenomatous polyposis coli ( APC ) tumour</category>
<category="Modifier">colon carcinoma</category>
<category="Modifier">colon carcinoma</category>
<category="SpecificDisease">cancer</category>

For our purposes, we will consider any identified category (such as "Modifier", "Specific Disease", and a few others) to generally be a "disease".

Let's download the dataset.

In [ ]:

DATA_DIR = "DATA_DIR"
os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(os.path.join(DATA_DIR, 'NER'), exist_ok=True)

In [ ]:

print('Downloading NCBI data...')
wget.download('https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/NCBI_corpus.zip', DATA_DIR)
! unzip -o {DATA_DIR}/NCBI_corpus.zip -d {DATA_DIR}

In [ ]:

# If you want to see more examples, you can explore the text of the corpus using the file browser to the left, or open files directly, for example typing a command like the following in a code-cell:

! head -1 $DATA_DIR/NCBI_corpus_testing.txt

We have two datasets derived from this corpus: a text classification dataset and a named entity recognition (NER) dataset. The text classification dataset labels the abstracts among three broad disease groupings. We'll use this simple split to demonstrate the NLP text classification task. The NER dataset labels individual words as diseases. This dataset will be used for the NLP NER task.

Pre-process dataset

A pre-processed NCBI-disease dataset for NER can be found here or here.
We download the files under {DATA_DIR/NER} directory.

In [ ]:

NER_DATA_DIR = f'{DATA_DIR}/NER'
wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/train.tsv', NER_DATA_DIR)
wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/devel.tsv', NER_DATA_DIR)
wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/test.tsv', NER_DATA_DIR)

In [ ]:

!ls -lh $NER_DATA_DIR

Convert these to a format that is compatible with NeMo Token Classification module, using the conversion script.

In [ ]:

! mv $NER_DATA_DIR/devel.tsv $NER_DATA_DIR/dev.tsv

In [ ]:

wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/data/import_from_iob_format.py')

In [ ]:

! python import_from_iob_format.py --data_file=$NER_DATA_DIR/train.tsv
! python import_from_iob_format.py --data_file=$NER_DATA_DIR/dev.tsv
! python import_from_iob_format.py --data_file=$NER_DATA_DIR/test.tsv

The NER task requires two files: the text sentences, and the labels. Run the next two cells to see a sample of the two files.

In [ ]:

!head $NER_DATA_DIR/text_train.txt

In [ ]:

!head $NER_DATA_DIR/labels_train.txt

IOB Tagging

We can see that the abstract has been broken into sentences. Each sentence is then further parsed into words with labels that correspond to the original HTML-style tags in the corpus.

The sentences and labels in the NER dataset map to each other with inside, outside, beginning (IOB) tagging. Anything separated by white space is a word, including punctuation. For the first sentence we have the following mapping:

Identification of APC2 , a homologue of the adenomatous polyposis coli tumour suppressor .
O              O  O    O O O         O  O   B           I         I    I      O          O  

Recall the original corpus tags:

Identification of APC2, a homologue of the <category="Modifier">adenomatous polyposis coli tumour</category> suppressor .

The beginning word of the tagged text, "adenomatous", is now IOB-tagged with a B (beginning) tag, the other parts of the disease, "polyposis coli tumour" tagged with I (inside) tags, and everything else tagged as O (outside).

Model configuration

Our Named Entity Recognition model is comprised of the pretrained BERT model followed by a Token Classification layer.

The model is defined in a config file which declares multiple important sections. They are:

• model: All arguments that are related to the Model - language model, token classifier, optimizer and schedulers, datasets and any other related information

• trainer: Any argument to be passed to PyTorch Lightning

In [ ]:

WORK_DIR = "WORK_DIR"
os.makedirs(WORK_DIR, exist_ok=True)
MODEL_CONFIG = "token_classification_config.yaml"

In [ ]:

# download the model's configuration file 
config_dir = WORK_DIR + '/configs/'
os.makedirs(config_dir, exist_ok=True)
if not os.path.exists(config_dir + MODEL_CONFIG):
    print('Downloading config file...')
    wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG, config_dir)
else:
    print ('config file is already exists')

In [ ]:

# this line will print the entire config of the model
config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'
print(config_path)
config = OmegaConf.load(config_path)
# Note: these are small batch-sizes - increase as appropriate to available GPU capacity
config.model.train_ds.batch_size=8
config.model.validation_ds.batch_size=8

In [ ]:

print(OmegaConf.to_yaml(config))

Model Training

Setting up Data within the config

Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.

We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. So, to start model training, we simply need to specify model.dataset.data_dir, like we are going to do below.

Also notice that some config lines, including model.dataset.data_dir, have ??? in place of paths, this means that values for these fields are required to be specified by the user.

Let's now add the data directory path to the config.

In [ ]:

# in this tutorial train and dev datasets are located in the same folder, so it is enought to add the path of the data directory to the config
config.model.dataset.data_dir = os.path.join(DATA_DIR, 'NER')

# if you want to decrease the size of your datasets, uncomment the lines below:
# NUM_SAMPLES = 1000
# config.model.train_ds.num_samples = NUM_SAMPLES
# config.model.validation_ds.num_samples = NUM_SAMPLES

Building the PyTorch Lightning Trainer

NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.

Let's first instantiate a Trainer object

In [ ]:

print("Trainer config - \n")
print(OmegaConf.to_yaml(config.trainer))

In [ ]:

# lets modify some trainer configs
# checks if we have GPU available and uses it
cuda = 1 if torch.cuda.is_available() else 0
config.trainer.gpus = cuda

# for PyTorch Native AMP set precision=16
config.trainer.precision = 16 if torch.cuda.is_available() else 32

# remove distributed training flags
config.trainer.accelerator = 'ddp'

trainer = pl.Trainer(**config.trainer)

Setting up a NeMo Experiment

NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:

In [ ]:

exp_dir = exp_manager(trainer, config.get("exp_manager", None))

# the exp_dir provides a path to the current experiment for easy access
exp_dir = str(exp_dir)
exp_dir

In [ ]:

# complete list of supported BERT-like models
print(nemo_nlp.modules.get_pretrained_lm_models_list())

# specify BERT-like model, you want to use
PRETRAINED_BERT_MODEL = "biomegatron-bert-345m-cased"

In [ ]:

# add the specified above model parameters to the config
config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL

Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation. Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model.

In [ ]:

model_ner = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)

Monitoring training progress

Optionally, you can create a Tensorboard visualization to monitor training progress. If you're not using Colab, refer to https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks if you're facing issues with running the cell below.

In [ ]:

try:
    from google import colab
    COLAB_ENV = True
except (ImportError, ModuleNotFoundError):
    COLAB_ENV = False

# Load the TensorBoard notebook extension
if COLAB_ENV:
    %load_ext tensorboard
    %tensorboard --logdir {exp_dir}
else:
    print("To use tensorboard, please use this notebook in a Google Colab environment.")

In [ ]:

# start model training
trainer.fit(model_ner)

Inference

To see how the model performs, we can run generate prediction similar to the way we did it earlier

In [ ]:

# let's first create a subset of our dev data
! head -n 100 $NER_DATA_DIR/text_dev.txt > $NER_DATA_DIR/sample_text_dev.txt
! head -n 100 $NER_DATA_DIR/labels_dev.txt > $NER_DATA_DIR/sample_labels_dev.txt

Now, let's generate predictions for the provided text file. If labels file is also specified, the model will evaluate the predictions and plot confusion matrix.

In [ ]:

model_ner.evaluate_from_file(
    text_file=os.path.join(NER_DATA_DIR, 'sample_text_dev.txt'),
    labels_file=os.path.join(NER_DATA_DIR, 'sample_labels_dev.txt'),
    output_dir=exp_dir,
    add_confusion_matrix=False,
    normalize_confusion_matrix=True,
    batch_size=1
)
# Please check matplotlib version if encountering any error plotting confusion matrix:
# https://stackoverflow.com/questions/63212347/importerror-cannot-import-name-png-from-matplotlib

Training Script

If you have NeMo installed locally, you can also train the model with nlp/token_classification/token_classification_train.py.

To run training script, use:

python token_classification_train.py model.dataset.data_dir=PATH_TO_DATA_DIR PRETRAINED_BERT_MODEL=biomegatron-bert-345m-cased

The training could take several minutes and the result should look something like

[NeMo I 2020-05-22 17:13:48 token_classification_callback:82] Accuracy: 0.9882348032875798
[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 weighted: 98.82
[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 macro: 93.74
[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 micro: 98.82
[NeMo I 2020-05-22 17:13:49 token_classification_callback:89] precision    recall  f1-score   support
    
    O (label id: 0)     0.9938    0.9957    0.9947     22092
    B (label id: 1)     0.8843    0.9034    0.8938       787
    I (label id: 2)     0.9505    0.8982    0.9236      1090
    
           accuracy                         0.9882     23969
          macro avg     0.9429    0.9324    0.9374     23969
       weighted avg     0.9882    0.9882    0.9882     23969

In [ ]: