Thus, current research trends are as follows: The new NLP paradigm is “pre-training + fine-tuning”. Transfer learning has dominated NLP research over the last two years. ULMFiT, CoVe, ELMo, OpenAI GPT, BERT, OpenAI GPT-2, XLNet, RoBERTa, ALBERT – this is a non-exhaustive list of important pre-trained language models introduced recently. Even though transfer learning has definitely pushed NLP to th