The selection of areas and methods is heavily influenced by my own interests; the selected topics are biased towards representation and transfer learning and towards natural language processing (NLP). I tried to cover the papers that I was aware of but likely missed many relevant ones—feel free to highlight them in the comments below. In all, I discuss the following highlights: Scaling up—and down

This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019. For each highlight, I summarise the main advances that took place this year, briefly state why I think it is important, and provide a short outlook to the future. The full list of highlights is here: Universal unsupervised pretrainingLottery ticketsThe Neural Tangent KernelUnsupervised multilingual le

Update 16.10.2020: Added Chinese and Spanish translations. This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. The tutorial was organized by Matthew Peters, Swabha Swayamdipta, Thomas Wolf, and me. In this post, I highlight key insights and takeaways and provide updates based on recent work. You can see the structure of this post below: The slides, a Colaboratory notebook, an

NAACL 2019 Highlights This post discusses highlights of NAACL 2019. It covers transfer learning, common sense reasoning, natural language generation, bias, non-English languages, and diversity and inclusion. Update 19.04.20: Added a translation of this post in Spanish. This post discusses highlights of the 2019 Annual Conference of the North American Chapter of the Association for Computational Li

10 Exciting Ideas of 2018 in NLP This post gathers 10 ideas that I found exciting and impactful this year—and that we'll likely see more of in the future. For each idea, it highlights 1-2 papers that execute them well. This post gathers 10 ideas that I found exciting and impactful this year—and that we'll likely see more of in the future. For each idea, I will highlight 1-2 papers that execute the

Optimization for Deep Learning Highlights in 2017 Different gradient descent optimization algorithms have been proposed in recent years but Adam is still most commonly used. This post discusses the most exciting highlights and most promising recent approaches that may shape the way we will optimize our models in the future. This post discusses the most exciting highlights and most promising direct

Word embeddings in 2017: Trends and future directions Word embeddings are an integral part of current NLP models, but approaches that supersede the original word2vec have not been proposed. This post focuses on the deficiencies of word embeddings and how recent approaches have tried to resolve them. This post discusses the deficiencies of word embeddings and how recent approaches have tried to res

Deep Learning for NLP Best Practices Neural networks are widely used in NLP, but many details such as task or domain-specific considerations are left to the practitioner. This post collects best practices that are relevant for most tasks in NLP. This post gives an overview of best practices relevant for most tasks in natural language processing. Update July 26, 2017: For additional context, the Ha