はてなブックマーク

Recent Advances in Language Model Fine-tuning This article provides an overview of recent methods to fine-tune large pre-trained language models. Fine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing. Over the last three years (Ruder, 2018), fine-tuning (Howard & Ruder, 2018) has superseded the use of feature extra

ML and NLP Research Highlights of 2020 This post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020. The selection of areas and methods is heavily influenced by my own interests; the selected topics are biased towards representation and transfer learning and towards natural language processing (NLP). I tried to cover the papers that I was aware of but likely missed m

Why You Should Do NLP Beyond English 7000+ languages are spoken around the world but NLP research has mostly focused on English. This post outlines why you should work on languages other than English. Natural language processing (NLP) research predominantly focuses on developing methods that work well for English despite the many positive benefits of working on other languages. These benefits rang

10 Tips for Research and a PhD This post outlines 10 things that I did during my PhD and found particularly helpful in the long run. This advice should be most relevant to people studying machine learning (ML) and natural language processing (NLP) as that is what I did in my PhD. Having said that, this advice is not just limited to PhD students. If you are an independent researcher, want to start

10 ML & NLP Research Highlights of 2019 This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019. This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019. For each highlight, I summarise the main advances that took place this year, briefly state why I think it is important, and provide a short outlook to the future. T

NAACL 2019 Highlights This post discusses highlights of NAACL 2019. It covers transfer learning, common sense reasoning, natural language generation, bias, non-English languages, and diversity and inclusion. Update 19.04.20: Added a translation of this post in Spanish. This post discusses highlights of the 2019 Annual Conference of the North American Chapter of the Association for Computational Li

AAAI 2019 Highlights: Dialogue, reproducibility, and more This post discusses highlights of AAAI 2019. It covers dialogue, reproducibility, question answering, the Oxford style debate, invited talks, and a diverse set of research papers. This post discusses highlights of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19). I attended AAAI 2019 in Honolulu, Hawaii last week. Overa

10 Exciting Ideas of 2018 in NLP This post gathers 10 ideas that I found exciting and impactful this year—and that we'll likely see more of in the future. For each idea, it highlights 1-2 papers that execute them well. This post gathers 10 ideas that I found exciting and impactful this year—and that we'll likely see more of in the future. For each idea, I will highlight 1-2 papers that execute the

Requests for Research It can be hard to find compelling topics to work on and know what questions to ask when you are just starting as a researcher. This post aims to provide inspiration and ideas for research directions to junior researchers and those trying to get into research. This post aims to provide inspiration and ideas for research directions to junior researchers and those trying to get

Optimization for Deep Learning Highlights in 2017 Different gradient descent optimization algorithms have been proposed in recent years but Adam is still most commonly used. This post discusses the most exciting highlights and most promising recent approaches that may shape the way we will optimize our models in the future. This post discusses the most exciting highlights and most promising direct

Word embeddings in 2017: Trends and future directions Word embeddings are an integral part of current NLP models, but approaches that supersede the original word2vec have not been proposed. This post focuses on the deficiencies of word embeddings and how recent approaches have tried to resolve them. This post discusses the deficiencies of word embeddings and how recent approaches have tried to res

Deep Learning for NLP Best Practices Neural networks are widely used in NLP, but many details such as task or domain-specific considerations are left to the practitioner. This post collects best practices that are relevant for most tasks in NLP. This post gives an overview of best practices relevant for most tasks in natural language processing. Update July 26, 2017: For additional context, the Ha

An Overview of Multi-Task Learning in Deep Neural Networks Multi-task learning is becoming more and more popular. This post gives a general overview of the current state of multi-task learning. In particular, it provides context for current neural network-based methods by discussing the extensive multi-task learning literature. Note: If you are looking for a review paper, this blog post is also av

Transfer Learning - Machine Learning's Next Frontier Deep learning models excel at learning from a large number of labeled examples, but typically do not generalize to conditions not seen during training. This post gives an overview of transfer learning, motivates why it warrants our application, and discusses practical applications and methods. This post gives an overview of transfer learning and

This post discusses Command R and Command R+, the top open-weights model on Chatbot Arena at the time of its release and highlights their RAG and multilingual capabilities.

On word embeddings - Part 1 Word embeddings popularized by word2vec are pervasive in current NLP applications. The history of word embeddings, however, goes back a lot further. This post explores the history of word embeddings in the context of language modelling. This post presents the most well-known models for learning word embeddings based on language modelling. Table of contents: A brief hist

On word embeddings - Part 2: Approximating the Softmax The softmax layer is a core part of many current neural network architectures. When the number of output classes is very large, such as in the case of language modelling, computing the softmax becomes very expensive. This post explores approximations to make the computation more efficient. This post gives an overview of approximations that can

An overview of gradient descent optimization algorithms Gradient descent is the preferred way to optimize neural networks and many other machine learning algorithms but is often used as a black box. This post explores how many of the most popular gradient-based optimization algorithms such as Momentum, Adagrad, and Adam actually work. This post explores how many of the most popular gradient-based