ML and NLP Research Highlights of 2020
The selection of areas and methods is heavily influenced by my own interests; the selected topics are biased towards representation and transfer learning and towards natural language processing (NLP). I tried to cover the papers that I was aware of but likely missed many relevant ones—feel free to highlight them in the comments below. In all, I discuss the following highlights: Scaling up—and down
mT5: A massively multilingual pre-trained text-to-text transformer
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its s
The first AI model that translates 100 languages without relying on English data
The first AI model that translates 100 languages without relying on English data Facebook AI is introducing, M2M-100 the first multilingual machine translation (MMT) model that translates between any pair of 100 languages without relying on English data. It’s open sourced here. When translating, say, Chinese to French, previous best multilingual models train on Chinese to English and English to Fr
GitHub - megagonlabs/jrte-corpus: Japanese Realistic Textual Entailment Corpus (NLP 2020, LREC 2020)
Deep double descent
We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of thi
OpenNMT - Open-Source Neural Machine Translation
OpenNMT is an open source ecosystem for neural machine translation and neural sequence learning. Started in December 2016 by the Harvard NLP group and SYSTRAN, the project has since been used in several research and industry applications. It is currently maintained by SYSTRAN and Ubiqus. OpenNMT provides implementations in 2 popular deep learning frameworks:
Japanese Language Analysis by GPU Ready Open Source NLP Frameworks
Japanese Language Analysis by GPU Ready Open Source NLP Frameworks Hiroshi Matsuda GPU Technology Conference 2020 1 This document is published under CC BY 4.0 license from Megagon Labs, Recruit Co., Ltd. Contents 自然言語処理技術の進歩とGPUが与えた影響 ルールベースからTransformersまで NLP Frameworkを用いた日本語の解析 世界の全言語を統一的に扱うUniversal Dependenciesとその日本語化 GiNZAの文節API GPU Ready OSS NLP Frameworks spaCy v2.3 → v3.0 Stanza
How GPT3 Works - Visualizations and Animations
Jay Alammar Visualizing machine learning one concept at a time. @JayAlammar on Twitter. YouTube Channel Discussions: Hacker News (397 points, 97 comments), Reddit r/MachineLearning (247 points, 27 comments) Translations: German, Korean, Chinese (Simplified), Russian, Turkish The tech world is abuzz with GPT3 hype. Massive language models (like GPT3) are starting to surprise us with their abilities
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
gpt-3/dataset_statistics/languages_by_word_count.csv at master · openai/gpt-3
Which flavor of BERT should you use for your QA task?
jiant is an NLP toolkit: Introducing jiant 2.0
GitHub - junya-takayama/open2ch_chatbot_openNMT
Google Colab
dialog-ja
「自然言語処理の未来」HuggingFace 主席サイエンティストが語る NLP の最新トレンド
GitHub - ritchieng/the-incredible-pytorch: The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.
GitHub - elyase/awesome-gpt3
最先端自然言語処理ライブラリの最適な選択と有用な利用方法 / pycon-jp-2020
Text-to-Speech AI: Lifelike Speech Synthesis | Google Cloud
NLP Newsletter – dair.ai