An Analysis of Neural Language Modeling at Multiple ScalesMany of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-
Smerity on Twitter: "Simplicity is still competitive! Standard LSTM/QRNN models can achieve state-of-the-art results on character (PTB,… https://t.co/1h5QASLSL1"
