In the past two years, there have been numerous papers written on Transformer, and researchers are designing Transformer models for all kinds of tasks. However, is attention, the core module of Transformer, really stronger than convolution? This paper may bring to you a new perspective. Researchers from Microsoft Research Asia have looked into local attention and dynamic depth-wise convolution and
![ICLR 2022 Spotlight: Demystifying local attention and dynamic depth-wise convolution - Microsoft Research](https://cdn-ak-scissors.b.st-hatena.com/image/square/17124fbe31179ed371e2e5a8f285ce715e988b68/height=288;version=1;width=512/https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fuploads%2Fprod%2F2022%2F07%2Fimage010.png)