Attention in Neural Networks and How to Use ItAttention mechanisms in neural networks, otherwise known as neural attention or just attention, have recently attracted a lot of attention (pun intended). In this post, I will try to find a common denominator for different mechanisms and use-cases and I will describe (and implement!) two mechanisms of soft visual attention. What is Attention? Informally, a neural attention mechanism equips a neura
