Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks

State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed as alternatives to Transformer networks in language modeling, by incorporating gating, convolutions, and input-dependent token selection to mitigate the quadratic cost of multi-head attention. Although SSMs exhibit competitive performance, their in-context learning (ICL) capabilities, a remarkable emergent property of mod

[rawwell]

"TransformerとMambaのIn-Context Learning能力を人工的な問題で比較した結果、簡単な問題では両方同等の性能であり、難しい問題ではそれぞれできない問題があった（TransformerはSparse Parity、Mambaは決定木と"