An Empirical Study of Mamba-Based Language Models

What’s the largest Mamba model that has been trained so far?<p>Seems like it scales better than transformers, but this would only be really obvious at parameter counts far in excess of the experiments in this paper.