GitHub Repo: <a href="https://github.com/IBM/ModuleFormer">https://github.com/IBM/ModuleFormer</a><p>GitHub Description:
ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.