IBM is releasing a family of Granite code models to the open-source community. The aim is to make coding as easy as possible — for as many developers as possible.
[....]
We’re releasing a series of decoder-only Granite code models for code generative tasks, trained with code written in 116 programming languages. The Granite code models family consists of models ranging in size from 3 to 34 billion parameters, in both a base model and instruction-following model variants. These models have a range of uses, from complex application modernization tasks to on-device memory-constrained use cases.
[....]
For the 34B version of the model, we used a novel method called depth upscaling to train the model. First, we created a duplicated version of the 20B variant, which has 52 layers to it. We removed the final eight layers from the first version of the 20B, and the first eight from the second version. We then merged the two versions to create a new model with 88 layers. We used the same 8,192 token context window when pre-training both the 20B and 34B model.
[....]
Our models can outperform some twice their size, such as with Code Llama, and while some other models may perform slightly better in some tasks like code generation, no one model could perform at a high level at generation, fixing, and explanation — apart from Granite.