Viking 7B: open LLM for the Nordic languages trained on AMD GPUs

If you&#x27;re interested in this, don&#x27;t miss AI Sweden&#x27;s GPT-SW3 @ 126M to 40B trained on Nordic languages (not Finnish) and English. It&#x27;s funded by the Swedish government and partners, and freely available with a pretty lively Discord for ongoing AI research focusing on the Nordic languages. I think Viking is called &quot;first&quot; because it includes Finnish, because otherwise, GPT-SW3 was released earlier.<p><a href="https:&#x2F;&#x2F;huggingface.co&#x2F;AI-Sweden-Models" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;AI-Sweden-Models</a>

First thing I notice is that Finnish is part of a completely different language family from the other Nordic languages and English (Uralic vs. Indo-European). I wonder to what extent this affects the effectiveness of their low-resource training. Finnish is highly agglutinative, adding prefixes and suffixes to modify a root. My (amateur) take is that the tokenization and attention patterns may differ a lot? Would love to see more educated people than I discuss this.

The fact it was trained on HPC which covers 20% heat consumption in a city is absolutely wild and on par with how wild it is to have English&#x2F;Nordic model.<p>“ Further emphasizing digital sovereignty, Viking is trained on the EuroHPC supercomputer LUMI, utilizing up to 4096 AMD MI-250X GPUs. LUMI is not only Europe’s most powerful supercomputer and the 5th most powerful in the world, but also the 3rd greenest supercomputer among the top 500 supercomputers. LUMI’s energy consumption is covered with power produced 100% with hydroelectricity, and the waste heat of LUMI will account for about 20 percent of the district heating in the surrounding city of Kajaani. ”

Great talking points. These are highly relevant subjects and I&#x27;m delighted we in the Nordics are keeping up with current developments. This work is important for preserving our culture.<p>I hope to see this used to generate a customized curriculum for each neurodiverse child so that we can live in a more equitable society.

I have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.

Would an LLM trained on a smaller language have better cultural awareness etc than one trained in English? Because English is written all over the world by all kinds of people, an English LLM will average that (and for instance feel a bit off for an American). But a Norwegian LLM for instance, trained on a language mostly written by Norwegians, would that feel more natural to me in comparison?

Would love to know more about their experience training on AMD GPUs. Was it just as seamless as using Cuda?

I cannot seem to find a link to the actual model from this page or anywhere on the website. This appears to be it: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;LumiOpen&#x2F;Viking-7B" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;LumiOpen&#x2F;Viking-7B</a>

&gt; extends to include Danish, Finnish, Norwegian, Icelandic, Swedish<p>* cries in Faroese *

Is there something similar for romance or Germanic languages?<p>And how did they decide that, e.g., German or Dutch would make the model worse?

Although not nordic not including basque which I guess could also be considered an European low-resource language.

double slash in the shared link probably not ideal (though inconsequential)<p><a href="https:&#x2F;&#x2F;www.silo.ai&#x2F;blog&#x2F;viking-7b-the-first-open-llm-for-the-nordic-languages" rel="nofollow">https:&#x2F;&#x2F;www.silo.ai&#x2F;blog&#x2F;viking-7b-the-first-open-llm-for-th...</a>