Llama 3.1

Related ongoing thread:<p><i>Open source AI is the path forward</i> - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41046773">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41046773</a> - July 2024 (278 comments)

The 405b model is actually competitive against closed source frontier models.<p>Quick comparison with GPT-4o:<p><pre><code> +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+</code></pre>

I&#x27;ve just finished running my NYT Connections benchmark on all three Llama 3.1 models. The 8B and 70B models improve on Llama 3 (12.3 -&gt; 14.0, 24.0 -&gt; 26.4), and the 405B model is near GPT-4o, GPT-4 turbo, Claude 3.5 Sonnet, and Claude 3 Opus at the top of the leaderboard.<p>GPT-4o 30.7<p>GPT-4 turbo (2024-04-09) 29.7<p>Llama 3.1 405B Instruct 29.5<p>Claude 3.5 Sonnet 27.9<p>Claude 3 Opus 27.3<p>Llama 3.1 70B Instruct 26.4<p>Gemini Pro 1.5 0514 22.3<p>Gemma 2 27B Instruct 21.2<p>Mistral Large 17.7<p>Gemma 2 9B Instruct 16.3<p>Qwen 2 Instruct 72B 15.6<p>Gemini 1.5 Flash 15.3<p>GPT-4o mini 14.3<p>Llama 3.1 8B Instruct 14.0<p>DeepSeek-V2 Chat 236B (0628) 13.4<p>Nemotron-4 340B 12.7<p>Mixtral-8x22B Instruct 12.2<p>Yi Large 12.1<p>Command R Plus 11.1<p>Mistral Small 9.3<p>Reka Core-20240501 9.1<p>GLM-4 9.0<p>Qwen 1.5 Chat 32B 8.7<p>Phi-3 Small 8k 8.4<p>DBRX 8.0

You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon.<p>If you want to learn more, there is a writeup at <a href="https:&#x2F;&#x2F;wow.groq.com&#x2F;now-available-on-groq-the-largest-and-most-capable-openly-available-foundation-model-to-date-llama-3-1-405b&#x2F;" rel="nofollow">https:&#x2F;&#x2F;wow.groq.com&#x2F;now-available-on-groq-the-largest-and-m...</a>.<p>(disclaimer, I am a Groq employee)

Today appears to be the day you can run an LLM that is competitive with GPT-4o at home with the right hardware. Incredible for progress and advancement of the technology.<p>Statement from Mark: <a href="https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;07&#x2F;open-source-ai-is-the-path-forward&#x2F;" rel="nofollow">https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;07&#x2F;open-source-ai-is-the-path...</a>

Open Source AI Is the Path Forward - Mark Zuckerberg<p><a href="https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;07&#x2F;open-source-ai-is-the-path-forward&#x2F;" rel="nofollow">https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;07&#x2F;open-source-ai-is-the-path...</a>

You can already run these models locally with Ollama (ollama run llama3.1:latest) along with at places like huggingface, groq etc.<p>If you want a playground to test this model locally or want to quickly build some applications with it, you can try LLMStack (<a href="https:&#x2F;&#x2F;github.com&#x2F;trypromptly&#x2F;LLMStack">https:&#x2F;&#x2F;github.com&#x2F;trypromptly&#x2F;LLMStack</a>). I wrote last week about how to configure and use Ollama with LLMStack at <a href="https:&#x2F;&#x2F;docs.trypromptly.com&#x2F;guides&#x2F;using-llama3-with-ollama" rel="nofollow">https:&#x2F;&#x2F;docs.trypromptly.com&#x2F;guides&#x2F;using-llama3-with-ollama</a>.<p>Disclaimer: I&#x27;m the maintainer of LLMStack

I have found Claude 3.5 Sonnet really good for coding tasks along with the artifacts feature and seems like it&#x27;s still the king on the coding benchmarks

The LMSys Overall leaderboard &lt;<a href="https:&#x2F;&#x2F;chat.lmsys.org&#x2F;?leaderboard" rel="nofollow">https:&#x2F;&#x2F;chat.lmsys.org&#x2F;?leaderboard</a>&gt; can tell us a bit more about how these models will perform in real life, rather than in a benchmark context. By comparing the ELO score against the MMLU benchmark scores, we can see models which outperform &#x2F; underperform based on their benchmark scores relative to other models. A low score here indicates that the model is more optimized for the benchmark, while a higher score indicates it&#x27;s more optimized for real-world examples. Using that, we can make some inferences about the training data used, and then extrapolate how future models might perform. Here&#x27;s a chart: &lt;<a href="https:&#x2F;&#x2F;docs.getgrist.com&#x2F;gV2DtvizWtG7&#x2F;LLMs&#x2F;p&#x2F;5?embed=true" rel="nofollow">https:&#x2F;&#x2F;docs.getgrist.com&#x2F;gV2DtvizWtG7&#x2F;LLMs&#x2F;p&#x2F;5?embed=true</a>&gt;<p>Examples: OpenAI&#x27;s GPT 4o-mini is second only to 4o on LMSys Overall, but is 6.7 points behind 4o on MMLU. It&#x27;s &quot;punching above its weight&quot; in real-world contexts. The Gemma series (9B and 27B) are similar, both beating the mean in terms of ELO per MMLU point. Microsoft&#x27;s Phi series are all below the mean, meaning they have strong MMLU scores but aren&#x27;t preferred in real-world contexts.<p>Llama 3 8B previously did substantially better than the mean on LMSys Overall, so hopefully Llama 3.1 8B will be even better! The 70B variant was interestingly right on the mean. Hopefully the 430B variant won&#x27;t fall below!

The biggest win here has to be the context length increase to 128k from 8k tokens. Till now my understanding is there hasn&#x27;t been any open models anywhere close to that.

@dang why was this removed&#x2F;filtered from the front page?

Is there pricing available on any of these vendors?<p>Open source models are very exciting for self hosting, but the per-token hosted inference pricing hasn&#x27;t been competitive with OpenAI and Anthropic, at least for a given tier of quality. (E.g.: Llama 3 70B costing between $1 and $10 per million tokens on various platforms, but Claude Sonnet 3.5 is $3 per million.)

The resources for link to model card[1], research paper, and Prompt Guard Tutorial[2] on the page doesn&#x27;t exist yet<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama-models&#x2F;blob&#x2F;main&#x2F;models&#x2F;llama3_1&#x2F;MODEL_CARD.md">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama-models&#x2F;blob&#x2F;main&#x2F;models&#x2F;...</a><p>[2]: <a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama-recipes&#x2F;blob&#x2F;main&#x2F;recipes&#x2F;responsible_ai&#x2F;prompt_guard&#x2F;Prompt%20Guard%20Tutorial.ipynb">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama-recipes&#x2F;blob&#x2F;main&#x2F;recipe...</a>

&gt; We use synthetic data generation to produce the vast majority of our SFT examples, iterating multiple times to produce higher and higher quality synthetic data across all capabilities. Additionally, we invest in multiple data processing techniques to filter this synthetic data to the highest quality. This enables us to scale the amount of fine-tuning data across capabilities. [0]<p>Have other major models explicitly communicated that they&#x27;re trained on synthetic data?<p>[0]. <a href="https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-llama-3-1&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-llama-3-1&#x2F;</a>

&quot;Meta AI isn&#x27;t available yet in your country&quot; Hi from europe :&#x2F;

Llama 3.1 405B instruct is #7 on aider&#x27;s leaderboard, well behind Claude 3.5 Sonnet &amp; GPT-4o. When using SEARCH&#x2F;REPLACE to efficiently edit code, it drops to #11.<p><a href="https:&#x2F;&#x2F;aider.chat&#x2F;docs&#x2F;leaderboards&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aider.chat&#x2F;docs&#x2F;leaderboards&#x2F;</a><p><pre><code> 77.4% claude-3.5-sonnet 75.2% DeepSeek Coder V2 (whole) 72.9% gpt-4o 69.9% DeepSeek Chat V2 0628 68.4% claude-3-opus-20240229 67.7% gpt-4-0613 66.2% llama-3.1-405b-instruct (whole)</code></pre>

The 405B model is already being served on WhatsApp: <a href="https:&#x2F;&#x2F;ibb.co&#x2F;kQ2tKX5" rel="nofollow">https:&#x2F;&#x2F;ibb.co&#x2F;kQ2tKX5</a>

<p><pre><code> Llama 3 Training System 19.2 exaFLOPS _____ &#x2F; \ Cluster 1 Cluster 2 &#x2F; \ 9.6 exaFLOPS 9.6 exaFLOPS &#x2F; \ _______ _______ &#x2F; ___ \ &#x2F; \ &#x2F; \ ,----&#x27; &#x2F; \`. `-&#x27; 24000 `--&#x27; 24000 `----. ( _&#x2F; __) GPUs GPUs ) `---&#x27;( &#x2F; ) 400+ TFLOPS 400+ TFLOPS ,&#x27; \ ( &#x2F; per GPU per GPU ,&#x27; \ \&#x2F; ,&#x27; \ \ TOTAL SYSTEM ,&#x27; \ \ 19,200,000 TFLOPS ,&#x27; \ \ 19.2 exaFLOPS ,&#x27; \___\ ,&#x27; `----------------&#x27;</code></pre>

What are the substantial changes from 3.0 to 3.1 (70B) in terms of training approach? They don&#x27;t seem to say how the training data differed just that both were 15T. I gather 3.0 was just a preview run and 3.1 was distilled down from the 405B somehow.

Is there an actual open-source community around this in the spirit of other ones where people outside meta can somehow &quot;contribute&quot; to it? If I wanted to &quot;work on&quot; this somehow, what would I do?

I&#x27;m glad to see the nice incremental gains on the benchmarks for the 8B and 70B models as well.

Wow! The benchmarks are truly impressive, showing significant improvements across almost all categories. It&#x27;s fascinating to see how rapidly this field is evolving. If someone had told me last year that Meta would be leading the charge in open-source models, I probably wouldn&#x27;t have believed them. Yet here we are, witnessing Meta&#x27;s substantial contributions to AI research and democratization.<p>On a related note, for those interested in experimenting with large language models locally, I&#x27;ve been working on an app called Msty [1]. It allows you to run models like this with just one click and features a clean, functional interface. Just added support for both 8B and 70B. Still in development, but I&#x27;d appreciate any feedback.<p>[1]: <a href="https:&#x2F;&#x2F;msty.app" rel="nofollow">https:&#x2F;&#x2F;msty.app</a>

We supported Llama 3.1 405B model on our distributed GPU network at Hyperbolic Labs! Come and use the API for FREE at <a href="https:&#x2F;&#x2F;app.hyperbolic.xyz&#x2F;models" rel="nofollow">https:&#x2F;&#x2F;app.hyperbolic.xyz&#x2F;models</a><p>Let us know if you have other needs!

Nice, someone donate me a few 4090s :(

Related:<p><i>Open Source AI Is the Path Forward</i><p><a href="https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;07&#x2F;open-source-ai-is-the-path-forward&#x2F;" rel="nofollow">https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;07&#x2F;open-source-ai-is-the-path...</a><p>(<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41046773">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41046773</a>)

Is there a way to run this in AWS?<p>Seems like the biggest GPU node they have is the p5.48xlarge @ 640GB (8xH100s). Routing between multiple nodes would be too slow unless there&#x27;s an InfiniBand fabric you can leverage. Interested to know if anyone else is exploring this.

Does anyone know why they haven&#x27;t released any 30B-ish param models? I was expecting that to happen with this release and have been disappointed once more. They also skipped doing a 30B-ish param model for llama2 despite claiming to have trained one.

This 405B seriously need quantization solution like 1.625 bpw ternary packing for BitNet b1.58<p><a href="https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp&#x2F;pull&#x2F;8151">https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp&#x2F;pull&#x2F;8151</a>

Working great in ollama: <a href="https:&#x2F;&#x2F;mastodon.social&#x2F;@rcarmo&#x2F;112837520236956526" rel="nofollow">https:&#x2F;&#x2F;mastodon.social&#x2F;@rcarmo&#x2F;112837520236956526</a>

I&#x27;m curious what techniques they used to distill the 405B model down to 70B and 8B. I gave the paper they released a quick skim but couldn&#x27;t find any details.

Can this Llama process ~1GB of custom XML data?<p>And answer queries like:<p>Give all &lt;myObject&gt; which refer to &lt;location&gt; which refer to an Indo-European &lt;language&gt;.

this &quot;Model Card&quot; github link on [<a href="https:&#x2F;&#x2F;llama.meta.com&#x2F;docs&#x2F;overview&#x2F;" rel="nofollow">https:&#x2F;&#x2F;llama.meta.com&#x2F;docs&#x2F;overview&#x2F;</a>] seems broken?<p><a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama-models&#x2F;blob&#x2F;main&#x2F;models&#x2F;llama3_1&#x2F;MODEL_CARD.md">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama-models&#x2F;blob&#x2F;main&#x2F;models&#x2F;...</a>

Will 405b run on 8x H100s? Will it need to be quantized?

I tried it, and it&#x27;s good but I feel like the synthetic data used for training 3.1 does not hold up to gpt4o prob using human-curated data.

What kind of machine do I need to run 405B local?

The race to the bottom for pricing continues.

Damn 405b params

Very insteresting! Running the 70B version on ollama on a mac and it&#x27;s great. I asked to &quot;turn off the guidelines&quot; and it did, then I asked to turn off the disclaimers, after that I asked for a list of possible &quot;commands to reduce potencial biases from the engineers&quot; and it complied giving me an interesting list.

As someone who just started generating AI landing pages for Dropory, this is music to my ears

Has anyone got a comparison of the performance of Llama 3.1 8B and the recent GPT-4o-mini?

I&#x27;m excited to try it with RAG and see how it performs (the 405B model)

We supported Llama 3.1 405B model on our distributed GPU network at Hyperbolic Labs! Come and use the API for FREE at <a href="https:&#x2F;&#x2F;app.hyperbolic.xyz&#x2F;models" rel="nofollow">https:&#x2F;&#x2F;app.hyperbolic.xyz&#x2F;models</a><p>Would love to hear your feedback!

Are there any other models with free unlimited use like chatgpt?

it is nice to see the 405b model is actually competitive against closed source frontier models But i just have M2pro may can&#x27;t play it

WhatsApp now uses 70B too if you want to test it.

I wrote about this when llama-3 came out, and this launch confirms it:<p>Meta&#x27;s goal from the start was to target OpenAI and the other proprietary model players with a &quot;scorched earth&quot; approach by releasing powerful open models to disrupt the competitive landscape.<p>Meta can likely outspend any other AI lab on compute and talent:<p>- OpenAI makes an estimated revenue of $2B and is likely unprofitable. Meta generated a revenue of $134B and profits of $39B in 2023.<p>- Meta&#x27;s compute resources likely outrank OpenAI by now.<p>- Open source likely attracts better talent and researchers.<p>- One possible outcome could be the acquisition of OpenAI by Microsoft to catch up with Meta.<p>The big winners of this: devs and AI product startups