It turns out someone has written a plugin for my LLM CLI tool already: <a href="https://github.com/irthomasthomas/llm-cerebras">https://github.com/irthomasthomas/llm-cerebras</a><p>You need an API key - I got one from <a href="https://cloud.cerebras.ai/" rel="nofollow">https://cloud.cerebras.ai/</a> but I'm not sure if there's a waiting list at the moment - then you can do this:<p><pre><code> pipx install llm # or brew install llm or uv tool install llm
llm install llm-cerebras
llm keys set cerebras
# paste key here
</code></pre>
Then you can run lightning fast prompts like this:<p><pre><code> llm -m cerebras-llama3.1-70b 'an epic tail of a walrus pirate'
</code></pre>
Here's a video of that running, it's very speedy: <a href="https://static.simonwillison.net/static/2024/cerebras-is-fast.mp4" rel="nofollow">https://static.simonwillison.net/static/2024/cerebras-is-fas...</a>
Wonder if they'll eventually release Whisper support. Groq has been great for transcribing 1hr+ calls at a significnatly lower price compared to OpenAI ($0.36/hr vs. $0.04/hr).
Cerebras really has impressed me with their technicality and their approach in the modern LLM era. I hope they do well, as I've heard they are en-route to IPO. It will be interesting to see if they can make a dent vs NVIDIA and other players in this space.
When Meta releases the quantized 70B it will give another > 2X speedup with similar accuracy: <a href="https://ai.meta.com/blog/meta-llama-quantized-lightweight-models/" rel="nofollow">https://ai.meta.com/blog/meta-llama-quantized-lightweight-mo...</a>
Damn, that's some impressive speeds.<p>At that rate it doesn't matter if the first try resulted in an unwanted answer, you'll be able to run once or twice more in a fast succession.<p>I hope their hardware stays relevant as this field continues to evolve
For those looking to easily build on top of this or other OpenAI-compatible LLM APIs -- you can have a look at Langroid[1] (I am the lead dev): you can easily switch to cerebras (or groq, or other LLMs/Providers). E.g. after installing langroid in your virtual env, and setting up CEREBRAS_API_KEY in your env or .env file, you can run a simple chat example[2] like this:<p><pre><code> python3 examples/basic/chat.py -m cerebras/llama3.1-70b
</code></pre>
Specifying the model and setting up basic chat is simple (and there are numerous other examples in the examples folder in the repo):<p><pre><code> import langroid.language_models as lm
import langroid as lr
llm_config = lm.OpenAIGPTConfig(chat_model= "cerebras/llama3.1-70b")
agent = lr.ChatAgent(
lr.ChatAgentConfig(llm=llm_config, system_message="Be helpful but concise"))
)
task = lr.Task(agent)
task.run()
</code></pre>
[1] <a href="https://github.com/langroid/langroid">https://github.com/langroid/langroid</a>
[2] <a href="https://github.com/langroid/langroid/blob/main/examples/basic/chat.py">https://github.com/langroid/langroid/blob/main/examples/basi...</a>
[3] Guide to using Langroid with non-OpenAI LLM APIs <a href="https://langroid.github.io/langroid/tutorials/local-llm-setup/" rel="nofollow">https://langroid.github.io/langroid/tutorials/local-llm-setu...</a>
Wow, software is hard! Imagine an entire company working to build an insanely huge and expensive wafer scale chip and your super smart and highly motivated machine learning engineers get 1/3 of peak performance on their first attempt. When people say NVIDIA has no moat I'm going to remember this - partly because it does show that they do, and partly because it shows that with time the moat can probably be crossed...
I wonder at what point does increasing LLM throughput only start to serve negative uses of AI. This is already 2 orders of magnitude faster than humans can read. Are there any significant legitimate uses beyond just spamming AI-generated SEO articles and fake Amazon books more quickly and cheaply?
Could someone please bring Microsoft's Bitnet into the discussion and explain how its performance relates to this announcement, if at all?<p><a href="https://github.com/microsoft/BitNet">https://github.com/microsoft/BitNet</a><p>"bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. "