Groq surpasses 1,200 tokens/sec with Llama 3 8B

Groq is an insane company. SambaNova (discussed yesterday[0]) is also very promising. However, what I really want to see is local AI accelerator chips a la Tenstorrent Grayskull that can boost local generation to hundreds of tokens per second while being more efficient than GPUs.<p>[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40508797">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40508797</a>

Is groq related to Twitter&#x27;s grok or is that just a very unfortunate naming coincidence?

When reading Hacker News you develop a signal&#x2F;noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration.<p>My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds&#x2F;thousands of tokens per second?? Mega doubt.<p>But... then I tried their demo, and... yeah, it&#x27;s that good. Such an amazing company of talented individuals.

They&#x27;re not responsive to my questions on Twitter, so I&#x27;m asking here:<p><pre><code> When will Groq support a real API (not experimental beta preview)? When will Groq support logprobs?! When will Groq actually tell us what their rate limit is?! </code></pre> Until these aren&#x27;t answered, many of us can&#x27;t actually build on Groq.<p>Edit: It seems I&#x27;m getting downvoted by Groq employees...