LLaMa running at 5 tokens/second on a Pixel 6

This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.

Until it thermally throttles 40 seconds later. But yeah, it's really cool how many platforms the vanilla code in llama.cpp can be easily compiled on. And somehow I doubt they did the quantization step on the Pixel itself. My favorite was the person who did it on the rpi4. I know a guy working on getting it going on rpi3 but the ARM7/8 mixing , NEON support, and 64 bit ARM intrinsics are apparently non-trivial to convert.

Here is a thread to tweak the parameters which the model seems very sensitive to:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp&#x2F;issues&#x2F;129">https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp&#x2F;issues&#x2F;129</a>

This would be useful for predictive text. That&#x27;s exactly what LLMs are actually built for.

I&#x27;m waiting until it runs on my C64...

Did anyone get this to run on an iPhone or in a browser yet?

Does this in theory mean it should be relatively easy to port to coral TPU?

It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.

Any more details? I&#x27;m guessing they&#x27;re leveraging the NPU in the pixel?

So is this finally peak hipster coder and from this point on rust will diminish because all the cool kids start switching to zag?

would be even cooler if it employed the accelerator!<p>(unless this ggml library is doing that under the hood)<p>i assume it has unified memory, but maybe not little numbers...