Show HN: LocalScore – Local LLM Benchmark

Congrats on the effort - the local-first &#x2F; private space needs more performant AI, and AI in general needs more comparable and trustworthy benchmarks.<p>Notes: - Olama integration would be nice - Is there an anonymous federated score sharing? That way, users you approximate a model&#x27;s performance before downloading it.

Contributed scores for the M3 Ultra 512 GB unified memory: <a href="https:&#x2F;&#x2F;www.localscore.ai&#x2F;accelerator&#x2F;404" rel="nofollow">https:&#x2F;&#x2F;www.localscore.ai&#x2F;accelerator&#x2F;404</a><p>Happy to test larger models that utilize the memory capacity if helpful.

I don&#x27;t know if I should trust and run this code. If it was associated to Mozilla I would. It says it is a Mozilla Builders project, but <a href="https:&#x2F;&#x2F;builders.mozilla.org&#x2F;projects&#x2F;" rel="nofollow">https:&#x2F;&#x2F;builders.mozilla.org&#x2F;projects&#x2F;</a> does not list it. I don&#x27;t see a way to verify that localscore.ai is associated with Mozilla.

The run and&#x2F;or troubleshooting steps for Windows should probably include the note you need to install <a href="https:&#x2F;&#x2F;developer.nvidia.com&#x2F;cuda-downloads?target_os=Windows&amp;target_arch=x86_64" rel="nofollow">https:&#x2F;&#x2F;developer.nvidia.com&#x2F;cuda-downloads?target_os=Window...</a> if you have an Nvidia GPU (and probably something similar if you have an AMD GPU?). As it is right now the steps happily get you benchmarking your CPU and I&#x27;d say that might even be worth adding a &quot;Warning: The benchmark is operating in CPU only mode, press y to continue if this is intended&quot; type message to the program.<p>Edit: And for the same prompt and generated token counts it runs ~4x slower than `ollama run hf.co&#x2F;bartowski&#x2F;Qwen2.5-14B-Instruct-GGUF:Q4_K_M --verbose`. It&#x27;s possible I&#x27;m mixing up a few things there but my results also post in the same ballpark slower than others with the same GPU so it seems something is up with the application in either case.

This is super cool. I finally just upgraded my desktop and one thing I’m curious to do with it is run local models. Of course the ram is late so I’ve been googling trying to get an idea of what I could expect and there’s not much out there to compare to unless you’re running state of the art stuff.<p>I’ll make sure to run contribute my benchmark to this once my ram comes in.

Congrats on launching!<p>Stoked to have this dataset out in the open. I submitted a bunch of tests for some models I&#x27;m experimenting with on my M4 Pro. Rather paltry scores compared to having a dedicated GPU but I&#x27;m excited that running a 24B model locally is actually feasible at this point.

I&#x27;ve been waiting for something like this. Have you considered the following based on the benchmark data that&#x27;s submitted beyond the GPU?<p>1. User selects a model, size and token output speed and latency. The website generates a hardware list of components that should match the performance requirements.<p>2. User selects hardware components and the website generates a list of models that performant on that hardware.<p>3. Monetize through affiliate links the components to fund the project. Think like PC part picker.<p>I know there&#x27;s going to be some variability in the benchmarks due to the software stack, but it should give a AI enthusiasts, an educated perspective on what hardware can be relevant for their use case.

I’m curious: does this fundamentally <i>need</i> to contain an actual model, or would it be okay if it generated a synthetic model itself, full of random weights? I’m picturing downloading just, say, a 20MB file instead of the multi-gigabyte one, and…<p>Hang on, why is <a href="https:&#x2F;&#x2F;blob.localscore.ai&#x2F;localscore-0.9.2" rel="nofollow">https:&#x2F;&#x2F;blob.localscore.ai&#x2F;localscore-0.9.2</a> 380MB? I remember llamafile being only a few megabytes. From <a href="https:&#x2F;&#x2F;github.com&#x2F;Mozilla-Ocho&#x2F;llamafile&#x2F;releases">https:&#x2F;&#x2F;github.com&#x2F;Mozilla-Ocho&#x2F;llamafile&#x2F;releases</a>, looks like it steadily grew from adding support for GPUs on more platforms, up to 28.5MiB¹ in 0.8.12, and then rocketed up to 230MiB in 0.8.13:<p>&gt; <i>The llamafile executable size is increased from 30mb to 200mb by this release. This is caused by <a href="https:&#x2F;&#x2F;github.com&#x2F;ggml-org&#x2F;llama.cpp&#x2F;issues&#x2F;7156">https:&#x2F;&#x2F;github.com&#x2F;ggml-org&#x2F;llama.cpp&#x2F;issues&#x2F;7156</a>. We&#x27;re already employing some workarounds to minimize the impact of upstream development contributions on binary size, and we&#x27;re aiming to find more in the near future.</i><p>Ah, of course, <i>CUDA</i>. Honestly I might be more surprised that it’s <i>only</i> this big. That monstrosity will happily consume a dozen gigabytes of disk space.<p>llamafile-0.9.0 was still 231MiB, then llamafile-0.9.1 was 391MiB, now llamafile-0.9.2 is 293MiB. Fluctuating all over the place, but growing a lot. And localscore-0.9.2 is 363MiB. Why 70MiB extra on top of llamafile-0.9.2? I’m curious, but not curious enough to investigate concretely.<p>Well, this became a grumble about bloat, but I’d still like to know whether it would be feasible to ship a smaller localscore that would synthesise a suitable model, according to the size required, at runtime.<p>—⁂—<p>¹ Eww, GitHub is using the “MB” suffix for its file sizes, but they’re actually mebibytes (2²⁰ bytes, 1048576 bytes, MiB). I thought we’d basically settled on returning the M&#x2F;mega- prefix to SI with its traditional 10⁶ definition, at least for file sizes, ten or fifteen years ago.

Really awesome project!<p>Clicking on GPU is a nice simple visualization. I was thinking maybe try to put that type of visual representation intuitively accessible immediately on the landing page.<p>cpubenchmark.net could he an example technique of drawing the site visitor into the paradigm.

This is great, congrats for launching!<p>A couple of ideas .. I would like to benchmark a remote headless server, as well as different methods to run the LLM (vllm vs tgi vs llama.cpp ...) on my local machine, and in this case llamafile is quite limiting. Connecting over an OpenAI-like API instead would be great!

This looks super useful, especially with so many folks experimenting with local LLMs now. Curious how well it handles edge devices. Will give it a try!

Congrats, really cool tool!<p>Any plans to add a leaderboard, or more stats groupped by hardware?

Why choose a combibation of llama and qwen, when you could have used just qwen models with more permissive license?

Awesome stuff, congrats on launching!