Cost of self hosting Llama-3 8B-Instruct

<i>Instead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price.</i><p><i>Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.<p>Add in $1,000 to setup the rest of the rig and you have a final price of around:<p>$2,800 + $1,000 = $3,800</i><p>This whole exercise assumes that you&#x27;re using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you&#x27;re willing to buy GPU hardware from eBay, that will cost significantly less.<p>I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.

Llama-3 is one of the models provided by AWS Bedrock which offers pay as you go pricing. I&#x27;m curious how it would break down on that.<p>LLAMA 8B on Bedrock is $0.40 per 1M input tokens and $0.60 per 1M output tokens which is a lot cheaper than OpenAI models.<p>Edit: to add to that, as technical people we tend to discount the value of our own time. Bedrock and the OpenAI are both very easy to integrate with and get started. How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand &#x2F; how much is the cost of downtime if you have to wait to get a replacement part in the mail? That&#x27;s got to be part of the break-even equation.

Self hosting means hosting it yourself, not running it on Amazon. I think the distinction the author intends to make is between running something that can&#x27;t be hosted elsewhere, like ChatGPT, versus running Llama-3 yourself.<p>Overlooking that, the rest of the article feels a bit strange. Would we really have a use case where we can make use of those 157 million tokens a month? Would we really round $50 of energy cost to $100 a month? (Granted, the author didn&#x27;t include power for the computer) If we buy our own system to run, why would we need to &quot;scale your own hardware&quot;?<p>I get that this is just to give us an idea of what running something yourself would cost when comparing with services like ChatGPT, but if so, we wouldn&#x27;t be making most of the choices made here such as getting four NVIDIA Tesla T4 cards.<p>Memory is cheap, so running Llama-3 entirely on CPU is also an option. It&#x27;s slower, of course, but it&#x27;s infinitely more flexible. If I really wanted to spend a lot of time tinkering with LLMs, I&#x27;d definitely do this to figure out what I want to run before deciding on GPU hardware, then I&#x27;d get GPU hardware that best matches that, instead of the other way around.

3 year commit pricing with Jetstream + Maxtext on TPU v5e is $0.25 per million tokens.<p>On demand pricing put it at about $0.45 per million tokens.<p>Source: We use TPUs at scale at <a href="https:&#x2F;&#x2F;osmos.io" rel="nofollow">https:&#x2F;&#x2F;osmos.io</a><p>Google Next 2024 session going into detail: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=5QsM1K9ahtw" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=5QsM1K9ahtw</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;JetStream">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;JetStream</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;maxtext">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;maxtext</a>

I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing, LLM&#x27;s as a whole are becoming commoditized, and compute demand for training will drop off a cliff sooner than people expect.

Agreed with the sentiments here that this article gets a lot of the facts wrong, and I&#x27;ll add one: the cost for electricity when self-hosting is dramatically lower than the article says. The math assumes that each of the Tesla T4s will be using their full TDP (70W each) 24 hours a day, 7 days a week. In reality, GPUs throttle down to a low power state when not in use. So unless you&#x27;re conversing with your LLM literally 24 hours a day, it will be using dramatically less power. Even when actively doing inference, my GPU doesn&#x27;t quite max out its power usage.<p>Your self-hosted LLM box is going to use maybe 20-30% of the power this article suggests it will.<p>Source: I run LLMs at home on a machine I built myself.

Surprised no comments are pointing out that the analysis is pretty far off simply due to the fact that the author runs with batch size of 1. The cost being 100x - 1000x what API providers are charging should be a hint that something is seriously off, even if you expect some of these APIs to be subsidized.

No way you need $3,800 to run an 8B model. 3090 and a basic rig is enough.<p>That being said, the difference between OpenAI and AWS cost ($1 vs $17) is huge. Is OpenAI just operating at a massive loss?<p>Edit: Turns out AWS is actually cheaper if you don&#x27;t use the terrible setup in this article, see comments below.

A single synchronous request is not a good way to understand cost here unless your workload is truly singular tiny requests. Chatgpt handles many requests in parallel and this article&#x27;s 4 GPU setup certainly can handle more too.<p>It is miraculous that the cost comparison isn&#x27;t worse given how adversarial this test is.<p>Larger requests, concurrent requests, and request queueing will drastically reduce cost here.

Great mix of napkin math and proper analysis, but what strikes me most is how cheap LLM access is. For it being relatively bleeding edge, us splitting hairs on &lt; $20&#x2F;M tokens is remarkable itself, and something tech people should be thrilled about.

The T4 is a six year old card. A much better comparison would be a 3090, 4090, A10, A100, etc.

&gt;initial server cost of $3,800<p>Not following?<p>Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay. 700 for the card and another 500 for some 2nd hand basic gaming rig.<p>Plus you don&#x27;t need a 4 slot PCIE mobo. Plus it&#x27;s a gen4 pcie card (vs gen3). Plus skipping the complexity of multi-GPU. And wouldn&#x27;t be surprised if it ends up faster too (everything in one GPU tends to be much faster in my experience, plus 3090 is just organically faster 1:1)<p>Or if you&#x27;re feeling extra spicy you can do same on a 7900XTX (inference works fine on those &amp; it&#x27;s likely that there will be big optimisation gains in next months).

These costs don&#x27;t line up with my own experiments using vLLM on EKS for hosting small to medium sized models. For small (under 10B parameters) models on g5 instances, with prefix caching and an agent style workload with only 1 or a small number of turns per request, I saw on the order of tens of thousands of tokens&#x2F;second of prefill (due to my common system prompts) and around 900 tokens&#x2F;second of output.<p>I think this worked out to around $1&#x2F;million tokens of output and orders of magnitude less for input tokens, and before reserved instances or other providers were considered.

Does anyone know the impact of the prompt size in terms of throughput? If I&#x27;m only generating 10 tokens, does it matter if my initial prompt is 10 tokens or 8000 tokens? How much does it matter?

I just bought a $1099 MacBook Air M3, I get about 10 tokens&#x2F;s for a q5 quant. Doesn&#x27;t even get hot, and I can take it with me on the plane. It&#x27;s really easy to install ollama.

Until January this year I mostly used Google Colab for both LLMs and deep learning projects. In January I spent about $1800 getting Apple Silicon M2Pro 32G. When I first got it, I was only so-so happy with the models I could run. Now I am ecstatically happy with the quality of the models I can run on this hardware.<p>I sometimes use Groq Llama3 APIs (so fast!) or OpenAI APIs, but I mostly use my 32G M2 system.<p>The article calculates cost of self-hosting, but I think it is also good taking into account how happy I am self hosting on my own hardware.

I own an 8 GPU cluster that I built for super cheap &lt; $4,000. 180gb vram, 7 24gb + 1 24gb. There are tons of models that I run that&#x27;s not hosted by any provider. The only way to run it is to host myself. Furthermore, the author has 39 tokens in 6 seconds. For llama3-8b, I get almost 80 tk&#x2F;s and if parallel, can easily get up to 800 tk&#x2F;s. Most users at home infer only one at a time because they are doing chat or role play. If you are doing more serious work, you will most likely have multiple inference running at once. When working with smaller models, it&#x27;s not unusual to have 4-5 models loaded at once with multiple inference going. I have about 2tb of models downloaded, I don&#x27;t have to shuffle data back and forth to the cloud, etc. To each their own, the author&#x27;s argument is made today by many on why you should host in the cloud. Yet if you are not flush with cash and a little creative, it&#x27;s far cheaper to run your own server than in the cloud.<p>To run llama-3 8b. A new $300 3060 12gb will do, it will load fine in Q8 gguf. If you must load in fp16 and cash is a problem a $160 P40 will do. If performance is desired a used 3090 for ~$650 will do.

I agree with most of the criticisms here, and will add on one more: while it is generally true that you can’t beat “serverless” inference pricing for LLMs, production deployments often depend on fine-tuned models, for which these providers typically charge much more to host. That’s where the cost (and security, etc.) advantage for running on dedicated hardware comes in.

The energy costs in the bay area are double the reported 24c cost, so energy alone would be around $100-ish a month instead of $50-ish.

llama.cpp + llama-3-8b in Q8 run great on a single T4 machine. Cannot remember the TPS I got there, but it was much above 6 mentioned in the article.

deepinfra.com hosts Llama 3 8b for 8 cents per 1m tokens. I&#x27;m not sure it&#x27;s the cheapest but it&#x27;s pretty cheap. There may be even cheaper options.<p>(Haven&#x27;t used it in production, thinking to use it for side projects).

does aws not have lower vcpu and memory instances with multiple T4s? because with 192gbs of memory and 24 cores, you&#x27;re paying for a ton of resources you won&#x27;t be using if you&#x27;re only running inference.

This is a good way to do math. But honestly, how many products actually have 100% utilisation. I did some math a few months ago but mostly on the basis of active users, on what would be the % difference if you have 1k to 10K users&#x2F;mo. You can run this as low as $0.3K&#x2F;mo on Serverless GPUs and $0.7K&#x2F;mo on EC2.<p>The pricing is outdated now.<p>Here is the piece -<a href="https:&#x2F;&#x2F;www.inferless.com&#x2F;learn&#x2F;unraveling-gpu-inference-costs-for-llms-openai-aws-and-inferless" rel="nofollow">https:&#x2F;&#x2F;www.inferless.com&#x2F;learn&#x2F;unraveling-gpu-inference-cos...</a>

There&#x27;s also the option of platforms such as BentoML (I have no affiliation) that offer usage-based pricing so you can at least take the 100% utilization assumption off the table. I&#x27;m not sure how the price compares to EKS.<p><a href="https:&#x2F;&#x2F;www.bentoml.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.bentoml.com&#x2F;</a>

If we care about cost efficiency when running LLMs, the most important things are:<p>1. Don&#x27;t use AWS, because it&#x27;s one of the most expensive cloud providers<p>2. Use quantized models, because they offer the best output quality per money spent, regardless of the budget<p>This article, on the other hand, focuses exclusively on running an unquantized model on AWS...

This is another one of those &quot;I used this for 5 minutes and found this out&quot; naive posts which add nothing useful.<p>Check out the host LLM&#x27;s at home crowd. One app to look at is llama.cpp. Model compression is one of the first techniques to successfully run models on low capacity hardware.

There&#x27;s some dodgy maths<p>&gt;( 100 &#x2F; 157,075,200 ) * 1,000,000 = $0.000000636637738<p>Should be $0.64 so still expensive

I’ve used llama3 on my work laptop with ollama. It wrote an amazing pop song about k-nearest neighbours in the style of PJ and Duncan’s ‘Let’s Get Ready to Rhumble’ called ‘Let’s Get Ready to Classify’ For everything else it’s next to useless.

Ggml Q8 models on ollama can run on much cheaper hardware without losing much performance.

With dstack you can either utilize multiple affordable cloud GPU providers at once to get the cheapest GPU offer or also use an own cluster of on-prem servers. Dstack supports both altogether. Disclaimer: I’m a core contributor to dstack

Up until not too long ago I assumed that self-hosting an llm would come at an outrageous cost. I have a bunch of problems with LLM&#x27;s in general. The major one is that all LLMs(even openAI) will produce output which will give anyone a great sense of confidence, only to be later slapped across the face with the harsh reality-for anything involving serious reasoning, chances are the response you got was at large bullshit. The second one is that I do not entirely trust those companies with my data, be it OpenAI, Microsoft or Github or any other.<p>That said, a while ago there was this[1] thread on here which helped me snatch a brand new, unboxed p40 for peanuts. Really, the cost was 2 or 3 jars of good quality peanut butter. Sadly it&#x27;s still collecting dust since although my workstation can accommodate it, cooling is a bit of an issue - I 3D printed a bunch of hacky vents but I haven&#x27;t had the time to put it all together.<p>The reason why I went this road was phi-3, which blew me away by how powerful, yet compact it is. Again, I would not trust it with anything big, but I have been using it for sifting through a bunch of raw, unstructured text and extract data from it and it&#x27;s honestly done wonders. Overall, depending on your budget and your goal, running an llm in your home lab is a very appealing idea.<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39477848">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39477848</a>

Hetzner GPU servers at $200&#x2F;month for an RTX 4000 with 20GB seem competitive. Anyone have experience with what kind of token throughput you could get with that?

Running 13b code llama on my m1 macbook pro as I type this...

What do you use it for? What problems does it solve?

Half-OT: can I shard Llama3 and run it on multiple wasm processes?

this is not what I consider self hosting but ok<p>I would like to compare the costs vs hardware on prem, so this helps with one side of the equation

? you can run llama 3 8b with a 3060

Yeah, or you can get a gpu server with 20GB VRAM on hetzner for ~200 EUR per month. Runpod and DigitalOcean are also quite competitive on prices if you need a different GPU.<p>AWS is stupidly expensive.

Kind of a ridiculous approach, especially for this model. Use together.ai, fireworks.ai, RunPod serverless, any serverless. Or use ollama with the default quantization, will work on many home computers, including my gaming laptop which is about 5 years old.