Run Llama 13B with a 6GB graphics card

On my system, using `-ngl 22` (running 22 layers on the GPU) cuts wall clock time by ~60%.<p>My system:<p>GPU: NVidia RTX 2070S (8GB VRAM)<p>CPU: AMD Ryzen 5 3600 (16GB VRAM)<p>Here&#x27;s the performance difference I see:<p>CPU only (.&#x2F;main -t 12)<p><pre><code> llama_print_timings: load time = 15459.43 ms llama_print_timings: sample time = 23.64 ms &#x2F; 38 runs ( 0.62 ms per token) llama_print_timings: prompt eval time = 9338.10 ms &#x2F; 356 tokens ( 26.23 ms per token) llama_print_timings: eval time = 31700.73 ms &#x2F; 37 runs ( 856.78 ms per token) llama_print_timings: total time = 47192.68 ms </code></pre> GPU (.&#x2F;main -t 12 -ngl 22)<p><pre><code> llama_print_timings: load time = 10285.15 ms llama_print_timings: sample time = 21.60 ms &#x2F; 35 runs ( 0.62 ms per token) llama_print_timings: prompt eval time = 3889.65 ms &#x2F; 356 tokens ( 10.93 ms per token) llama_print_timings: eval time = 8126.90 ms &#x2F; 34 runs ( 239.03 ms per token) llama_print_timings: total time = 18441.22 ms</code></pre>

This is cool but are people actually getting stuff done with these models? I&#x27;m enthusiastic about their potential too but after playing with it for a day I&#x27;m at a loss for what to use it for anymore at this point

Why does AMD or Intel not release a medium performant GPU with minimum 128gb of memory for a good consumer price. These models require lots of memory to &#x27;single&#x27; pass an operation. Throughput could be bit slower. A 1080 Nvidia with 256gb of memory would run all these models fast right? Or am I forgetting something here.

From skimming, it looks like this approach requires CUDA and thus is Nvidia only.<p>Anyone have a recommended guide for AMD &#x2F; Intel GPUs? I gather the 4 bit quantization is the special sauce for CUDA, but I’d guess there’d be something comparable for not-CUDA?

What do you use to host these models (like Vicuna, Dolly etc) on your own server and expose them using HTTP REST API? Is there an Heroku-like for LLM models?<p>I am looking for an open source models to do text summarization. Open AI is too expensive for my use case because I need to pass lots of tokens.

This update is pretty exciting, I’m gonna try running a large model (65B) with a 3090. I have ran a ton of local LLM but the hardest part is finding out the prompt structure. I wish there is some sort of centralized data base that explains it.

See also:<p><a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;LocalLLaMA&#x2F;comments&#x2F;13fnyah&#x2F;you_guys_are_missing_out_on_gpt4x_vicuna&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;LocalLLaMA&#x2F;comments&#x2F;13fnyah&#x2F;you_guy...</a><p><a href="https:&#x2F;&#x2F;chat.lmsys.org&#x2F;?arena" rel="nofollow">https:&#x2F;&#x2F;chat.lmsys.org&#x2F;?arena</a> (Click &#x27;leaderboard&#x27;)

No need to convert models, 4bit LLaMA versions for GGML v2 available here:<p><a href="https:&#x2F;&#x2F;huggingface.co&#x2F;gotzmann&#x2F;LLaMA-GGML-v2&#x2F;tree&#x2F;main" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;gotzmann&#x2F;LLaMA-GGML-v2&#x2F;tree&#x2F;main</a>

I ran the 7B Vicuna (ggml-vic7b-q4_0.bin) on a 2017 MacBook Air (8GB RAM) with llama.cpp.<p>Worked OK for me with the default context size. 2048, like you see in most examples was too slow for my taste.

Could someone please share a good resource for building a machine from scratch, for doing simple-ish training and running open-source models like Llama? I&#x27;d love to run some of these and even train them from scratch, and I&#x27;d love to use that as an excuse to drop $5k on a new machine...<p>Would love to run a bunch of models on the machine without dripping $$ to OpenAI, Modal or other providers...

PSA:<p>If you&#x27;re using oobabooga&#x2F;text-generation-webui then you need to:<p>1. Re-install llama-cpp-python with support for CUBLAS:<p><pre><code> CMAKE_ARGS=&quot;-DLLAMA_CUBLAS=on&quot; FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir --force-reinstall </code></pre> 2. Launch the web UI with the --n-gpu-layers flag, e.g.<p><pre><code> python server.py --model gpt4-x-vicuna-13B.ggml.q5_1.bin --n-gpu-layers 24</code></pre>

I wish this used the webgpu c++ library instead, then it could be used in any GPU hardware.

Instructions are a bit rough. The Micromamba thing doesn’t work, doesn’t say how to install it… you have to clone llama.cpp too

What is the state of the art on evaluating the accuracy of these models? Is there some equivalent to an “end to end test”?<p>It feels somewhat recursive since the input and output are natural language and so you would need another LLM to evaluate whether the model answered a prompt correctly.

How about reloading parts of the model as the inference progresses instead of splitting it into GPU&#x2F;CPU parts? Reloading would be memory-limited to the largest intermediate tensor cut.

I&#x27;ve only ever been a consumer of ChatGPT&#x2F;Bard. Never set up any LLM stuff locally, but the idea is appealing to me.<p>I have a ThinkStation P620 w&#x2F;ThreadRipper Pro 3945WX (12c24t) with a GTX 1070 (and a second 1070 I could put in there) and there&#x27;s 512GB of RAM on the box.<p>Does this need to be bare metal, or can it run in VM?<p>I&#x27;m currently running RHEL 9.2 w&#x2F;KVM (as a VM host) with light usage so far.

If I really want to do some playing around in this area, would it be good to get a RTX 4000 SFF which has 20 GB of VRAM but is a low-power card, which I want as it would be running 24&#x2F;7 and energy prices are pretty bad in Germany, or would it make more sense to buy an Apple product with some M2 chip which apparently is good for these tasks as it shares CPU and GPU memory?

Why can&#x27;t these models run on the GPU while also using CPU RAM for the storage? That way people will performant-but-memory-starved GPUs can still utilize the better performance of the GPU calculation while also having enough RAM to store the model? I know it is possible to provide system RAM-backed GPU objects.

How long before it runs on a 4 gig card?

How many &quot;B&quot; (billions of parameters) is ChatGPT GPT-4?

This only uses llama correct? So the output should be the same as if you were only using llama.cpp. Am I the only one who doesn&#x27;t get nearly the same quality of output using a quantized model compared to GPU? Some models I tried get astounding results when running on a GPU, but create only &quot;garbage&quot; when running on a CPU. Even when not quantized down to 4bit llama.cpp just doesn&#x27;t compare for me. Am I alone with this?

Has anyone tried running encryption algorithms through these models? I wonder if it could be trained to decrypt.

What is HN’s fascination with these toy models that produce low quality, completely unusable output?<p>Is there a use case for them I’m missing?<p>Additionally, don’t they all have fairly restrictive licenses?

Is there a way to run any of these with only 4GB of VRAM?

wouldn&#x27;t i be faster to use ram as a swap for vram?

or like download oobabooga&#x2F;text-generation-webui, any prequantized variant, and be done.

[deleted]

The future is this - these models will be able to run on smaller and smaller hardware eventually being able to run on your phone, watch or embedded devices. The revolution is here and is inevitable. Similar to how computers evolved. We are still lucky that these models have no consciousness, still. Once they gain consciousness, that will mark the appearance of a new species (superior to us if anything). Also, luckily, they have no physical bodies and cannot replicate, so far...