For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year.<p>(If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to work, you can use the Vulkan build as a (less performant) fallback).<p>Having more options can't hurt, but ROCm 5.4.2 is almost 2 years old, and things have come a long way since then, so I'm curious about this being published freshly today, in October 2024.<p>BTW, I recently went through and updated my compatibility doc (focused on RDNA3) w/ ROCm 6.2 for those interested. A lot has changed just in the past few months (upstream bitsandbytes, upstream xformers, and Triton-based Flash Attention): <a href="https://llm-tracker.info/howto/AMD-GPUs" rel="nofollow">https://llm-tracker.info/howto/AMD-GPUs</a>
The rise of generated slop ml libraries is staggering.<p>This library is 50% print statements. And where it does branch, it doesn't even need to.<p>Defines two environment variables and sets two flags on torch.
It seems to use an old, 2 year old version of ROCm (5.4.2) which I'm doubtful would support my RX 7900 XTX. I personally found it easiest to just use the latest `rocm/pytorch` image and run what I need from there
On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards:<p><pre><code> # install dependencies
sudo apt -y update
sudo apt -y upgrade
sudo apt -y install git wget hipcc libhipblas-dev librocblas-dev cmake build-essential
# ensure you have permissions by adding yourself to the video and render groups
sudo usermod -aG video,render $USER
# log out and then log back in to apply the group changes
# you can run `rocminfo` and look for your GPU in the output to check everything is working thus far
# download a model, build llama.cpp, and run it
wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b3267
HIPCXX=clang-17 cmake -H. -Bbuild -DGGML_HIPBLAS=ON -DCMAKE_HIP_ARCHITECTURES="gfx803;gfx900;gfx906;gfx908;gfx90a;gfx1010;gfx1030;gfx1100;gfx1101;gfx1102" -DCMAKE_BUILD_TYPE=Release
make -j16 -C build
build/bin/llama-cli -ngl 32 --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -m ../dolphin-2.2.1-mistral-7b.Q5_K_M.gguf --prompt "Once upon a time"
</code></pre>
I'd suggest RDNA 3, MI200 and MI300 users should probably use the AMD-provided ROCm packages for improved performance. Users that need PyTorch should also use the AMD-provided ROCm packages, as PyTorch has some dependencies that are not available from the system packages. Still, you can't beat the ease of installation or the compatibility with older hardware provided by the OS packages.<p>¹ <a href="https://lists.debian.org/debian-ai/2024/07/msg00002.html" rel="nofollow">https://lists.debian.org/debian-ai/2024/07/msg00002.html</a>
² Not including MI300 because that released too close to the Ubuntu 24.04 launch.
³ Pre-Vega architectures might work, but have known bugs for some applications.
⁴ Vega and RDNA 2 APUs might work with Linux 6.10+ installed. I'm in the process of testing that.
⁵ The version of rocBLAS that comes with Ubuntu 24.04 is a bit old and therefore lacks some optimizations for RDNA 3. It's also missing some MI200 optimizations.
It has been like 8 months since I got Ryzen 8700G with NPU just for the purpose of inferencing NN, and so far only acceleration I'm getting is through vulkan on iGPU, not NPU (I'm using Linux only). On the bright side, with 64GB of RAM had no isues with trying models over 32GB. Kudos to llama.cpp for supporting vulkan backend!
So, this is all I needed to add to NixOS workstation:<p><pre><code> hardware.graphics.enable = true;
services.ollama = {
enable = true;
acceleration = "rocm";
environmentVariables = {
ROC_ENABLE_PRE_VEGA = "1";
HSA_OVERRIDE_GFX_VERSION = "11.0.0";
};
};</code></pre>
I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile.<p><pre><code> sudo apt install rocm
Summary:
Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0
Download size: 2,369 MB / 2,371 MB
Space needed: 35.7 GB / 822 GB available
</code></pre>
I don't understand how 36 GB can be justified for what amounts to a GPU driver.
This seems to be some AI generated wrapper around a wrapper of a wrapper.<p>> # Other AMD-specific optimizations can be added here<p>> # For example, you might want to set specific flags or use AMD-optimized libraries<p>What are we doing here, then?
What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image.<p>"Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.
If you're interested in how much the AMD graphics cards cost compared to the NVidia ones, I have <a href="https://gpuquicklist.com/" rel="nofollow">https://gpuquicklist.com/</a> which gives you a quick table view of lowest prices available on Amazon that I can find. </ selfpromotion>
is anyone using the new HX370 based laptops for any LLM work ? i mean the ipex-llm libraries of Intel's new Lunar Lake is already supporting Llama 3.2 (<a href="https://www.intel.com/content/www/us/en/developer/articles/technical/intel-ai-solutions-support-the-new-llama-3-2-model.html" rel="nofollow">https://www.intel.com/content/www/us/en/developer/articles/t...</a>), but AMD's new Zen5 chips dont seem to be much active here.
Why ROCm 5.4, and not the latest (6.2)?<p><a href="https://github.com/slashml/amd_inference/blob/main/Dockerfile#L2">https://github.com/slashml/amd_inference/blob/main/Dockerfil...</a><p>Also looks like the Docker image provided by this project doesn't successfully build: <a href="https://github.com/slashml/amd_inference/issues/2">https://github.com/slashml/amd_inference/issues/2</a>
I'm all for having more open source projects, but I do not see how it can be useful in this ecosystem, especially for people with newer AMD GPUs (not supported in this project) which are already supported in most popular projects?