Hi folks, I've done some cpu compilers for x86, RISC-V, LLVM (mostly for fun and some for profit). I'm eager to learn more about GPU/TPU compilers and have been looking into Triton and XLA. What resources would be useful to learn HPC and/or gpu compilers in depth (or any adjacent areas)?<p>Any such books, courses etc will be much appreciated.
Newer editions of Computer Organization and Design: The Hardware Software Interface covers GPUs [1]<p>Multiflow still has some relevant ideas [2]<p>Programming on Parallel Machines: GPU, Multicore, Clusters and More. Gives you a look at some of the issues [3]<p>SPIRV-VM is a virtual machine for executing SPIR-V shaders [4]<p>NyuziRaster: Optimizing Rasterizer Performance and Energy in the Nyuzi Open Source GPU [5]<p>Ocelot is a modular dynamic compilation framework for heterogeneous systems, providing various backend targets for CUDA programs and analysis modules for the PTX virtual instruction set. [6]<p>glslang is the Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.<p>[1]: <a href="https://www.goodreads.com/book/show/83895.Computer_Organization_Design" rel="nofollow">https://www.goodreads.com/book/show/83895.Computer_Organizat...</a><p>[2]: <a href="https://en.wikipedia.org/wiki/Multiflow" rel="nofollow">https://en.wikipedia.org/wiki/Multiflow</a><p>[3]: <a href="http://heather.cs.ucdavis.edu/parprocbook" rel="nofollow">http://heather.cs.ucdavis.edu/parprocbook</a><p>[4]: <a href="https://github.com/dfranx/SPIRV-VM">https://github.com/dfranx/SPIRV-VM</a><p>[5]: <a href="https://www.cs.binghamton.edu/~millerti/nyuziraster.pdf" rel="nofollow">https://www.cs.binghamton.edu/~millerti/nyuziraster.pdf</a><p>[6]:<a href="https://code.google.com/archive/p/gpuocelot/" rel="nofollow">https://code.google.com/archive/p/gpuocelot/</a><p>[7]: <a href="https://github.com/KhronosGroup/glslang">https://github.com/KhronosGroup/glslang</a>
If you're already familiar with compilers in the abstract, start by implementing some high-level solutions leveraging a GPU, some low-level performance-optimized kernels, and build up a bit of intuition. The high-level code depends on your goals, but for low-level code maybe try optimizing something equivalent to a binary tree (both with leaves smaller and larger than 4kb), something benefiting from operator fusion (e.g., a matmul followed by an element-wise exponential), and something benefiting from deeply understanding the memory hierarchy (e.g., multiplying two very large square matrices, and also inner/outer producting two very narrow matrices).<p>From there, hopefully you'll have the intuition to actually evaluate whether a given resource being recommended here is any good.
Here is a Phd thesis for a GPU programming language that compiles on the gpu itself:<p><a href="https://scholarworks.iu.edu/dspace/items/3ab772c9-92c9-4f59-bd95-40aff99e8c7a" rel="nofollow">https://scholarworks.iu.edu/dspace/items/3ab772c9-92c9-4f59-...</a>
For Intel you can just look into sources: <a href="https://github.com/intel/intel-graphics-compiler">https://github.com/intel/intel-graphics-compiler</a> ;)
Also AMDGPU and NVPTX targets in LLVM might be interesting.
Fair warning that the dominant model in GPU compilers is to use an ISA that looks like the GPUs did long ago. Expect to see i32 used to represent a machine vector of ints where you might reasonably expect <32 x i32>. There are actual i32 scalar registers as well, which are cheap to branch on, that are also represented in IR as i32. That is, the same as the vector. There are intrinsics that sometimes distinguish.<p>This makes a rather spectacular mess of the tooling. Instead of localising the cuda semantics in clang, we scatter it throughout the entire compiler pipeline, where it does especially nasty things to register allocation and generally obstructs non-cuda programming models. It's remarkably difficult to persuade GPU people that this is a bad thing.<p>Also the GPU programming languages use very large compiler runtimes to do a degree of papering over the CPU-host GPU-target assumption that also dates from long ago, so expect to find a lot of complexity in multiple libraries acting somewhat like compiler-rt. Those are optional in reality but the compiler usually emits a lot of symbols that resolve to various vendor libraries.
The Deep Learning Compiler: A Comprehensive Survey (2020) is in fact a comprehensive-as-of-2020 but now somewhat dated survey of the field. Unfortunately, I am not aware of better overall survey more up to date.<p><a href="https://arxiv.org/abs/2002.03794" rel="nofollow">https://arxiv.org/abs/2002.03794</a>
<a href="https://theartofhpc.com/" rel="nofollow">https://theartofhpc.com/</a><p>Great book series on the subject of HPC. Not sure if it actually touches GPU. Great material anyway. BONUS: it's free!
Staged FP in Spiral: <a href="https://www.youtube.com/playlist?list=PL04PGV4cTuIVP50-B_1scXUUMn8qEBbSs" rel="nofollow">https://www.youtube.com/playlist?list=PL04PGV4cTuIVP50-B_1sc...</a><p>Some of the stuff in this playlist might be relevant to you, though it is mostly about programming GPUs in a functional language that compiles to Cuda. The author (me) sometimes works on the language during the video, either fixing bugs or adding new features.
For NVIDIA,<p>1. play around with the NVPTX LLVM backend and/or try compiling CUDA with Clang,<p>2. get familiar with the PTX ISA,<p>3. play around with ptxas + nvdisasm.
Faith Ekstrand has an impressive track record of compiler work and has written a few blog posts[1], [1a]. Her Mastodon[2] is also worth a follow.<p>SPIR-V is important in the compute shader space, especially because DXIL and Metal's AIR are similar. I'm going to link three articles critical of SPIR-V: [3], [4], [5].<p>WebGPU [6] is interesting for a number of reasons, largely because they're trying to actually nail down the semantics, and also make it safe (see the uniformity analysis [7] in particular, which is a very "compiler" approach to a GPU-specific problem). Both Tint and naga projects are open source, with lots of high quality discussion in the issue trackers.<p>Shader languages suck, and we really need a good one. Promising approaches are Circle [8] (which is C++ based, very advanced but not open source), and Slang [9] (an evolution of HLSL). The Vcc work (also related to [4]) is worth studying.<p>Best of luck! This is a fascinating, if frustrating, space, and there's lots of room to improve things.<p>[1]: <a href="https://www.gfxstrand.net/faith/blog/" rel="nofollow">https://www.gfxstrand.net/faith/blog/</a><p>[1a]: <a href="https://www.collabora.com/news-and-blog/blog/2024/04/25/re-converging-control-flow-on-nvidia-gpus/" rel="nofollow">https://www.collabora.com/news-and-blog/blog/2024/04/25/re-c...</a><p>[2]: <a href="https://mastodon.gamedev.place/@gfxstrand" rel="nofollow">https://mastodon.gamedev.place/@gfxstrand</a><p>[3]: <a href="https://kvark.github.io/spirv/2021/05/01/spirv-horrors.html" rel="nofollow">https://kvark.github.io/spirv/2021/05/01/spirv-horrors.html</a><p>[4]: <a href="https://xol.io/blah/the-trouble-with-spirv/" rel="nofollow">https://xol.io/blah/the-trouble-with-spirv/</a><p>[5]: <a href="https://themaister.net/blog/2022/08/21/my-personal-hell-of-translating-dxil-to-spir-v-finale/" rel="nofollow">https://themaister.net/blog/2022/08/21/my-personal-hell-of-t...</a><p>[6]: <a href="https://github.com/gpuweb/gpuweb">https://github.com/gpuweb/gpuweb</a><p>[7]: <a href="https://www.w3.org/TR/2022/WD-WGSL-20220505/#uniformity-overview" rel="nofollow">https://www.w3.org/TR/2022/WD-WGSL-20220505/#uniformity-over...</a><p>[8]: <a href="https://www.circle-lang.org/site/index.html" rel="nofollow">https://www.circle-lang.org/site/index.html</a><p>[9]: <a href="https://github.com/shader-slang/slang">https://github.com/shader-slang/slang</a>