Deep Neural Networks from Scratch in Zig

I&#x27;ve been working on mix of ML performance and abstraction for a while and this is a breath of fresh air. Zig is truly a fascinating language, and the semantics revealed in this post are surprisingly powerful.<p>&quot;By hard coding the forward and backward methods at comptime we have some more comfort with the general correctness and expected errors we would receive if we passed in incorrectly shaped data at runtime.&quot;<p>This solves the issues of shape checking incredibly cleanly. Python libraries have been struggling with this for a decade. It seems like you could also extending comptime usage to also calculate allocations ahead of time.<p>Honestly, this whole thing makes me want to invest quite a bit of time into using Zig. Great post!

This is interesting, thanks. I know this is just a demo, but I&#x27;m convinced we&#x27;re going to see a swing back to simple, purpose written NNs for many simple applications, when the alternative is bringing in a couple GB of python and cuda libraries, which is serious overkill for something on the scale of MNIST (which many real problems are).<p>That said, I&#x27;m curious about how well the compiler can optimize matrix operations, in Zig or other, say C or Rust, and when it&#x27;s worth linking in BLAS or mkl some other library. I wonder if there is a sweet spot where it&#x27;s worth doing.

I didn&#x27;t investigate this more deeply , but I think if you used an arena allocator on each iteration of the loop, and simply freed the whole allocator after each iteration, you might get much better performance.<p>EDIT: the MNIST link requires authentication: <a href="http:&#x2F;&#x2F;yann.lecun.com&#x2F;exdb&#x2F;mnist&#x2F;" rel="nofollow">http:&#x2F;&#x2F;yann.lecun.com&#x2F;exdb&#x2F;mnist&#x2F;</a> How can we get access to the test run at the end?

Lacking a Linear Algebra &#x2F; Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided:<p><pre><code> while (i &lt; I * O): (i += 1) { self.weights[i] -= 0.01 * grads[i]; } </code></pre> What are the options for vector &#x2F; matrix &#x2F; tensor operations in zig?

It would be cool too see a port of llama.cpp and ggml to Zig<p><a href="https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp">https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;llama.cpp</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;ggml">https:&#x2F;&#x2F;github.com&#x2F;ggerganov&#x2F;ggml</a>

Awesome, thanks for the write up! Profiling would be a great addition to see.

I want DNN from scratch in Scratch.

&gt; I would guess the constant memory allocation and frees in the training loop are the bottleneck<p>No, the bottleneck would be not utilizing the idling GPU.

Machine learning feels a lot like cpu fabrication in that a homegrown solution is almost certainly going to be inferior to just taking whats on the market and configuring it to fit your needs. If you aren&#x27;t going to specialize in this field, is there a point in learning how to roll your own ML anymore?