Tinygrad: A simple and powerful neural network framework

&gt; It compiles a custom kernel for every operation, allowing extreme shape specialization.<p>This doesn&#x27;t matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they&#x27;re dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won&#x27;t help much.<p>&gt; All tensors are lazy, so it can aggressively fuse operations.<p>This matters. PyTorch teams are trying to implement that now (they have LazyTensor, AITemplate, TorchDynamo), but I&#x27;m not sure of the status (it&#x27;s been tried repeatedly).<p>&gt; The backend is 10x+ simpler, meaning optimizing one kernel makes everything fast.<p>The first part of that sentence matters, the second part doesn&#x27;t. Kernels are already fast and their reuse outside of being fused into each other (which you need a full linear algebra compiler to do) isn&#x27;t very high. If you make sum fast, you have not made matrix multiplication fast even though MM has a sum in it. It just isn&#x27;t that easy to compose operations and still hit 80+% of hardware efficiency.<p>But it is easier to iterate fast and build a seamless lazy compiler if your backend is simple. You can pattern match more easily and ensure you handle edge cases without insanely complicated things like alias analysis (which PyTorch has to do).

<p><pre><code> &gt; It&#x27;s extremely simple, and breaks down the most complex networks into 4 OpTypes: &gt; &gt; - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... &gt; - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... &gt; - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX &gt; - MovementOps operate on one tensor and move the data around, copy-free with ShapeTracker. RESHAPE, PERMUTE, EXPAND, etc... &gt; &gt; But how...where are your CONVs and MATMULs? Read the code to solve this mystery. </code></pre> Ok, I was curious, so I read the code. The answer is that it represents a MATMUL as a 1x1 CONV. And it lied about CONV, which is a ProcessingOps.CONV and explicitly represented and implemented: <a href="https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;c0050fab8ff0bc667e40da11980f4ac4c21affda&#x2F;tinygrad&#x2F;llops&#x2F;ops_cpu.py#L40" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;c0050fab8ff0bc667e40...</a> Quite the letdown of figuring out this &#x27;mystery&#x27;.

I must say they gained instant credibility with the minimalistic website given how fast it loaded.<p>Code looks simple and easy to follow, and I love how the comments are constantly mentioning hardware characteristics, making maxing the hardware the goal. It seems that it’s trying to achieve this by jitting optimal code for the operations at hand rather than hand-optimizing kernels, and betting that the small number of operations will make tuning the codegen tractable.<p>I haven’t kept up much with what’s happening in ML, but at least in the realm of columnar database engines, interpreting a series of hand-optimized kernels seems to be the dominant approach over compiling a vectorized query plan. Are compilers good enough at optimizing ML operations that specializing on input shape makes a difference over hand-tuned kernels?

I love those tiny DNN frameworks, some examples that I studied in the past (I still use PyTorch for work related projects) :<p>thinc.by the creators of spaCy <a href="https:&#x2F;&#x2F;github.com&#x2F;explosion&#x2F;thinc" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;explosion&#x2F;thinc</a><p>nnabla by Sony <a href="https:&#x2F;&#x2F;github.com&#x2F;sony&#x2F;nnabla" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;sony&#x2F;nnabla</a><p>LibNC by Fabrice Bellard <a href="https:&#x2F;&#x2F;bellard.org&#x2F;libnc&#x2F;" rel="nofollow">https:&#x2F;&#x2F;bellard.org&#x2F;libnc&#x2F;</a><p>Dlib dnn <a href="http:&#x2F;&#x2F;dlib.net&#x2F;ml.html#add_layer" rel="nofollow">http:&#x2F;&#x2F;dlib.net&#x2F;ml.html#add_layer</a>

I love this website. Their style tag literally is:<p><pre><code> &lt;style&gt; body { font-family:&#x27;Lucida Console&#x27;, monospace } &lt;&#x2F;style&gt; </code></pre> Also look like a very cool project.

There&#x27;s an interesting roadmap in the &quot;cherry&quot; folder of the git repo[0]. It begins by bringing up a design on FPGA and ends with selling the company for $1B+ by building accelerator cards to compete with NVIDIA:<p><pre><code> Cherry Three (5nm tapeout) ===== * Support DMA over PCI-E 4.0. 32 GB&#x2F;s * 16 cores * 8M elements in on board RAM of each core (288 MB SRAM on chip) * Shared ~16GB GDDR6 between cores. Something like 512 GB&#x2F;s * 16x 32x32x32 matmul = 32768 mults * 1 PFLOP @ 1 ghz (finally, a petaflop chip) * Target 300W, power savings from process shrink * This card should be on par with a DGX A100 and sell for $2000 * At this point, we have won. * The core Verilog is open source, all the ASIC speed tricks are not. * Cherry will dominate the market for years to come, and will be in every cloud. * Sell the company for $1B+ to anyone but NVIDIA </code></pre> [0] <a href="https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;master&#x2F;accel&#x2F;cherry&#x2F;README#L22-L68" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;master&#x2F;accel&#x2F;cherry&#x2F;...</a>

As it was recently discussed at length here on HN [0] (401 comments), George Hotz (the lead of tinygrad) is taking time off his self-driving startup comma.ai [1]. Curious if this would help or hurt tinygrad progress.<p>[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=33406790" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=33406790</a><p>[1] <a href="https:&#x2F;&#x2F;comma.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;comma.ai&#x2F;</a>

Of course, the stable diffusion tie-in is not to be missed!<p><a href="https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;master&#x2F;examples&#x2F;stable_diffusion.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;master&#x2F;examples&#x2F;stab...</a>

How is it compared to JAX? After TensorFlow and PyTorch, JAX seems very simple, basically an accelerated numpy with just a few additional useful features like automatic differentiation, vectorization and jit-compilation. In terms of API I don&#x27;t see how you can go any simpler.

Can someone from the ML crowd ELI5 to me what tinygrad does, how it plugs into an ML pipeline and what it&#x27;s use cases are?

“Almost 9k stars” is actually 7.3k stars…<p>But otherwise very cool project :)

It&#x27;s funny that geohot&#x2F;tinygrad chooses to not meet the PEP8 standards [0] just to stay on brand (&lt;1000 lines). Black [1] or any other python autoformatter would probably 2x the lines of code.<p>[0] <a href="https:&#x2F;&#x2F;peps.python.org&#x2F;pep-0008&#x2F;" rel="nofollow">https:&#x2F;&#x2F;peps.python.org&#x2F;pep-0008&#x2F;</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;psf&#x2F;black" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;psf&#x2F;black</a>

tinygrad core is over 1000 loc now[1]. If anyone was looking for a fun weekend project :)<p><a href="https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;master&#x2F;.github&#x2F;workflows&#x2F;test.yml#L9" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;geohot&#x2F;tinygrad&#x2F;blob&#x2F;master&#x2F;.github&#x2F;workf...</a>

I understand that the Python code is mostly driving faster low-level code, but I wonder how much time is effectively wasted by not using a lower-level language.<p>From my experience with game engines, it often turns out to be a bad idea (for performance and maintainability) to mix C&#x2F;C++ and Lua or C#.

Geohot at it again, this guy nails everything.

&gt; almost 9000 GitHub stars<p>I wouldn&#x27;t say that 7500 stars is almost 9000 stars ;)

If you care exclusively about numerical stability and performance, why _this_ set of operators (e.g., there’re plenty of good reasons to include expm1 or log1p and certainly trigonometric functions)? It’d be an interesting research problem to measure and identify the minimal subset of operators (and I suspect it’d look differently than what you’d expect from an FPU).<p>If you care exclusively about minimalism, why not limit yourself to the Meijer-G function (or some other general-purpose alternative)?

The code is very easy to read. Doesn&#x27;t seem like there&#x27;s data&#x2F;model parallelism support for training, which will be important for real-world use.

How does this compare on embedded systems for performance? For example PyTorch vs tinygrad, or Darknet vs tinygrad?

Does anyone know of a neural network that is written in C and GLSL and that runs on normal OpenGL?

It was ok as an educational tool, but now they don&#x27;t count GPU implementation in 1000 lines, so it is not small. Considering the code style it is closer to 20k+ lines when formatted and GPU code included.<p>It also doesn&#x27;t support bfloat16 so is doomed to be 2x slower.

I believe neural networks are over hyped sometimes.<p>They are not always the best tool for the job. There are lots of other ML techniques such as SVM, naive Bayes, k-nearest neighbor, decision tree, logistic regression, random forest etc. nobody is using because they lack the hype factor.<p>If something lacks some keywords like neural network, deep learning, reinforced learning, than it is deemed not cool.

No Bible quotes? I&#x27;m disappointed...

If anybody is dealing with procrastination watch George Hotz live streaming 10h straight working on this library [1][2]. Does he take some supplements to do this? There is even 19.5h stream [3].<p>Actually I have local obs setup to record myself, just instead of streaming I do recordings for my own inspection. Important part is to do the inspection after. It works wonders.<p>[1] <a href="https:&#x2F;&#x2F;youtu.be&#x2F;GXy5eVwnL_Q" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;GXy5eVwnL_Q</a><p>[2] <a href="https:&#x2F;&#x2F;m.youtube.com&#x2F;watch?v=Cb2KwcnDKrk" rel="nofollow">https:&#x2F;&#x2F;m.youtube.com&#x2F;watch?v=Cb2KwcnDKrk</a><p>[3] no joke, 19.5h stream <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=xc0jGZYFQLQ" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=xc0jGZYFQLQ</a>

4 OpCodes, I think Geohot is taking a cue from his favorite intellectual&#x27;s Curtis Yavin&#x27;s Urbit project

I think posts like this are only getting upvotes because George Hotz owns the project. I do see value in simple code, but the constraint of 1000 LOC makes little sense to me, especially when the code is formatted poorly.<p>This will get downvoted, but reading the comments here I dont understand the (cult&#x2F;respect) for him. Siding with the most successful CTF-team ever (PPP) he won defcon two times. He made a startup with funding that makes a cool &#x27;niche&#x27; product.<p>I just think a guy like Chris Lattner or Dave Cutler who made so much impact on real computing deserve so much more respect, but I guess that the norm here is to admire this guy.