Also see Einops: <a href="https://github.com/arogozhnikov/einops" rel="nofollow">https://github.com/arogozhnikov/einops</a>, which uses a einsum-like notation for various tensor operations used in deep learning.<p><a href="https://einops.rocks/pytorch-examples.html" rel="nofollow">https://einops.rocks/pytorch-examples.html</a> shows how it can be used to implement various neural network architectures in a more simplified manor.
<a href="https://github.com/mcabbott/Tullio.jl" rel="nofollow">https://github.com/mcabbott/Tullio.jl</a>:<p><pre><code> - better syntax (because Julia has proper macro/metaprogramming)
- faster
- automatically works with GPU arrays.</code></pre>
If you are looking for something like this in C++, here's my attempt at implementing it: <a href="https://github.com/dsharlet/array#einstein-reductions" rel="nofollow">https://github.com/dsharlet/array#einstein-reductions</a><p>It doesn't do any automatic optimization of the loops like some of the projects linked in this thread, but, it provides all the tools needed for humans to express the code in a way that a good compiler can turn it into really good code.
I'd really like to use einsum more often, because it allows me to code my expressions the same way I derive them on pen and paper. Unfortunately, as mentioned in the article, it's slow, because it converts your formula to a for loop.<p>So usually, I rewrite my formulas into messy combinations of broadcasts, transposes and array multiplications. Is there a package or an algorithm that does this conversion automatically? It seems to be a pretty straightforward problem, at least for most expressions I use.
I've found einsum to be amazing at consolidating my code into something more readable, particularly for implementing architectures from scratch.<p>Here's a good video that explains why its so good: <a href="https://www.youtube.com/watch?v=pkVwUVEHmfI" rel="nofollow">https://www.youtube.com/watch?v=pkVwUVEHmfI</a><p>Also check out Lucid Rains Github, who uses it extensively to build transformer architectures from scratch: <a href="https://github.com/lucidrains" rel="nofollow">https://github.com/lucidrains</a>
\<p>* Example: <a href="https://github.com/lucidrains/alphafold2/blob/d59cb1ea536bc5d2e0eee81056d786cb1011ac57/alphafold2_pytorch/alphafold2.py#L150" rel="nofollow">https://github.com/lucidrains/alphafold2/blob/d59cb1ea536bc5...</a>
Also see: <a href="https://github.com/ahwillia/Einsum.jl" rel="nofollow">https://github.com/ahwillia/Einsum.jl</a>
If you're into tensor algebra i can only recommend the beautiful piece of Software Cadabra is:<p><a href="https://cadabra.science/" rel="nofollow">https://cadabra.science/</a><p>We wrote an article with it once, 40th order in the Lagrangian, perhaps 50k pages of calculations when all printed. Amazing tool! Thanks Kasper!
There's also an implementation in R: <a href="https://github.com/const-ae/einsum" rel="nofollow">https://github.com/const-ae/einsum</a>