This is a great master class, if you will, on iterative optimization for core loops. They progressively make improvements to their method until they approach the performance of standard, optimized BLAS implementations. (About as fast as BLIS, less than Intel's MKL, which is about as fast as OpenBLAS. [0]) Timing Eigen without linking against BLAS is a little misleading, since Eigen is meant to be linked against a system BLAS.<p>You wouldn't want to use this code, but it shows you the sorts of things to start paying attention to in this performance-critical sections. I was most surprised by the fact that reordering operations to spread the same instructions apart made a significant difference.<p>(As an aside, your best bet in practical tools is using a metaprogramming library [Blaze seems to be the best], wrapping core operations in a fast BLAS implementation. I personally choose to use Blaze on top of OpenBLAS.)<p>[0] <a href="https://news.ycombinator.com/item?id=10114830" rel="nofollow">https://news.ycombinator.com/item?id=10114830</a>
I've implemented the whole course in Nim in my tensor library: <a href="https://github.com/mratsim/Arraymancer/tree/master/src/tensor/fallback" rel="nofollow">https://github.com/mratsim/Arraymancer/tree/master/src/tenso...</a><p>All BLAS libraries currently only care about float32 and float64, I wanted very fast routines for integer matrix multiplication and use this as a fallback for integers while
using OpenBLAS/MKL/CLBlast (OpenCL)/CuBLAS (CUDA) for floats.<p>Thanks to this I achieved 10x speed compared to Julia and 22x speed compared to Numpy on a 1500x1500 int64 matrix multiplication on CPU: <a href="https://github.com/mratsim/Arraymancer#micro-benchmark-int64-matrix-multiplication" rel="nofollow">https://github.com/mratsim/Arraymancer#micro-benchmark-int64...</a>
I appreciate the work done at the hardware level for streaming computations of structured data. But from the high level view, it's more critical to be able to compute efficiently matrix decomposition, it helps to solve systems of linear equations and obviously discretised PDEs.