R, OpenMP, MKL, Disaster

In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always the case back then). I benched this extensively for all my R usecases and for several systems that i cared about back then. So there is usually no need to use MKL in the first place.

&gt; As a good citizen he wanted to file a documentation bug to have this behaviour documented. But R’s bug tracker seems not to be open to the public. So the story has to be recorded here for Google to find.<p>Huh? The bug tracker is here:<p><a href="https:&#x2F;&#x2F;bugs.r-project.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;bugs.r-project.org&#x2F;</a><p>Yes, for filing a bug you need to request an account because they apparently were overwhelmed with spam, as documented here:<p><a href="https:&#x2F;&#x2F;www.r-project.org&#x2F;bugs.html" rel="nofollow">https:&#x2F;&#x2F;www.r-project.org&#x2F;bugs.html</a>

Systems that fail without producing an error or warning are worrying and unsettling to me.

I had a similar problem in a prediction pipeline a few years back. If I remember correctly, someone updated a R package to the next minor version. The package was to read an obscure file format. The fix installed a new C++ library. That C++ library somehow interacted with a second R package (using a specialized type of linear model) when compiled at source and all the results coming out of our package were subtly wrong but only with large files.<p>It turns out the way the second R package would determine the required precision of floats in sparse arrays was based on the compiled linear algebra libraries available. It took a week for us to debug and ultimately it was easier for us to just rewrite the whole thing in Python.<p>Renv has made things easier but I don&#x27;t think packrat&#x2F;renv allows you to lock C&#x2F;C++ libraries as well as R ones.

It&#x27;s perhaps worth saying that if you must mix OpenMP libraries built against the LLVM (Kuck&#x2F;Intel) runtime, and GNU GOMP on GNU&#x2F;Linux: Ensure libomp is built with GOMP compatibility, however that&#x27;s configured, make a shim from the result, like<p><pre><code> gcc -shared -Wl,-soname=libgomp.so.1 -o libgomp.so.1 empty.c -lomp5 </code></pre> where empty.c is an empty file, and put the result on LD_LIBRARY_PATH ahead of the real libgomp. Alternatively, preload the compatible libomp5. On Debian 11 there&#x27;s already a libgomp in the llvm packaging. Dynamic linking assumed, as is right and fitting.

I don&#x27;t have MKL to try this out, but I&#x27;d check that the MKL threading choice actually didn&#x27;t break the initialization to 1.0 loop.<p>That is, instead of checking after doing the x[i] *= SCALE bit with cblas, I would check both before and after the scaling.

In this excellent article Patrick Li, the author of a new optionally type language Stanza and co-founder of JITX (YC S18), provided a compelling reason to design a new language [1]. TL;DR, a powerful language like Ruby enabled the creation of powerful RoR library and framework that help spawned unicorn size startups like Github and Twitter, that&#x27;s otherwise not feasible.<p>I want to add another dimension to this argument, what if we can maintain an existing language eco-system (library, community, etc) but modernize the engine that&#x27;s running and compilation of the R language. This new engine can avoid the dreaded global locking limitation, provides native multi-thread applications and seamless interface with non-native R libraries in C&#x2F;C++. Interestingly someone has tried this, with a sponsorship from Oracle no less, and presented this futile effort in the last year&#x27;s R flagship conference keynote [2].<p>IMHO he will be more successful in his endeavour if using D language in his previous endeavors. What so special about D language you may ask? I would refer to the fact most of the languages do not provide Ruby on Rails (RoR) like tool except D but that for another story (see ref [1]). There&#x27;s also the fact that D has a working alternative library to OpenBLAS and MKL, and it&#x27;s even faster than both them five years back [3]! D also supports open method as an alternative for multiple dispatches that is much touted by Julia language community. D is also bringing native support for borrow checker feature that&#x27;s always mentioned in the same sentence as Rust language. In addition D also has second to none FFI support for C and C++ language. Heck the latest D compiler has standard C compiler built-in. I can go on furthermore but I think you&#x27;ve probably already got the pictures.<p>My not so humble proposal to R and D language community is to compile R on top of D language. Essentially you a have dynamic language of R that is compiled at runtime (CTFE) on top of static D language. This approach is becoming more popular now as posted recently for the new Val and Valet language combination [4]. Just think of CTFE as the new JVM, but provides truly static and native compilation for R.<p>[1] What makes a programming language productive? “Stop designing languages. Write libraries instead.”:<p><a href="https:&#x2F;&#x2F;jaxenter.com&#x2F;stop-designing-languages-write-libraries-instead-126738.html" rel="nofollow">https:&#x2F;&#x2F;jaxenter.com&#x2F;stop-designing-languages-write-librarie...</a><p>[2] Why R? 2020 Keynote - Jan Vitek - How I Learned to Love Failing at Compiling R:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=VdD0nHbcyk4" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=VdD0nHbcyk4</a><p>[3] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen:<p><a href="http:&#x2F;&#x2F;blog.mir.dlang.io&#x2F;glas&#x2F;benchmark&#x2F;openblas&#x2F;2016&#x2F;09&#x2F;23&#x2F;glas-gemm-benchmark.html" rel="nofollow">http:&#x2F;&#x2F;blog.mir.dlang.io&#x2F;glas&#x2F;benchmark&#x2F;openblas&#x2F;2016&#x2F;09&#x2F;23&#x2F;...</a><p>[4] Show HN: Val - A powerful static and dynamic programming language(val-lang.org):<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=28683171" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=28683171</a>

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R.<p>R is good for explorative data analysis but useless for everything else.