Validating UTF-8 bytes using only 0.45 cycles per byte (AVX edition)

I see a lot of applications trying to take advantage of SIMD, but what when you try to run them on systems that don't support these instructions? My guess is that you need to write multiple files taking advantage of different sets of instructions and then dynamically figure out which to use at runtime with cpuid, but isn't that cumbersome and a way to inflate a codebase dramatically?

Under the new string model in java &gt; 8 a fairly frequent workflow is:<p>1) get external string<p>2) figure out if it is UTF-8, UTF-16, or some other recognizable encoding<p>3) validate the byte stream<p>4) figure out if the code points in the incoming string can be represented in Latin-1<p>5) instantiate a java string using either the Latin-1 encoder or the UTF-16 encoder<p>I know some or all of these steps are done using hotspot intrinsics, and then the JIT&#x2F;VM does inlining, folding and so on, but I wonder how fast a custom assembly function to do all these steps at once could be.

Previous blog post on HN:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=17081571" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=17081571</a>

I wonder about the Joules per byte. AFAIK AVX units are quite expensive energy-wise.

What does linux utilities like sed, awk use for text manipulation because they were very slow when I was changing a few table names in a sql file.