Supporting half-precision floats is annoying

The big problem in this post seems to be compiling via C (and the awkward GPU extensions thereof), which has lousy Float16 support. LLVM has adequate support for at least representing Float16s. Because of this (and a lot of work by a lot of people), Julia has pretty good support for Float16. If you&#x27;re running on hardware with native Float16 support like a GPU, it works and is fast; if you&#x27;re running on hardware without native Float16 support, operations are implemented by converting to Float32 and back, which is slow but gives the same results. So you can run the program either way and get the same results and it&#x27;s fast if your hardware has native support. Same deal with BFloat16 [1], which is the native 16-bit floating point type on Google&#x27;s TPUs.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;JuliaMath&#x2F;BFloat16s.jl" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;JuliaMath&#x2F;BFloat16s.jl</a>

For what it&#x27;s worth, the Arm &quot;alternative half-precision&quot; format is not very different from IEEE binary16. The difference is that instead of using the maximum exponent value (0x1f) to represent infinities and NaNs, the AHP format uses it the same way as any other non-zero exponent, to represent normalized fp values. The tradeoff is that you lose NaNs and infinities, but you double the range of numbers you can represent (the max value goes from 65504 to 131008).<p>You have to select AHP by setting an FP config register bit, so (unlike bfloat16 vs binary16) it&#x27;s a &quot;for this whole chunk of code I am going to use this format&quot; choice.

The article makes a lot of good points. But there are some cases where f16 is very useful. In the context of deep learning it&#x27;s frequently useful to move from f32 -&gt; f16. This can allow you to double the size of your models in memory (system or GPU&#x2F;TPU). Since network size is often determinant of performance, doubling the number of parameters&#x2F;activations in your model can make a big difference.

Some years ago I used them to transfer real-time waveforms from a medical device via USB and BLE links since they provided more than enough precision for the clinical application. The 2x bandwidth increase without resorting to compression (and its computational overhead) allowed us to meet the project specifications just by changing the type of the data array and recompiling.

Great rant.<p>&quot;just don&#x27;t use f16&quot; seems like the course of wisdom here.<p>I had a 3x 5 bit value, packed structure in something where memory pressure was <i>severe</i>, and it was <i>such</i> a bitch on a SPARC to deal with 16 bit quantities that actually running the data as two passes using more memory wound up being an immensely better approach. The Alphas would diddle yer bits any way you liked at speed, but that was clearly an aberrant ability.

16-bit floating point (as a weight storage format, not used for math) is essential for fast Winograd and Fourier convolution on AVX-512 CPUs. See <a href="https:&#x2F;&#x2F;NN-512.com" rel="nofollow">https:&#x2F;&#x2F;NN-512.com</a>

If the CPU only has 32-bit ALUs (afaik all CPUs today have 32+ bit ALUs) there is no reason to &quot;support&quot; half-floats (other than converting floats to half-floats for the GPU which doesn&#x27;t need to be fast or pretty since you do it beforehand and send it directly to the GPU from the model file format).<p>On the GPU on the other hand 16-bit floats are becoming the standard (the M1 GPU for instance has more 16-bit ALUs than 32-bit). With enough precision for possible resolutions&#x2F;worlds and you save 2x the memory which makes it a no-brainer really.

The assertion that C doesn&#x27;t support fp16 is just plain wrong. _Float16 is defined in standards committee work. There&#x27;s also the __fp16 storage-only type widely supported by compilers.<p>The main issue is many compilers have issues on x86 platforms due to Intel&#x27;s bizarre slowness in defining how to pass fp16 parameters in their official ABI.

This reminds me that at one point Lucene was using an 8-bit floating point format for its normalization factors (I don&#x27;t know if they still do):<p><a href="https:&#x2F;&#x2F;lucene.apache.org&#x2F;core&#x2F;3_0_3&#x2F;fileformats.html#N107EF" rel="nofollow">https:&#x2F;&#x2F;lucene.apache.org&#x2F;core&#x2F;3_0_3&#x2F;fileformats.html#N107EF</a>

Athas, now that you’ve added fp16, why not add bf16 as well? (A100s support bf16 natively, as do upcoming server CPUs).

.NET recently landed (storage-only) support for f16 as well. Their article is nowhere near as interesting as TFA, but for reference: <a href="https:&#x2F;&#x2F;devblogs.microsoft.com&#x2F;dotnet&#x2F;introducing-the-half-type&#x2F;" rel="nofollow">https:&#x2F;&#x2F;devblogs.microsoft.com&#x2F;dotnet&#x2F;introducing-the-half-t...</a>

What are the reasons for changing the allocation of bits in bf16 vs. f16? Why are there no (few?) similar alternative allocation schemes for f32 and f64? Was IEEE&#x27;s choice perfect for f32 &#x2F; f64? How did they know? Why not for f16?<p>Does any hardware offer &quot;configurable&quot; bit allocation like f16[e=4,m=11]?

Huh? Half precision floats speeds up machine learning training and inference enormously. It&#x27;s weird to me to argue that we should sacrifice hours of sliced off waiting time in favor of preserving the ability to write a new language over a weekend.

Sometimes you need float packed in memory efficiently.<p>This might be useful in graphics, for example.<p>In general, I believe it is good to give people options. You never know what people will find useful.<p>But if you don&#x27;t want support it and you feel it will make your language better, just don&#x27;t support it.<p>But please, quit bitchin&#x27; about it.<p>Java does not support unsigned integer type and everybody is fine.<p>And Java devs don&#x27;t write rants about how unsigned integers are useless, supporting them is annoying and everybody should just forget about them.