Why Does Integer Addition Approximate Float Multiplication?

The most significant bits of a floating point representation are basically a logarithm of the number. Logarithms have this relation between multiplication and addition.

This is the core trick in the Fast Inverse Square Root[0], as made famous by the Quake III source code.<p>It uses a shift (equivalent to dividing) and a subtraction in integer-land to estimate x^(-0.5) in float-land.<p>[0]: <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Fast_inverse_square_root" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Fast_inverse_square_root</a>

Neat. Of course it works for the exponent, but that it&#x27;s not that far off for the mantissa is unexpected. It helps that the mantissa is normalized.

Previous discussion of the paper:<p><i>Addition is all you need for energy-efficient language models</i> - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41784591">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41784591</a> - <i>Oct 2024</i> - 126 comments

This can be very worth it in circuit design for custom accelerators. Floating point operation is often multicycle. If you can get close enough with addition it will save a ton of resources and probably also simplify the design.

It&#x27;s math and not just for integers. That&#x27;s also how slide rule works.<p>a×b = 10^(log(a×b))<p>log(a×b) = log(a)+log(b)<p>thus a×b = 10^(log(a)+log(b))

&gt; The paper talks about using this to save power, but that’s probably not worth it for a few reasons<p>It&#x27;s probably not worth it to do this in software. But in hardware, it might be!<p>A similar approach with a hardware prototype. <a href="https:&#x2F;&#x2F;research.nvidia.com&#x2F;publication&#x2F;2022-12_lns-madam-low-precision-training-logarithmic-number-system-using-multiplicative" rel="nofollow">https:&#x2F;&#x2F;research.nvidia.com&#x2F;publication&#x2F;2022-12_lns-madam-lo...</a>

I wonder if there was any inspiration from Quake III.<p>Didn’t the inverse square root trick rely on bit-level floating point and subtraction of a bias?

float32 (1+m)×2^e as int32 = e&lt;&lt;23 | m<p>(1+m1)×2^e1 × (1+m2)×2^e2 = (1+m1+m2+m1×m2)×2^(e1+e2)<p>If m1×m2 is small, that&#x27;s approximately float32(m1+m2, e1+e2).

Would subtraction also approximate division?