Title has a misleading domain name (gwern.net). Link is to a PhD thesis titled "Scaling Laws for Deep Learning" by Jonathan Rosenfeld. Not sure why wasn't linked more directly,<p><a href="https://arxiv.org/abs/2108.07686" rel="nofollow">https://arxiv.org/abs/2108.07686</a><p><a href="https://arxiv.org/pdf/2108.07686#page=85" rel="nofollow">https://arxiv.org/pdf/2108.07686#page=85</a>
I think the basic premise of this paper is wrong. Very few natural signals are bandlimited - if images were, they would be no need to store in high resolution, you could just upsample. Natural spectra tend to be pink (decaying ~3dB/octave), which can be explained by the fractal nature of our world (zoom in on details and you find more detail).
We've seen band limited CNNs <a href="https://nvlabs.github.io/stylegan3/" rel="nofollow">https://nvlabs.github.io/stylegan3/</a><p>What would the implementation of a band limited LLM look like?
> In particular, this minimal frequency is twice the bandwitdh of the
function.<p>Careful, this is misleading.<p>If the peaks of the frequency align with your samples, you'll get the full bandwidth.<p>If the 0-crossings align with your samples, you'll miss the frequency.<p>These are why people swear by things like HD audio, SACD/DSD, even though "you can't hear over 20khz"
Wasn't there an paper on band limiting generative CNNs, that fixed texture pinning? Basically by blurring the results of the kernel with neighbors, you get rid of all this aliasing?