Did you investigate other search processes besides SGD? I'm thinking of those often termed "biologically plausible" (e.g. forward-forward, FA). Are their internal representations closer to the fractured or unified representations?
I am glad they evaluated this hypothesis using weight decay which is primarily thought of to induce a structured representation. My first thought was that the entire paper was useless if they didn't do this experiment.<p>I find it rather interesting that the structured representations go from sparse to full to sparse as a function of layer depth. I have noticed that applying weight decay penalty as an exponential function of layer depth gives improved results over using a global weight decay.
> Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance.<p>Scaling up almost always leads to better performance. If you're only getting linear gains though then there is absolutely nothing to be excited about. You are in a dead end.
This is an interesting line of research but missing a key aspect: there's (almost) no references to the linear representation hypothesis. Much work on neural network interpretability lately has shown individual neurons are polysemantic, and therefore practically useless for explainability. My hypothesis is fitting linear probes (or a sparse autoencoder) would reveal linearly semantic attributes.<p>It is unfortunate because they briefly mention Neel Nanda's Othello experiments, but not the wide array of experiments like the NeurIPS Oral "Linear Representation Hypothesis in Language Models" or even golden gate Claude.
"I looked at the representations of a network and I don't like them".<p>Great! There's no mathematical definition of what a fractured representation is. It's whatever art preferences you have.<p>Our personal preferences aren't a good predictor of which network will work well. We wasted decades with classical AI and graphical models encoding our aesthetic into models. Just to find out that the results are totally worthless.<p>Can we stop please? I get it. I too like beautiful things. But we can't hold on to things that don't work. Entire fields like linguistics are dying because they refuse to abandon this nonsense.