Let's try to understand AI monosemanticity

There&#x27;s actually a somewhat reasonable analogy to human cognitive processes here, I think, in the sense that humans tend to form concepts defined by their connectivity to other concepts (c.f. Ferdinand de Saussure &amp; structuralism).<p>Human brains are also a &quot;black box&quot; in the sense that you can&#x27;t scan&#x2F;dissect one to build a concept graph.<p>Neural nets do seem to have some sort of emergent structural concept graph, in the case of LLMs it&#x27;s largely informed by human language (because that&#x27;s what they&#x27;re trained on.) To an extent, we can observe this empirically through their output even if the first principles are opaque.

I just skimmed through it for now, but it has seemed kinda natural to me for a few months now that there would be a deep connection between neural networks and differential or algebraic geometry.<p>Each ReLU layer is just a (quasi-)linear transformation, and a pass through two layers is basically also a linear transformation. If you say you want some piece of information to stay (numerically) intact as it passes through the network, you say you want that piece of information to be processed in the same way in each layer. The groups of linear transformations that &quot;all process information in the same way, and their compositions do, as well&quot; are basically the Lie groups. Anyone else ever had this thought?<p>I imagine if nothing catastrophic happens we&#x27;ll have a really beautiful theory of all this someday, which I won&#x27;t create, but maybe I&#x27;ll be able to understand it after a lot of hard work.

The idea of a smaller NN outperforming what you might think it could do by simulating a larger one reminds me of something I read about Portia spiders (going on a deep dive into them after reading the excellent &#x27;Children of Time&#x27; by Adrian Tchaikovsky). The idea is that they&#x27;re able to do things with their handful - on the order of tens of thousands - of neurons that you&#x27;d think would require 4 or 5 orders of magnitude more, by basically time-sharing them; do some computation, store it somehow, then reuse the same neurons in a totally different way.

Isn&#x27;t this also(just?) a description of how high-dimensional embedding spaces work? Putting every kind of concept all in the same space is going to lead to some weird stuff. Different regions of the latent space will cover different concepts, with very uneven volumes, and local distances will generally be meaningful (red vs green) but long-distances won&#x27;t (red vs. ennui).<p>I guess we could also look at it the other way; embedding spaces work this way because the underlying neurons work this way.

Before finishing my read, I need to register an objection to the opening which reads to me so as to imply it is the only means:<p>&gt; Researchers simulate a weird type of pseudo-neural-tissue, “reward” it a little every time it becomes a little more like the AI they want, and eventually it becomes the AI they want.<p>This isn&#x27;t the only way. Back propagation is a hack around the oversimplification of neural models. By adding a sense of location into the network, you get linearly inseparable functions learned just fine.<p>Hopfield networks with Hebbian learning are sufficient and are implemented by the existing proofs of concept we have.

At least the first part reminded me of Hyperion and how AIs evolved there (I think the actual explanation is in The Fall of Hyperion), smaller but more interconnected &quot;code&quot;.<p>Not sure about actual implementation, but at least for us concepts or words are not pure nor isolated, they have multiple meanings that collapse into specific ones as you put several together

&gt; No one knows how it works. Researchers simulate a weird type of pseudo-neural-tissue, “reward” it a little every time it becomes a little more like the AI they want, and eventually it becomes the AI they want.<p>There is a distinction to be made in &quot;knowing how it works&quot; on architecture vs weights themselves.

Personally I find the original paper much better written and easier to understand: <a href="https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2023&#x2F;monosemantic-features&#x2F;index.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2023&#x2F;monosemantic-features&#x2F;...</a>

By the same token, thinking in memes all the time may be a form of impoverished cognition.<p>Or, is it enhanced cognition, on the part of the interpreter having to unpack much from little?

As described in the post, this seems quite analogous to the operation of a bloom filter, except each &quot;bit&quot; is more than a single bit&#x27;s worth of information, and the match detection has to do some thresholding&#x2F;ranking to select a winner.<p>That said, the post is itself clearly summarizing much more technical work, so my analogy is resting on shaky ground.

All this anthropomorphizing of activation networks strikes me as very odd. None of these neurons &quot;want&quot; to do anything. They respond to specific input. Maybe humans are the same, but in the case of artificial neural networks we at least know it&#x27;s a simple mathematical function. Also, an artificial neuron is nothing like a biological neuron. At the most basic -- artificial neurons don&#x27;t &quot;fire&quot; except in direct response to inputs. Biological neurons fire <i>because of their internal state</i>, state which is modified by biological signaling chemicals. It&#x27;s like comparing apples to gorillas.

&gt; Shouldn’t the AI be keeping the concept of God, Almighty Creator and Lord of the Universe, separate from God-<p>This seems wrong. God-zilla is using the concept of God as a superlative modifier. I would expect a neuron involved in the concept of godhood to activate whenever any metaphorical &quot;god-of-X&quot; concept is being used.

I feel like we&#x27;re a few more paradigm shifts away from self-driving cars, and this is one of them - being able to actually understand neural nets and modify them in a constructive way more directly - aka engineering.<p>Some more:<p><pre><code> cheaper sensors (happening now) better sensor integration (happening now, kind of) better tools for ml grokking and intermediate engineering (this article, kind of) better tools for layering ml (probably the same thing as above) a new model for insurance&#x2F;responsibility&#x2F;something like this (unsure) better communication with people inside and outside the car (barely on the radar)</code></pre>

This reminds of research done on category-specific semantic deficits, where there can be neurodegeneration that impacts highly specific or related knowledge (for example, brain trauma that affects a person&#x27;s ability to understand living things like zebras and carrots, but not non-living things like helicopters or pliers).<p><a href="https:&#x2F;&#x2F;academic.oup.com&#x2F;brain&#x2F;article&#x2F;130&#x2F;4&#x2F;1127&#x2F;278057" rel="nofollow noreferrer">https:&#x2F;&#x2F;academic.oup.com&#x2F;brain&#x2F;article&#x2F;130&#x2F;4&#x2F;1127&#x2F;278057</a>

such a wonderful article, really enjoyed reading it

When LLMs are trained on text, are the words annotated to indicate the semantic meaning, or is the LLM training process expected to disambiguate the possibly hundreds of semantic meanings of an individual common word such as &quot;run&quot;?

see <a href="https:&#x2F;&#x2F;www.nature.com&#x2F;articles&#x2F;nature12160" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.nature.com&#x2F;articles&#x2F;nature12160</a> &quot;mixed selectivity neurons&quot;

Superposition makes sense when you understand all of ML as a convolution.

COOL~ interpreting a big AI with a bigger is like interpreting 42 with Earth~

The original is quite long, but quite interesting.[1] Reading it makes me feel like I did reading A Brief History of Time as a middle schooler - concepts that are mainly just out of reach, with a few flashes that I actually understand.<p>One particularly interesting topic is the &quot;theories of superposition&quot; section, which gets into how LLMs categorize concepts. Are concepts all distinct or indistinct? Are they independent or do they cluster? It seems that the answer is all of the above.<p>This ties into linguistic theories of categorization[2] that I saw referenced in (of all places) a book about the partition of Judaeo-Christianity in the first centuries CE.<p>Some categories have hard lines - something is a &quot;bird&quot; or it is not. Some categories have soft lines - like someone being &quot;tall.&quot; Some categories work on prototypes, making them have different intensities within the space - A sparrow, swallow, or robin is more &quot;birdy&quot; than a chicken, emu, or turkey. Apparently Wittgenstein was the first to really explore with Family Resemblances that a category might not have hard boundaries, according to people who study these things.[3] These sorts of &quot;manifolds&quot; seem to appear, where some concepts are not just distinct points that are or aren&#x27;t.<p>It&#x27;s exciting to see that LLMs may give us insights into how our brains store concepts. I&#x27;ve heard people criticize them as &quot;just predicting the next most likely token,&quot; but I&#x27;ve found myself lost when speaking in the middle of a garden path sentence many times. I don&#x27;t know how a sentence will end before I start saying it, and it&#x27;s certainly plausible that LLMs actually do match they way we speak.<p>Probably the most exciting piece is seeing how close they seem to get to mimicking how we communicate and think, while being fully limited to language with no other modeling behind it - no concept of the physical world, no understanding of counting or math, just words. It&#x27;s clear when you scratch the surface that LLM outputs are bullshit with no thought underneath them, but it&#x27;s amazing how much is covered by linking concepts with no logic other than how you&#x27;ve heard them linked before.<p>[1] <a href="https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2023&#x2F;monosemantic-features&#x2F;index.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2023&#x2F;monosemantic-features&#x2F;...</a><p>[2] <a href="https:&#x2F;&#x2F;www.sciencedirect.com&#x2F;science&#x2F;article&#x2F;abs&#x2F;pii&#x2F;001002857690013X" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.sciencedirect.com&#x2F;science&#x2F;article&#x2F;abs&#x2F;pii&#x2F;001002...</a><p>[3] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Family_resemblance" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Family_resemblance</a>

I’ve never understood the hidden layers argument. Ultimately these models are executing code. You can examine the code. Why can’t that be done?

This is egregiously false. ML models are math and code. They are not tissue; there are no neurons in an ML model. None. The analogy is false. And it is used here to make you think that ML models are more human than they are, or more superhuman. There is no &quot;bigger AI waiting to get out&quot;. Shoggoth anyone? These are scare tactics from hyperventilating EAs (both Scott and the Anthropic team).

[disclaimer: after talking to many people much smarter than me, I might, just barely, sort of understand this. Any mistakes below are my own]<p>Thanks for the heads up! AI is better at making up stories than humans already. Hodl my beer while I go buy some BSitCoins.

Every training set will produce a different set of weights, even the same training set with produce different weights with different initialization, leave alone slightly different architectures.<p>So what exactly is the point, except &quot;look at us, we are so clever&quot;?

As long as you read it with the skeptics &quot;yes, but it&#x27;s not intelligence&quot; its a good read.<p>It&#x27;s when you read it with the &quot;at last, I can understand how reasoning and inference with meaning is going to emerge from this&quot; you have a problem.<p>It&#x27;s a great read but what you bring to it, informs what you take from it.