Hi HN,<p>Author here. I had been fascinated with Andrej Karpathy's article (<a href="https://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow">https://karpathy.github.io/2015/05/21/rnn-effectiveness/</a>) -- especially where it shows neurons being activated in response to brackets and indentation.<p>I built Ecco to enable examining neurons inside Transformer-based language models.<p>You can use Ecco to simply interact with a language model and see its output token by token(as it's built on the awesome Hugging Face transformers package). But more interestingly you can use it to examine neuron activations. The article explains more: <a href="https://jalammar.github.io/explaining-transformers/" rel="nofollow">https://jalammar.github.io/explaining-transformers/</a><p>I have a couple more visualizations I'd like to add in the future. It's open source, so feel free to help me improve it.
Interesting. The non-negative matrix factorization on the first level kinda highlight some semantic groupings : paragraph, verbs, auxiliaries, commas, pronouns, nominal propositions.<p>I tried to look at higher level layers, and the grouping were indeed of higher level : for example at level 4 there was a grouping which highlighted for any punctuation (and not just comma). The grouping were also qualifying more : for example ("would deliberately" whereas at lower level it was just would).<p>But it's not as clear as I had hoped it would be. I hoped it would somehow highlight grouping of higher and higher size, that could nicely map to the equivalent of a parse-tree.<p>The problem I have with this kind of visualizations, is that they often require interpretation. Also, they don't tell me if the structure was really present by the neural network but was just not apparent because the prism of the Non-negative Matrix Factorization hid it.<p>For my own networks, instead of visualizing, I like to quantify things a little more. I give the neural network some additional layers, and I try to make the neural network produce the visualization directly. I give it some examples of what I'd like the visualization to look like, and jointly train/fine-tune the neural network so that it solve simultaneously his original task, and the production of the visualization which is then easier to inspect.<p>Depending on how many additional layers I had to add, and depending on where they were added, and depending on how accurate (measured by a Loss Function!) the network prediction are, I can better infer how it's working internally, and whether or not the network is really doing the work or if it is taking some mental shortcuts.<p>For example in my Colorify [1] browser extension, which aims to reduce the cognitive load of reading, I use neural networks to predict simultaneously visualizations of sentence-grouping, linguistic features, and even the parse-tree.<p>[1] <a href="https://addons.mozilla.org/en-US/firefox/addon/colorify/" rel="nofollow">https://addons.mozilla.org/en-US/firefox/addon/colorify/</a>
Helping people understand "what the ai is thinking" is really important when you are trying to get organizations to adopt the technology. Great work.
This work is awesome!<p>Are there theoretical reason to choose NMF over other dimensionality reduction algorithms, e.g. UMAP?<p>Is it easy to add other DR algorithms? I may submit a PR adding those in if it is...
Thank you, Jay! I appreciate the addition of the colab notebooks with code examples. I can't wait to play around with this and investigate how language models _speak_.
Wow, love the NNMF visualization. Like all great visualizations, it does a very good job of showing and not telling me what's going on. More of this, please. One question: how does this kind of thing line up with what people describe as "explainable AI?"