Cynthia Rudin and interpretable ML models

I’ve always wondered what an sufficient explanation of a neural network would entail.<p>At a very low level, there’s no secret: all of the weights are there, all of the relationships between weights are known. The problem is that it doesn’t tell you anything about the emergent properties of the network, in the same way that quantum physics doesn’t give much insight into biology.<p>It may be possible that there is no English sentence you can utter about the network which is both explanatory and fully accurate. What is the network doing? It’s trying to approximate the function you’ve given it. That’s it.<p>You can try other things like ablation to find the effects of lobotomizing the network in certain ways, but this also can’t fully explain 2nd and higher order relationships.

Getting AI to show its work isn&#x27;t just for accountability. &quot;Showing your work&quot; gives you a clearer picture of the problem &#x2F; solution and prevents &#x2F; fixes bugs in implicit reasoning, the key problem current AI has which prevents it from being truly autonomous.<p>Ask GPT4 to do a task, and then ask it to the same task showing its work; you&#x27;ll find that GPT4 is less likely to make mistakes on the latter. This is especially apparent for tasks like counting # of words and multi-step problems, which GPT normally has trouble with.<p>But GPT4 still tends to struggle even breaking the task down, to the point where it starts producing extremely obvious mistakes (e.g. &quot;the turtle moves 1 unit up, from (1, 0) to (2, 0)&quot;). One possibility is that it isn&#x27;t actually showing its work, it&#x27;s just generating backwards explanations from a latent conclusion. Maybe this research will clarify whether this is the case, and help us develop a more coherent LLM.

&gt;If you want to trust a prediction, you need to understand how all the computations work.<p>I disagree with the premise here. You don&#x27;t understand how all the computations work in the brains of people&#x27;s predictions whom you trust. You simply have a mental calculation of their batting average through exposure to their track record and this batting average functions as a proxy for trust.<p>I find this is more or less the same way that I learn whether or not I can rely on GPT-4 for a particular use-case. If it&#x27;s batting average is north of a certain % for a given use-case, then it doesn&#x27;t need to be right 100% of the time for me to derive value from relying on it.<p>I think we are slowly crossing a threshold where we accept indeterminism and mistakes from machines in a way that we haven&#x27;t in the past.

The <i>models</i>, <i>neural networks</i> Professor Rudin is considering have a LOT of parameters, dimensions, <i>neurons,</i> neuron values, etc.<p>Okay. Since apparently no one has the <i>explanations</i> desired, we have to guess. So, let&#x27;s do some guessing:<p>Given so many parameters, etc., we have in some sense -- in some case of geometry, <i>spaces</i>, maybe <i>vector spaces</i>, maybe as in linear algebra -- a lot of <i>dimensions</i>.<p>Then something surprising holds (once we get precise about a space, easy enough to prove): Given a sphere in the space, we can calculate its volume. We can do this for the space of any finite dimension. Here is the surprise: As we have a lot of dimensions, there is a LOT of volume in that sphere, and nearly all that volume is just inside the surface of that sphere. E.g., if do some work in <i>nearest neighbors</i>, discover this surprise in strong terms.<p>Net, in the space being considered, there is a LOT of volume. Then ...: There is plenty of volume to put faces of cats over here, dogs over there, men another place, women still well separated, essays on bone cancer far away, ..., for thousands, millions, ..., more things, thoughts, topics, etc. Then given some new data, say, a white cat not in the <i>training</i> data, likely the data on that white cat will settle on the volume with the cats instead of dogs, monkeys, etc. and, thus, we will have <i>recognized</i> a cat via some <i>emergent</i> functionality.<p>Just a guess.

I view explainability or interpretability of a network as the ability to take a network and replace it with a drastically smaller set of functions and tables that (a) you can explain and (b) work pretty much the same as the network does.<p>Because we understand these functions and tables, we understand exactly how well the network will work, and also what is missing (i.e., how we can expand its accuracy.)<p>I think this is a very hard problem, but it is one that needs to be solved.

I&#x27;ve always seen interpretability and explainability as different sides of the same coin.<p>If you take an information-theoretic approach to it, and think of a DL model like any other model, there is a certain equivalence in understanding the model features and how it behaves with reference to the universe of data it is applied to.<p>It was an interesting article but I felt like it created problems that need not be there (or maybe it&#x27;s just describing problems that others created?)

To trust a prediction you do not need to understand the underlying computations. What you do need is an on demand understandable rational justifucation of how the prediction was derived at the right semantic domain level.

&gt; They extract deeply hidden patterns in large data sets that our limited human brains can’t parse.<p>I think that the bulk of ML has so far produced what our brains in fact easily see. We can easily perform classification, or generation.

Sometimes I think the interpretation of model parameters is all bunk. I think the legacy of over interpretation of parameters and results has resulted in the occification of some very shaky science.

We have no idea how the human brain works but no one seems to care.

Not sure if it was intentional,but the title of this article is pretty hilarious considering she explicitly badmouths explainability (trying to peer inside a black box) and advocates for interpretability (building models that are less black-boxy in nature).<p>Also, man, quanta feels really rough and popsci-y when it comes to CS.