This is not just an issue for neural nets, but also for brains. Our interpretations of our own actions should always be considered posthoc rationalizations in the absence of some falsifiable experiment being conducted to demonstrate the validity of the interpretation. Human brains are excellent at creating a coherent story about the world they experience based on the data at hand, thus we suffer the same kinds of issues, mitigated only by the fact that we have inherited developmental programs that have been subjected to a huge variety of adverse situations that have rigorously tested their performance (by killing anything that failed).
The pneumonia-asthma example seems to be an example of a Simpson's paradox [1]. The doctors acted on a strong (accurate) belief about asthma sufferers contracting pneumonia and acted in such a way that the data obscured an actual causal link (asthma as an aggravating factor to pneumonia). This is opposed to the canonical Simpson's paradox where doctors acted on a strong (inaccurate) belief about severe kidney stones [1a] and again produced lopsided data that hid the best treatment option until the paradox was identified.<p>Humans have a very hard time uncovering so-called "lurking variables" [2] and identifying such paradoxes. I don't see how a neural network (or other machine learning tool) could do so on their own, but I don't know that much about machine learning. So, I guess I have two questions for the experts out there:<p>* If all training data is affected by a confounding variable, can a machine learning algorithm identify its existence, or is it limited by only knowing a tainted world?<p>* Once we have identified such lopsided data and understood its cause, how do you feed that back into your algorithm to correct for it?<p>---<p>[1] <a href="https://en.wikipedia.org/wiki/Simpson%27s_paradox" rel="nofollow">https://en.wikipedia.org/wiki/Simpson%27s_paradox</a><p>[1a] <a href="https://en.wikipedia.org/wiki/Simpson%27s_paradox#Kidney_stone_treatment" rel="nofollow">https://en.wikipedia.org/wiki/Simpson%27s_paradox#Kidney_sto...</a><p>[2] <a href="https://en.wikipedia.org/wiki/Confounding" rel="nofollow">https://en.wikipedia.org/wiki/Confounding</a>
The answer is no. The problem is, is that we don't trim the neural networks of their spurious connections and instead we're stuck staring at these fully (visually) connected layered networks.<p>Once you start to trim out the spurious connections you start to see that you are left with a logic design with integration/threshold circuits instead of straight binary circuits that we're used to seeing. There are even certain universal network patterns what will emerge to perform different functions just like in binary circuit design.<p>I wrote a paper about this in 2008 that's now been cited about 150 times. It's using Artificial Gene Regulatory Networks instead of Artificial Neural Networks, but the math is the same and the principle still holds:<p><a href="http://m.msb.embopress.org/content/4/1/213.abstract" rel="nofollow">http://m.msb.embopress.org/content/4/1/213.abstract</a>
The trick to accurate interpretability is to decouple accuracy from explanations.<p>Just like an International Master commentator can explain most of the moves of a Super GM, so can an interpretable simple model explain the predictions of a very complex black box model.<p>The work by Caruana referenced in this article actually culminated in a method to get both very accurate models and still retain interpretability.<p><a href="https://vimeo.com/125940125" rel="nofollow">https://vimeo.com/125940125</a><p><a href="http://www.cs.cornell.edu/~yinlou/projects/gam/" rel="nofollow">http://www.cs.cornell.edu/~yinlou/projects/gam/</a><p>More recently there was LIME:<p><a href="https://homes.cs.washington.edu/~marcotcr/blog/lime/" rel="nofollow">https://homes.cs.washington.edu/~marcotcr/blog/lime/</a><p>And there are workshops:<p><a href="http://www.blackboxworkshop.org/pdf/Turner2015_MES.pdf" rel="nofollow">http://www.blackboxworkshop.org/pdf/Turner2015_MES.pdf</a><p>We will get there. 'Permanent' is a very long time and in the grand scale of things, deep learning is relatively new.
When I try to explain neural nets (specifically in vision systems) to people I basically explain how you take inputs in the form of images, label pixels/pixel groups in the images with what you want them to output in the future, and then do that thousands of times and continue to test the results.<p>Critically though, I will say something to the effect of "but if you try and break the net open and see <i>how</i> this specific net came to it's result, it will look like spaghetti"<p>So it's a roundabout way of saying "junk in; junk out." That holds true for <i>any</i> learning system, including human animals. The thought process of humans is inscrutable thus far, and I think that future computing will be similarly inscrutable if we do it correctly.
I think this issue of "Explainable Machine Learning" and interpretability is just going to get more and more important as ML grows. It will also be important for verifying ML-based systems - another problem area.<p>See [1] for a discussion of both.<p>[1] <a href="https://blog.foretellix.com/2016/08/31/machine-learning-verification-and-explainable-ai/" rel="nofollow">https://blog.foretellix.com/2016/08/31/machine-learning-veri...</a>
The human brain also does massive dimensionality reduction on very large amounts of data, and a lot of unconscious processing, with much of it being beyond our capabilities of conscious introspection.<p>I think eventually, within a couple of decades, we will have AI that correlates well enough with human thought process, and has enough knowledge of the world, to be able to introspect and explain in various levels of detail, in natural language, images and other human readable constructs, why it has reached a certain conclusion. And we will be able to experimentally verify those explanations.
I've been saying that ML is much like alchemy than science. They've pretty much given up to understand the underlying mechanism because it's so complex, but that doesn't stop them experimenting because they still get something that looks like a result. And hey, they can get paid for it.<p>Eventually it might grow into a full-fledged science, but it will probably take an awful lot of time.
Isn't this all simply about correlation vs causation? Machine learning can find strong correlations and we can make predictions based on those correlations, but at the end of the day, the machine knows nothing about what is causing any of it, and hence is "inscrutable".<p>So it is up to us to fill the gap in our understanding because that is what machine learning ultimately says about the subject. It tells us what we don't know. If we knew all about the subject, our predictions would match the predictions of the machine because there is only one reality we're both observing. But if there is any gap, then the machine is telling us what we don't know, not what it (of all things) knows. It's just crunching numbers. It doesn't "know" anything.
Interesting article. Some things are weird. I don't know why a support vector machine is ranked better than Bayesian nets, or why they are both worse than ensemble methods w.r.t. interpretability.<p>However, I think the human should not be in the loop. The network should have another semantic layer that serves communication. It can be done from the ground up like Steels or Vogt have been doing.<p>In other words, yes we need insight, but I prefer it through introspective networks. The network should be able to explain itself.
This isn't unique to neural networks at all. There was a machine learning system <i>designed</i> to produce interpretable results, called Eureqa. Eureqa is a fantastic piece of software that finds simple mathematical equations that fit your data as good as possible. Emphasis on the "simple", it searches for the smallest equations it can find that works, and gives you a choice of different equations at different levels of complexity.<p>But still, the results are very difficult to interpret. Yes you can verify that the equation works, that it predicts the data. But why does it work? Well who knows? No one can answer that. Understanding even simple math expressions can be quite difficult.<p>One biologist put his data into the program, and found, to his surprise, that it found a simple expression that almost perfectly explained one of the variables he was interested in. But he couldn't publish his result, because he couldn't understand it himself. You can't just publish a random equation with no explanation. What use is that?<p>I think the best method of understanding our models, is not going to come from making simpler models that we can compute by hand. Instead I think we should take advantage of our own neural networks. Try to train humans to predict what inputs, particularly in images, will activate a node in a neural network. We will learn that function ourselves, and then it's purpose will make sense to us.<p>There is a huge amount of effort put into making more accurate models, but much less into trying to interpret them. I think this is a huge mistake, because understanding a model lets you see it's weaknesses. The things that it can't learn, and the mistakes it makes.
We're using these things and we're not even sure how they work. Love it.<p>At least we should have a standard for characterizing their accuracy or something like that ...
Maybe the community needs a little simulated annealing.
It seems the communal views, approaches, and focus are stuck in a local optimum.<p>Think Different! O'well.
To label so-called causative factors or even actual relationships (in a shifting...virtual...hyperspace) among potential relationships is a separate task than to make meaningful predictions or predictable changes. The Universe is inherently a system-less set of potentials. The <i></i>strongest system is the one that is indeterminate in its methodologies. Systems are survivors of reduction processes.
Adrian Thompson's 1996 paper was about Genetic Algorithms. A poor overfitting example considering the whole article is prominently about Artificial Neural Networks. Thompson's FPGA components were trained at room temperature and the creatures were unable to function well when the temperature deviates too much from 10 deg. C.
<i>“What machines are picking up on are not facts about the world,” [Dhruv] Batra says. “They’re facts about the dataset.”</i><p>This seems analogous to 90% of (random, unreplicable) science these days.