This is important because there exists a trade-off in statistical learning models: in general, the more flexible your models are, the less understandable they become[0]. Modern machine learning techniques are typically very flexible.<p>To gain intuition and reasoning of a model is to have understanding and trust--transparency. When you strike a nail with a hammer, it's pretty predictable what might happen: the nail could get hit, the hammer could miss, or very rarely, the hammer's head may fly off of the handle. When you replace the hammer with a black box that works correctly 99.999% of the time, but for 0.001%, something completely unpredictable happens, then there's a problem with volatility because that unpredictable event may have unacceptable consequences. I think explainable AI could help with intuitive and more fine-grained risk analysis, and that's certainly a good thing in high-stakes applications such as defense.<p>[0] ISLR, Page 25: <a href="http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.pdf" rel="nofollow">http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.p...</a>
This is fantastic. DARPA gets it. I look forward to whatever fruits come from this labor. Maybe one day I won't have to look at stack traces and reverse engineer 3rd-party dependencies to figure out why things are breaking. Maybe one day error messages will have explanatory power. Maybe one day IDEs will understand abstractions other than ASTs and types and instead will understand things that convey human intent that is not so closely tied to rigid constructs like type systems. What a wonderful world that will be.
Direct link to detailed specification [PDF]: <a href="https://www.fbo.gov/utils/view?id=ae0b129bca1080cc7c517e8dadfa3ca2" rel="nofollow">https://www.fbo.gov/utils/view?id=ae0b129bca1080cc7c517e8dad...</a><p>Related FAQ [PDF]: <a href="http://www.darpa.mil/attachments/XAIFAQ8-26.pdf" rel="nofollow">http://www.darpa.mil/attachments/XAIFAQ8-26.pdf</a>
This brings Lime [1] to mind. "Explaining the predictions of any machine learning classifier"<p>[1] <a href="https://github.com/marcotcr/lime" rel="nofollow">https://github.com/marcotcr/lime</a>
There was a machine learning system designed to produce interpretable results, called Eureqa. Eureqa is a fantastic piece of software that finds simple mathematical equations that fit your data as good as possible. Emphasis on the "simple", it searches for the smallest equations it can find that works, and gives you a choice of different equations at different levels of complexity.<p>But still, the results are very difficult to interpret. Yes you can verify that the equation works, that it predicts the data. But why does it work? Well who knows? No one can answer that. Understanding even simple math expressions can be quite difficult. Imagine trying to learn physics from just reading the math equations involved and nothing else.<p>One biologist put his data into the program, and found, to his surprise, that it found a simple expression that almost perfectly explained one of the variables he was interested in. But he couldn't publish his result, because he couldn't understand it himself. You can't just publish a random equation with no explanation. What use is that?<p>I think the best method of understanding our models, is not going to come from making simpler models that we can compute by hand. Instead I think we should take advantage of our own neural networks. Try to train humans to predict what inputs, particularly in images, will activate a node in a neural network. We will learn that function ourselves, and then it's purpose will make sense to us. Just looking at the gradients of the input conveys a huge amount of information of which inout features are the most and least important. And by about how much.<p>But mostly I think the effort towards explainability is fundamentally misguided. In the domains where they are supposedly the most desirable, like medicine, accuracy should matter above all. A less accurate model could cost lives. Accuracy is easy to verify through cross validation, but explainability is a mysterious unmeasurable goal.
(Looks over shoulder)<p>Anyone else thinking this mirrors their own experimental work?<p>Anyone else thinking of putting in an abstract?<p>Abstract Due Date: September 1, 2016, 12:00 noon (ET)<p>Proposal Due Date: November 1, 2016, 12:00 noon (ET)
(I am relatively excited about this research direction. It seems like the sort of thing that might lead to genuinely useful components of a safer AGI system later.)
Yeah, all they want is a simple mechanism of how to jump from a mere "blind", mechanistic feature extraction to the notion that creatures of this Nature usually have two eys and make a hard-wired heuristic, a short-cut which improves pattern recognition in orders of magnitude with less computational cost.<p>Every child will tell you that cars have eyes, and even a crow could track the direction of your gaze.<p>Well, I would also give away some govt. printed money to know how to make this kind of a jump from raw pixels to high-level shapes.)<p>The answer, by the way, is that the code (which is data) should be evolved too, not just weights of a model. This is an old fundamental idea from the glorious times of using Lisp as AI language - everything in the brain is a structure made out of conses^W neurons.<p>And feature extraction and heuristics should be "guided". In the process of evolution it is guided by way too many iterations of training and <i>random</i> selection of emerging features. Eventually a short-cut "creatures have eyes" will be found and selected as much more efficient. We need just a few millions of years or so of brute forcing.<p>Hey, Darpa, do you fund lone gunmen?)
Integration of Neural Networks with Knowledge-Based Systems
<a href="https://www.uni-marburg.de/fb12/datenbionik/pdf/pubs/1995/ultsch95integration2" rel="nofollow">https://www.uni-marburg.de/fb12/datenbionik/pdf/pubs/1995/ul...</a>
If there is real progress towards Explainable AI, this would also be very useful for _verifying_ machine-learning-based systems (i.e. finding the bugs in them).<p>I wrote about this in [1], but I am not a machine-learning expert (I am coming from the verification side), so would love to hear comments from other people.<p>[1] <a href="https://blog.foretellix.com/2016/08/31/machine-learning-verification-and-explainable-ai/" rel="nofollow">https://blog.foretellix.com/2016/08/31/machine-learning-veri...</a>
This is sorely needed for machine learning if it's to get both more complex and more accurate. Coincidentally Alan Kay brought the "expert systems" idea up in his recent AMA as well. It'd be inconceivable to write code today that couldn't be thoroughly debugged, so we should expect the same of our machine learning systems.
Bringing some form of feature introspection to deep neural networks will probably involve clever ways of visualizing the feature activations of unstructured data <a href="https://arxiv.org/abs/1603.02518" rel="nofollow">https://arxiv.org/abs/1603.02518</a>
My Aunt worked on systems for explaining early generation networks for medical diagnoses: <a href="http://link.springer.com/article/10.1007/BF01413743" rel="nofollow">http://link.springer.com/article/10.1007/BF01413743</a>
"I know of an uncouth region whose librarians repudiate the vain and superstitious custom of finding a meaning in books and equate it with that of finding a meaning in dreams or in the chaotic lines of one's palm ... "<p>JL Borges, The Library of Babel
Off topic: I checked other FizBizOpp listings (which is now famous thanks to the War Dogs film in theaters). There is a listing for a "Big Ass Fan" 16' long for the Air Force: <a href="https://www.fbo.gov/index?s=opportunity&mode=form&id=8de699e71f1dc9a084509651f2221cf3&tab=core&_cview=0" rel="nofollow">https://www.fbo.gov/index?s=opportunity&mode=form&id=8de699e...</a>