The concept is very cool, but it's not surprising that dimensionality reduction through a non-linear process is going to result in sections of input parameters that yield incorrect (and weird) results. Our visual system, while not the same as these systems, is extremely well developed and robust, yet the list of optical illusions that can fool us is quite long. In this study, the optical illusions are really just surprising because they aren't like anything that would fool humans.<p>This isn't to take away from the research; the most interesting result was just how close to valid inputs these erroneously classified images are.<p>But again, this isn't some fatal flaw. This summary completely neglects the fact that the paper <i>also</i> recommends that -- just like distorted images are added to training sets today (you wouldn't want something common like optical aberration from the camera lens screwing up your classifier) -- in the future, these adversarial examples should be added to training sets to mitigate their effects.<p>> <i>In some sense, what we describe is a way to traverse the manifold represented by the network in an efficient way (by optimization) and finding adversarial examples in the input space. The adversarial examples represent low-probability (high-dimensional) “pockets” in the manifold, which are hard to efficiently find by simply randomly sampling the input around a given example. Already, a variety of recent state of the art computer vision models employ input deformations during training for increasing the robustness and convergence speed of the models [9, 13]. These deformations are, however, statistically inefficient, for a given example: they are highly correlated and are drawn from the same distribution throughout the entire training of the model. We propose a scheme to make this process adaptive in a way that exploits the model and its deficiencies in modeling the local space around the training data.</i>[1]<p>[1] <a href="http://cs.nyu.edu/~zaremba/docs/understanding.pdf" rel="nofollow">http://cs.nyu.edu/~zaremba/docs/understanding.pdf</a>
Let's not forget that the word "imperceptible" is a heavily laden term in this context. There are numerous modifications to the data that would be "imperceptible" to a machine learning system, but would completely confuse a human. For example if you were to divide the image into a grid, and shuffle the squares, many ML systems would be tolerant to this kind of modification because some training regimes do this anyway. To that system you haven't changed anything important about the image and it would correctly classify it.<p>What this result says to me is that there are <i>really useful</i> features of the data that can <i>successfully classify</i> images that humans are totally unaware of! And that's neat.
The key claim, from the original paper:<p>> Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous [...] Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation [...] the same perturbation can cause a different network that was trained on a different subset of the dataset, to misclassify the same input.<p>It's an interesting outcome -- but there are many deep-learning approaches and many different benchmarks, so it will be important to see if this is a misleading anecdote or indicative of a systematic problem.<p>[1] <a href="http://cs.nyu.edu/~zaremba/docs/understanding.pdf" rel="nofollow">http://cs.nyu.edu/~zaremba/docs/understanding.pdf</a>
First thought:<p>Can I turn all digital pictures of me into 'adversarial examples', so the eye of sauron can't identify me from pictures?<p>I'm sure it's not as simple as that, presumably any algorithmic modification to an 'adversarial' nature can be countered by other algorithms.<p>But I predict a new realm of 'arms race' here in the future.
IIRC even the human brain has the 'adversarial' image flaw (these images will be unique to each person), but one simple workaround is to alter the input image via eye movement (which happens unconsciously).
Not an example from deep learning, but [1] also demonstrates that Bayesian systems also have similar problems with sensitivity to initial conditions that are quite similar.<p>It is also rather striking that these DLNs seem to be tricked by what we would typically think of as noise.<p>1. <a href="http://arxiv.org/abs/1308.6306" rel="nofollow">http://arxiv.org/abs/1308.6306</a>
This problem was observed 20+ years ago with linear models used for protein structure prediction. For any given model of what described a properly folded protein, one could locate conformations of the same protein that were rated as folded even better than the correct conformation (I called them doppelgangers, but the name "decoy" is what caught on).<p>The statistical naivete of the field led to all sorts of inadvertent mixing of training and test set data which generated a lot of spurious claims for solving the problem. That is until one attempted to find those decoys and they were <i>always</i> found. This led to the creation of the CASP competition to weed this out and the field finally moved forward.<p><a href="http://en.wikipedia.org/wiki/CASP" rel="nofollow">http://en.wikipedia.org/wiki/CASP</a><p>The key similarity to what I described above is that adversarial search is done posterior to the training of the deep neural network. That makes all the difference in the world IMO. These adversaries may just be strange, otherwise hard to reach bad neighborhoods in image space without using a roadmap. Or they may be an unvaoidable consequence of the curse of dimensionality.<p><a href="http://en.wikipedia.org/wiki/Curse_of_dimensionality" rel="nofollow">http://en.wikipedia.org/wiki/Curse_of_dimensionality</a><p>But given that neural networks have a gradient, it doesn't shock me that it can serve as a roadmap to locate a set of correlated but seemingly minor changes to an example in order to flip its classification. Doing so is simply back-propagation with constant weight values to propagate the gradient to the input data itself - literally a couple lines of code.<p>IMO there are two interesting experiments to do next (not that anyone will take this seriously I expect, but ya know, hear me now, believe me later):<p>1. Characterize the statistical nature of the changes in input images and then use those summary statistics as the basis of an image altering algorithm to see if that can be used to flip the classification of any image on its own. If it can, be afraid, your driverless car may have blind spots. If not, then this is probably just a narrower form of overfitting.<p>2. If it's likely overfitting, attempt an expectation maximization-like fix to the problem. Train the network. Generate adversaries, Add them to the training set, train again and then lather rinse repeat until either the network can't be trained or the problem goes away.<p>Expensive? Yes. But you're Google/Facebook/Microsoft and you have lots of GPUs. No excuses...<p>Failing that, the above is on my todo list so I'm throwing it out there to see if anyone can poke holes in the approach.
I think just like any machine learning algorithm, especially with computer vision, you need to prepare things properly. More robust data and images such as moving window on image patches and image rotations, even going as far as applying filters like sobel for edge detection will make your algorithm a lot better.<p>Any algorithms have weaknesses. It's a matter of being aware of them and compensating for them in your model, possibly by using another model.
But then we wouldn't see faces in clouds...<p>Neutral networks are not perfect solutions, they are solutions that get an organism to reproduce successfully.<p>Read any book on color vision humans have similar problems, yet for the most part we see things, and realize that clouds are just clouds and not faces, except for the religious, they lose their shit when faces appear in clouds.