Adversarial Examples Are Not Bugs, They Are Features

Deep convolutional networks, by design, are unable to integrate contextual and ambient information present in an image (or in preceding images) to inform how to interpret local features they use. So it&#x27;s no surprise they struggle with unconstrained images. Images where ambient context varies.<p>It&#x27;s intriguing how much focus there is on adversarial examples. You don&#x27;t need adversarial examples to make a deep network fail - in a sense that&#x27;s overkill. Just point the poor deep network at a sequence of images from the real world -- images from a self driving car, security camera, or webcam. You&#x27;ll see it make spontaneous errors. No matter how much training data you gave it.<p>The field will advance when&#x2F;if practitioners recognize that classifying pixel patterns in isolation isn&#x27;t sufficient for robust visual perception, and adopt alternative neural network designs that can interpret what they perceive in light of (no pun intended) context and physical expectations.<p>It worked for our prototype.[0]<p>[0] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1607.06854" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1607.06854</a>

There&#x27;s a good summary of the paper by the authors here, for people who don&#x27;t want to digest the pdf: <a href="http:&#x2F;&#x2F;gradientscience.org&#x2F;adv&#x2F;" rel="nofollow">http:&#x2F;&#x2F;gradientscience.org&#x2F;adv&#x2F;</a>

Very interesting paper. With some surprising insights (need to read it a couple more times for sure).<p>The conclusion states:<p>&gt; Overall, attaining models that are robust and interpretable will require explicitly &gt; encoding human priors into the training process.<p>I feel that is true, though another part of the solution IMO lies in coming up with classifiers that can do more than output a probability alone. I agree that classifiers being sensitive to well-crafted adversarial attacks is something that can&#x27;t be avoided (and perhaps even shouldn&#x27;t be avoided at the train-data level), but the problem lies mainly at the output end. As a user, the model gives no insights towards how &quot;sure&quot; it feels about its prediction or whether the inputs deviate from the train set (especially in the useful non-robust feature set). This is especially a problem given that we stick softmax on almost all neural networks, which has a tendency to over-estimate the probability of the rank 1 prediction which confuses humans. Most adversarial attacks show [car: 99%, ship: 0.01%, ...] for the original image and [ship: 99%, car: 0.01%, ...] for the perturbed image.<p>Using interpretability and explanatory tools to inspect models is a good start, though I&#x27;d like to see more attention being given to:<p>- Feedback with regards to whether a given instance deviates from the training set, and to which extent<p>- Bayesian constructs w.r.t. uncertainty being incorporated, instead of only probabilities. Work exists that tries to do this already [1,2] with very nice results, though is not really &quot;mainstream&quot;<p>[1]: <a href="https:&#x2F;&#x2F;alexgkendall.com&#x2F;computer_vision&#x2F;bayesian_deep_learning_for_safe_ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;alexgkendall.com&#x2F;computer_vision&#x2F;bayesian_deep_learn...</a><p>[2]: <a href="https:&#x2F;&#x2F;eng.uber.com&#x2F;neural-networks-uncertainty-estimation&#x2F;" rel="nofollow">https:&#x2F;&#x2F;eng.uber.com&#x2F;neural-networks-uncertainty-estimation&#x2F;</a>

One thing I don&#x27;t agree with is that notion of robustness is human specified, when they clearly measure robustness of a given feature before classification is changed.<p>Robustness is a systems statistical notion of amount or degrees of freedom of state perturbation required to change output, also taking into consideration the magnitude of change. It is related to but not same as system theoretical stability. There&#x27;s nothing human about the definition. Robust features need not be human derived.<p>The desired degree of robustness vs absolute accuracy or precision or bias trade-off is human specified but generally the trade-off is not huge between these variables.

One of the first authors here - happy to answer any questions!

Mirage