In the second example the author does the following:<p><pre><code> $ ACCEPT="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
$ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'
</code></pre>
The author fails to prefix $ACCEPT with the actual header key. It should be:<p><pre><code> $ curl -v -H "$UA" -H "Accept: $ACCEPT" $URL |& grep '< HTTP'</code></pre>
This is just the configuration for a single site. The author did not even try it against www.akamai.com:<p>$ URL=<a href="http://www.akamai.com" rel="nofollow">http://www.akamai.com</a><p>$ UA="User-Agent: Mozilla/5.0 My API Client"<p>$ ACCEPT="Accept: <i>/</i>"<p>$ curl -v -H "$UA" -H "$ACCEPT" $URL<p>< HTTP/1.1 301 Moved Permanently<p>< Content-Length: 0<p>< Location: <a href="https://www.akamai.com" rel="nofollow">https://www.akamai.com</a><p>< Date: Tue, 02 May 2017 14:46:59 GMT<p>< Connection: keep-alive
Is it Akamai? Or is it a single site on Akamai? CDN customers can configure their sites in a million ways.<p>My guess is a single site that was getting DDoS'd added this as an attack signature and forgot about it.
Did Akamai recently just make this change?<p>I'm asking because I've been running a web crawler for years now, and in the past week, I have noticed that the crawler is being rejected in more websites then usual.
I disagree with the author's title (and I see it was submitted with a different title).<p>This is actually a report of two bugs:<p>1- the standard doesn't require an order<p>2 - the IETF's admonition that you be liberal in what you accept.