Advertisers scramble as ‘non-human traffic’ eats up online budgets

Back in July, hundreds (thousands?) of sites noticed a surge in traffic in their analytics. Hundreds to thousands of additional visits per day above their normal traffic rates. Curiously, all this traffic identified as Internet Explorer 7 on various version of Windows, but each visit came from a different residential IP, and viewed only one page per session.<p>Eventually the common thread between all these sites was identified -- they had retargeting tags from sites like AdRoll: <a href="http://www.thesempost.com/adroll-retargeting-bot-attack-behind-ie7-traffic-surges/" rel="nofollow">http:&#x2F;&#x2F;www.thesempost.com&#x2F;adroll-retargeting-bot-attack-behi...</a><p>For months now, huge botnets have been producing huge amounts of fake web traffic, picking up retargeting cookies in order to later be served ads and have whoever&#x27;s behind the network paid for those views. That surge has never ended. It&#x27;s only grown (including becoming less easy to spot), yet nobody seems to be talking about it anymore, including the retargeting companies.<p>I&#x27;d love to know who&#x27;s behind it and how they control so many computers.

I envision a future where a significant share of the advertising market is comprised of automated ad-bots pitching services and wares to automated web-crawlers. There&#x27;s something obtusely philosophical about that and what it says about us, though I am not sure what that is.

I&#x27;ve been in the online advertising world for 10 years and it still befuddles me to see that most ad dollars still haven&#x27;t switched to cost-per-action (CPA), i.e. only paying for actual conversions. One big benefit of advertising online after all is that tracking conversions and their referrers is much easier than with offline.

&quot;Half the money I spend on advertising is wasted; the trouble is I don&#x27;t know which half.&quot;<p><a href="http://en.wikipedia.org/wiki/John_Wanamaker" rel="nofollow">http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;John_Wanamaker</a>

Advertising in its current form is broken.<p>I was recently in a class where we performed a case study on Google. The professor asked if anyone had ever clicked on an ad. One kid out of 60 had clicked on ad for cheap protein supplements, but hadn&#x27;t bought any. Everyone else scoffed, pointed out their use of AdBlock, and cited their gullible parents as the only people they knew who might ever possibly accidentally click on an ad. Furthermore, the strong majority of our class viewed targeted ads on Facebook and Gmail as extremely creepy and unconvincing.<p>There are two ad models that work in my opinion:<p>1. Native advertising: Ads disguised as useful content. This works for changing people&#x27;s perceptions, especially if its done through official or generally impartial channels, such as news outlets. People think they&#x27;re getting honest information about BP&#x27;s charitable acts and Coke&#x27;s health benefits, but they&#x27;re really being lied to. This doesn&#x27;t generate any direct sales, but it can really influence consumer sentiment on different brands which can have a massive impact on sales indirectly. This method is extremely sneaky and unethical, and that&#x27;s probably why it works.<p>2. Snapchat&#x27;s new model: Opt-in, high quality ads. I liken these ads to movie trailers. They are well-crafted and curated, and you only have to watch them if you want to. This way, people who click on them are already intrigued and have committed to watching the ad, making for a more engaged and curious user base. This method is still being developed but I can see it become big in the next couple of years.

Besides the non-human traffic, most online ads are created to not be seen. I was even thinking about creating a blog post to call out the marketers out there to change this, but I am too lazy... specially to make them make more money. I am so used to how ads that cover the entire page work that I manage to pass by them without even knowing which brand was advertising there.<p>Ads that force you to watch a video for X seconds, but in those X seconds they don&#x27;t even show a logo or something, are money wasted indeed, again, IMHO.<p>Another example, ads that follow me. Oh god how I hate these. I visit a website, usually for a product that I already own, and the freaking ad from that company follows me wherever I go. This is money wasted.<p>Another example, I use Waze almost everyday to figure out the best route for my comute. My 4&quot; screen phone sits there right on top of the radio. Whenever I stop the car, an ad pops up. But they make this ad so tiny that if I need to know what brand or product it is for, I have to pick my phone and look closer. This is money wasted.<p>The online advertising system is pretty broken... can I fix it?

I wouldn&#x27;t be surprised to find that the larger firms are crawling their own ad platforms. When ads comprise 90% of your revenue, it&#x27;s hard to imagine that someone hasn&#x27;t gamed the system.

there&#x27;s a certain irony when advertisers--whose intention is to manipulate the preferences&#x2F;behaviors&#x2F;purchases of consumers--cry foul about their manipulation systems being manipulated

I have an (unfinished) anti-bot PHP project that does customizable automated bot tests and blocking (both via 403 forbidden, and iptables). A custom version of this script is running at one e-commerce site since 2012. The 403 page has a &quot;ticket ID&quot; so the (very few) real users caught can still contact the website admins to be unblocked.<p>The (almost feature-complete) version running at the site above was able to automatically identify and successfully block a number of bots found scraping (or, attempting to scrape) the site since 2012. Though it wasn&#x27;t an easy task, it has been quite successful at blocking most of them since their second (often first) request.<p>In fact since the beginning, and largely upto now, most bots and botnets can still be easily spotted. Some by their user-agent, some falling into bot traps (&quot;honeypots&quot;) which are invisible to regular users, some failing to load images (something that a regular user wouldn&#x27;t do), some crawling the site at an unreasonable pace... most usually don&#x27;t provide an http referer, some even appear to have a number of cookies set for your domain.<p>It&#x27;s not always easy to spot them though - some botnets can only be undone by blocking their full IP subnet, or permanently banning their (sub)domains, like .cn, .ua or .br (if you don&#x27;t have legitimate traffic from those domains).<p>Unfortunately, I haven&#x27;t had enough spare time to complete my project (which tests all of the above conditions to spot bots, but is not ready for a release), but I could if there is some market - or I could sell the project if somebody is interested.

So now 25% to 50% of impressions go to ad bots. It&#x27;s known that most ad clicks from about 10% of human users, who will click on anything but buy little.<p>The real winner in this is Amazon. Amazon doesn&#x27;t pay anybody when someone clicks on a product listing. They have to buy. Same for eBay. So there&#x27;s no point in &#x27;bots clicking on those sites.<p>Online ad rates have been dropping for some time now. Most of the ads run by Google Adsense (the ones on third party sites) are there because Google pushes advertisers hard to use that feature. Most of the conversions from Google Ads come from the ones on search results, which are displayed when the user is looking for something. Those are useful. Most other forms of advertising are interruptions for users interested in something else.<p>The classic tactic ad-based companies try as ad prices drop is increasing the ad density. That&#x27;s called &quot;pulling a Myspace&quot;. It didn&#x27;t work for Myspace, which tanked. Twitter seems to be headed that way - usage up, profits not too good.<p>The way this is going, online advertising is going to have the conversion (to a sale) rate of spam - one in thousands.

&quot;Wah, I can&#x27;t advertise using old methods and might have to come up with something new. Waaah&quot;

There&#x27;s a lot of room for innovation in this space. Some friends (well, clients) of mine have a startup (Unlockable, <a href="https://www.unlockable.com/" rel="nofollow">https:&#x2F;&#x2F;www.unlockable.com&#x2F;</a>) that embeds advertising-based games into other games. The level of interactivity is way more than a bot could handle.

&gt; Do you hate how easy it is for your computer to become slow and riddled with malware?<p>No, I can&#x27;t say I do. If such crime is so rampant, one would think it was ripe for law enforcement to do more about it. I wonder if reporting it to the police every time one visits a relative and removes crap from their computer would start a disruption.

There are two pretty obvious approaches: only pay per conversion, or accept that not all views are human.<p>Just like when you advertise in a classical news paper, you only know how many copies are typically sold, not how many humans actually look at your ad.

This is a problem I have seen first hand. But this is not really as big a problem as it is made out to be. The real issue here is that it is hard to predict of the given impression is a bot impression or a genuine impression. As a consequence any CPM based dealing is difficult. Difficult only means falling CPMs. That is what I have observed.<p>The advertisers will not have to put in more efforts to measure conversions and accordingly bid for the traffic.

This reminds me of <a href="http://dhowe.github.io/AdNauseam/" rel="nofollow">http:&#x2F;&#x2F;dhowe.github.io&#x2F;AdNauseam&#x2F;</a> , which was on HN recently: <a href="https://news.ycombinator.com/item?id=8515398" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8515398</a>

I&#x27;ve wondered about this before if advertising was just designed to be a big negative sum game for advertisers.<p>Interestingly if these same bots were capable of making real purchases and were run by ad companies this behavior may be sustainable because some people are making money. Almost like wall street.

WhiteOps is working on ad-fraud detection: <a href="http://blogs.wsj.com/cmo/tag/whiteops/" rel="nofollow">http:&#x2F;&#x2F;blogs.wsj.com&#x2F;cmo&#x2F;tag&#x2F;whiteops&#x2F;</a> &amp; <a href="http://www.whiteops.com/" rel="nofollow">http:&#x2F;&#x2F;www.whiteops.com&#x2F;</a>

Good. Maybe they&#x27;ll stop throwing so much wasted money at Google.

We stopped all advertising on Facebook. It&#x27;s a mess.