Trouble In the House of Google

Its that old issue. If you are paying you are the customer - if you aren't paying you're the product.<p>With Google the customer is the person placing the ads and the product is you.<p>The content farms are the middle man - they try and place you (the product) onto a paying page (the customer) and stop you going to a non-paying page (that doesn't pay-per-click).<p>Google has 2 business models:<p>* I search for an advert and Google sells me directly to the customer<p>* I search for something and Google takes me to a middle man who sells me to a customer<p>The first business model works great and I often search Google for an advert.<p>The second business model is broken - because I (the user) want a search engine that takes me to my destination - if something that triggers a purchase happens along the way, fine).<p>1/3 of the web now consists of Google's Middlemen selling Google's ads for Google.<p>When my (then) colleague Dale had 500,000 page views from his HTML5 pacman (<a href="http://news.ycombinator.com/item?id=1549056" rel="nofollow">http://news.ycombinator.com/item?id=1549056</a>) he didn't put Google Ads on it because 'Google Ads are Cheap'.<p>But then my (non-technical) customer Tim specifically said he wouldn't put Google adds on <a href="http://cyclingbibliography.org/" rel="nofollow">http://cyclingbibliography.org/</a> which is designed to make him income, I thought, Oh!.<p>At Xmas my 12 year old was moaning about Google when looking for something.<p>It has now reached the point where a page ranking algorithm which penalises sites with Google Ads would be welcomed by many people.<p>Google's problem is that only way out is to reduce its income - when it has been tweaking its software to increase its yield.

It's certain that annoying gadgets such as "Instant" reflect poorly on Google priorities.<p>The mission of Google is to <i>help users find stuff</i>, not generate the maximum possible number of ad impressions per query (which would be quite short-sighted).<p>But, is it really getting worse? It's never been possible to use Google effectively to research dishwashers. Never. I remember using a Firefox extension to block specific domains from Google search for a long time (it's now called "OptimizeGoogle" but had another name before that).<p>Dishwashers aside, I still find Google pretty effective.<p>Jeff's post starts with a chart that shows that 88.2% of SO's traffic comes from Google; if Google was that bad, wouldn't users start to use something else? Where is the increase in traffic from Bing (0.9% from the same chart!)? Where's the nascent but so powerful traffic from blekko...??!?

This is really just a rehash of other posts from the last month (linked in article).<p>This post basically complains about two things: the finer points of SEO and content farms.<p>Content farms is an easy one. They're the Web equivalent of spam and I'm talking about the likes of Associated Ontent and Demand Media. They re a relatively new (last few years) phenomenon.<p>My personal view is that no one is better placed to deal with this new threat than Google. Email spam is basically a solved problem on Gmail. Thats not ss there aren't false positives and negatives but it's oohing like it used to be or could be. It'll take time but I believe that content farms are a transitory and doomed business model.<p>As for product searches, this encompasses many things. Anecdotally I recently searched for "&#60;camera make and model&#62; review" and found what I wanted no problem. Prices I found on pricegrabber (they have an iPad app).<p>SEO is a trickier beast. For one it's a constantly moving target. A combination of suboptimal source SEO and content farm SEO gaming allows the scrapers to survive. I can't say that keyword position matters all that much. Anecdotally Jeff claims it does but many factors are at pay so it's always best to be careful about making absolute claims.<p>Jeff claims not to want to be acquired. I'm reminded of a story I heard. Basically: if you wanted money (from angels) ask for advice. If you wanted advice, ask for money (IIRC this story came from either Mark Suster or Jason Calacanis, can't remember).<p>So, if you want to be acquired, say you don't?<p>Lastly, I'll reiterate my own opinion that social search isn't the answer in the general case (ie it will have specific use cases).<p>Content curation is a mixed bag. I believe there will (for at least a very long time) be a place for niche verticals. For example, dpreview is a vertical for cameras. General purpose models like Mahalo I think are doomed for much the same reason that Jeff and Joel have contended that general Q&#38;A sites are doomed.

It looks like the biggest thing that efreedom.com (the most prolific stackoverflow mirror) does to rank higher in google searches is put the category as the first word in the title. What stackoverflow titles as "How do I use MediaRecorder to record video without causing a ..." efreedom titles as "Android: How do I use MediaRecorder to record video without causing a ...". So when I search for "android mediarecorder segmentation fault" all other things being equal efreedom wins.

It's not a very difficult problem to solve. 95% of the content-farm spam comes from a few domains. In the same way that spam-blacklists have proved to be the most-effective way to combat e-mail spam, Google just needs to decide to shut these content-farms out. They don't need to do anything sophisticated like tweak their algorithm... just shut them out. The fact that it hasn't been done yet suggests to me that Google doesn't want to.

I don't understand what people are complaining about when they use google for generic product searches like this.<p>What do you expect "iPhone 4 cases" to return?<p>Links to reviews? Links to Apple's online stores? Links to other retailer's stores? Links to information about what the cases are manufactured from?<p>I don't understand what a search engine is supposed to do in this use case. How can it divine which of the many things related to iPhone cases you're interested in? This generic search could go in many different directions.<p>Personally I would never think go search google directly for a product review like this. Amazon is the best-known place to find reviews from fellow general-consumers.<p>When you use a search engine, I think the key to efficiency is having a firm idea of what type of results you'd like it to return before you press the "Search" button.

<i>We want the whole world to teach each other and learn from the questions and answers posted on our sites. Remix, reuse, share – and teach your peers! That's our mission. That's why I get up in the morning.</i><p><i>However, implicit in this strategy was the assumption that we, as the canonical source for the original questions and answers, would always rank first.</i><p>Translation:<p>We thought syndicating content would give us Google juice but it backfired ...

<i>when was the last time you clicked through to a page that was nothing more than a legally copied, properly attributed Wikipedia entry encrusted in advertisements? Never, right? </i><p>Jeff gets it wrong yet again. Has he never heard of (or clicked a search result that led to) answers.com?

He seems to think this has never happened before, but I can remember Google search quality apparently declining repeatedly in the past... sometimes it seemed to return all the way to where it had been, and sometimes part way, but it isn't as though this is unprecedented. Additionally:<p><i>when was the last time you clicked through to a page that was nothing more than a legally copied, properly attributed Wikipedia entry encrusted in advertisements? Never, right?</i><p>It's not too <i>common</i>, but it's not like it never happens. Again, at times in the past, this has happened regularly for a while, to the point where you have to add "wikipedia" as a search term, but it has always returned to normality after a few days or so.<p>Since this happens from time to time for me, I'm wondering now if Jeff has been doing something right that I'm failing to do when searching.

It is almost as if a dam has cracked and we are seeing the first trickles of "Google sucks lately" stories. It is increasingly becoming an arms race - Google tweaks its algorithms to defeat SEO, Spam and other Gamers and the gamers tweak their tactics to outwit Google's tweaks. Anybody else see an opportunity in this phenomenon to supplant algorithmic search with curated search?

In evolutionary terms, Google are gaining a very solid advantage every day. If Bing were to start growing suddenly, their tools for beating black-SEO and spam would be more primitive due to the lack of natural "predatory pressure". Bing's lack of immunity against some attacks would then set them back.

I just don't understand the problem that Google is having. Why can't they simply penalise sites/domains that are full of rubbish? Or manually boost domains and sites that aren't.<p>The lack of innovation in search worries me - there are big commercial incentives for Google's results to be poor. Though the emergence of viable alternatives will change this.<p>I'm sure I read that the average revenue per search was $0.08 or something around that mark. At that level it's worth having some human intervention. Perhaps Yahoo had something after all!

That's why they're called "search engines" and not "find engines".

The search "iphone 4 case" seems to be particularly susceptible to crap results. Even DDG (<a href="https://duckduckgo.com/?q=iphone+4+case" rel="nofollow">https://duckduckgo.com/?q=iphone+4+case</a>) and Bing (<a href="http://www.bing.com/search?q=iphone+4+case" rel="nofollow">http://www.bing.com/search?q=iphone+4+case</a>) give shady results.

What happens if other sites are scraping content faster than Google can crawl it? In these cases, will Google really be able to guess which site is the original? For all they know, SO is scraping a lot of its content from other sites.<p>If this kind of uncertain-originator is any part of the problem, one solution might be for Jeff to temporarily block robots other than google/bing/etc. from retrieving new content, until say, ten minutes later. This gives the search engine a chance to figure out who the original is, while still (I think) remaining within the spirit of CC-SA. A Google API call (<i>I'm high reputation, please crawl this new page now!</i>) might be even better.<p>edit: clarified API suggestion.

An easy easy to get better results for searches when looking for reviews as well a getting up to date content for searches has become my personal goal for unscatter.com. My first piece is up, using the blekko api. I will be adding more search filters powered by different apis in the future. At the moment it's basically a wrapper around the blekko api I admit but already useful for searching for iphone 4 cases I think. <a href="http://www.unscatter.com/search/?q=Iphone%204%20case&#38;f=reviews&#38;s=date&#38;t=Iphone+4+case" rel="nofollow">http://www.unscatter.com/search/?q=Iphone%204%20case&#38;f=r...</a>

Google has always had "bad neighborhoods" -- places where results weren't so good. What folks are finding is that the bad neighborhoods are on the rise, at least when it comes to short, popular searches. Now it appears the screen scrapers are busy at work targeting tech questions. In the last couple of months, when I had a technical question I got total junk for an answer -- lists of questions that took me to landing pages, re-dos of Stack Overflow pages, and random questions that didn't even have answers.<p>I use Google extensively for search. About once a month or so, I'll be looking for something in a bad neighborhood. It's not a pleasant experience. It's a shame to see tech questions end up like this.<p>But the problem, as another poster pointed out, is that nothing is for free. You are either paying money, in which case you are the customer, or you are the product. There's no "in-between" In Google's business model you are the product.<p>I think the business model can continue for a good, long time, but there is always going to be cross-incentives between people who want free stuff and providers who have to pay money to provide you with stuff. Not everybody can be a wikipedia and raise money with pictures of Jimmy Wales. They are an outlier.<p>My conclusion is that these are browser problems. After all, it's none of my business what people put on the web, and aside from liking Google and wishing them well, I really don't have a dog in the fight for their struggle. In fact, it's better for me to have a dozen search companies all using different algorithms -- makes it harder to game the system.<p>So what I want is a browser. A browser that uses multiple search engines automatically and completely eliminates any "fluff" from rendered pages -- perhaps even combining various pages into much simpler displays.<p>I'd pay for that, and that would make me the customer. Then I would have whatever web experience I desired, instead of the one that I get for free. I'd much rather be in the position of writing a check to the best browser provider that condensed and filtered information than the situation we have now.<p>(By the way, if anybody is interested in this browser project, please contact me, as it's been a pet project of mine for some time)

I think this contributes to an ongoing trend and even bigger threat for google. The way we access the content of the web isnt the same way it was back in 2000. Back then a search engine was your only starting point for the web. Now a growing part part of redirects comes thru in some way curated (mostly social) channels. The poor search results will only increase this trend.

I don't understand why this is such a hard problem to solve.<p>I assume that every business that manages to farm content and SEO it up to the first page must be making a decent investment in time and resources to achieve this. It doesn't happen overnight.<p>So wouldn't it be easy enough to maintain a blacklist or at least a de-value list that would bring the return below the investment? Shouldn't there be a streamlined process for assembling this blacklist? They must already be doing something along these lines and no doubt quite a bit more involved than what I'm describing here.<p>Could they add in a crowdsourcing <i>flag</i> link next to all search results. This wouldn't blacklist anything automatically obviously but would assist in identifying which results should be investigated further?<p>Why is it still an issue? Is it a legal problem? Can they be sued for maintaining a blacklist?<p>I'm not trying to say I know better, so I must be missing something. Maybe someone can shed some light on my ignorance?

Lately, I've been exploring the theory that too many Google employees exist:<p>Thousands of highly motivated employees attempt to expand their resumes + make an impact -&#62; blind expansion of site features + sources of ad revenue -&#62; loss of company character + restraint<p>Once the profit appears, no one dares to backtrack.<p>Does that make sense, or am I just speculating?

It seems like the obvious solution is a crawl on demand service provided by Google-so that when you publish new content, or your content is updated you can get Google to index your new content, and associate it as original content based on first appearance.<p>Then, it would be up to Google to prioritize content originators over farmers.

I suspected the search results I was getting over the past few months were of a lesser quality but thought it was just an aberration.

The current search situation as described by such posts seems analogous to the search quality deteriorating during the emergence of blogging. Google stepped in and cleaned it up rather well. I'd trust them to do the same with whatever tricks the rehash sites are using.

Extrapolation is such a tiring business. Google is constantly changing and developing. How can you make generalizing comments about the future without knowing what they're working on?<p>For future reference, replace Google by pretty much anything.

There should be an attribute which you can add to html elements to state that this is the original content source. Then if Google comes across a large website that has a tonne of "original source" content which lots of other sites are claiming "original source" for, then they can automatically identify it as a scraper site and penalise/flag for manual checking. Something like this, but more extensible:<p>&#60;p original-source="true"&#62;<p><pre><code> This is some content which was generated on this website </code></pre> &#60;/p&#62;

The scrapers are probably doing lot of SEO optimization. It is time for stack overflow to hire some SEO services. Wikipedia is not monetizing in anyway other than donations whereas stackoverflow does display ads of its own so why not hire someone to do SEO and stay on top?

that's why I use DDG with !so

<i>spammers, scrapers, and SEO'ed-to-the-hilt content farms are winning</i><p>Spammers, scrapers: sure, they're a problem.<p>SEO'd sites: there is nothing wrong with optimizing your site for search engines. And a site that's optimized <i>ought</i> to win.