Amazing that getting some press on this issue, which is very, very old, took some former Googlers who are trying to compete with their ex-employer. Monopoly in search does rely in part on cooperating websites, who allow Google to crawl but block others. Why. Because Google pays them or helps them get paid. Or at least, Google is the only way the website operator believes her site can be found.<p>If all websites offered sitemap.xml files (links to these files are found in robots.txt), then "crawling" would in many cases be unnecessary.