Robots.txt doesn't confer copyright.<p>What about domains that have been sharked ? Does controlling robots.txt now give me the right to suppress all content ever originating from that domain, for as long as I control robots.txt?<p>Internet archive is right to spider the site, but defer showing. Collection != dissemenation.<p>The IA isn't synthesizing, selling, cross referencing or afaict doing anything nefarious with the data.<p>You are literally picking on the last org on the internet that needs to get picked on.
Meh. I'm pretty ambivalent about voluntary restrictions like robots.txt. As far as I'm concerned it's mostly useful as a way for site operators to document endless dynamic content or requests that are prohibitively expensive (but not so much that they restrict access).<p>I figure if it's on the web and a human can read it, my computer ought to be able to read it too.