Honestly, some sites are so ridiculously malconfigured in their anti-bot zeal that it becomes a Heisenberg like dilemma.<p>E.g. I want to pull in the rss. It is there specifically for m2m. If I dare get the robots.txt, i'm flagged as a bot, and denied the whole site. <i>including</i> not just the rss but even the parts that are not denied per the robots.txt