A/B testing is just one component of a good product development cycle. The easiest pitfall, though, is that the data only tells you what -- not why. This can lead you to make data-based design decisions blindly.<p>I think the ideal design and development cycle incorporates A/B testing at the end to spot any outliers: You should already have done in-person usability and user testing. It's extremely cheap and, in some cases, almost magical. Seeing a real person use your product will guarantee you plenty of "a-ha!" moments.<p>After testing with real users, I think it's appropriate to test with statistical users: You've already tackled the most glaring issues with what you're building, and the knowledge of how real people use your website can often help show you the "why" of the data, rather than just "what".<p>Steve Krug's "Don't Make Me Think" has a chapter near the end on user testing on a shoestring budget. I'd highly recommend it.
<i>If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.</i><p>You'd better be sure those "1 in a 100" results had a confidence level well above 99%. I think misunderstandings of statistical analysis play a large part in the mistrust some people (mistakenly) have in A/B testing. If you get your analysis wrong, you'll slowly realise that the promised gains of the test turn out to be lies, hence the comparison to snake oil.
In the spirit of A/B testing, perhaps you could point to some reasonable and well-planned tests that produced <i>no</i> actionable results. Contrast that against a test that did produce actionable results. Then compare what went wrong and what, if any, lessons can be extrapolated. That would make for an interesting---and possibly meta---blog post.