Somebody really needs to write a Bayesian takedown of all these A/B testing articles. A/B testing is a Bayesian decision problem. There's really no other way to think about it. Determining sample size and frequentist confidence intervals are only relevant insofar as they approximate Bayesian concepts.<p>The issue is the proper tradeoff between exploration and exploitation. What drives the decision is outstanding uncertainty conditional on the data observed (not conditional on the null hypothesis of zero effect and some non-sequential iid sampling process), the discount rate (which is totally absent in this article), and the reward structure (which is not a Type I and Type II error).<p>The absurdity of the frequentist approach is clear from the admonition not to look at the results of the tests too often.
I rarely see many people take into account the opportunity cost of letting a really close A/B test reach 99.99% confidence when the benefit is by definition very marginal (that's why its taking so long, right?). I mean, is it really that bad to go on "close enough" results and move on to bigger and better tests?
Another way to see this, is to use this online calculator: <a href="http://visualwebsiteoptimizer.com/ab-split-significance-calculator/" rel="nofollow">http://visualwebsiteoptimizer.com/ab-split-significance-calc...</a>
If you are lazy, you can get the functions coded in PHP here:
<a href="http://abtester.com/calculator/" rel="nofollow">http://abtester.com/calculator/</a>