I'm disappointed that this did not generate a discussion on HN. About two years ago, I was made accurately aware that methodology is one of the most -- <i>if not the most</i> -- important factor in conducting research, and that my methodology was shit. I've spent that time almost obsessed with methodology, hoping to strengthen this weakness.<p>This is the first post I've seen about the importance of methodology in A/B testing. Given that the underlying distributions can be difficult to estimate, there is a susceptibility to both Type-I and Type-II statistical errors. Consequently, I think this post was much more important then it seemed when looking at the lack of discussion.