"1. A/B testing requires sacrificing some conversions as part of the experiment."
Yes, because it is a net gain if you do it right. If you want 5 signups today instead of 50 in two weeks you're doing it wrong.<p>"2. A/B testing measures only quantity, not quality."
If you have anything that can quantify your 'quality' of customers then include it in your metrics (I understand this can be hard). If you don't then it is not quantifiable and any analysis of 'quality' won't be based on statistics.<p>I agree on the other points. Overall I think A/B & multivariate testing is a good approach but can significantly be improved.<p>It seems like we use it in 'one size chunks' rather than as continuous improvement. Yes we do it ever 'x' days/weeks.
What would be interesting is some kind of machine learning algorithm applied to A/B testing. It has the success metrics, you set a rotation period and it focuses in on the part that works.<p>You would have to integrate a factor for #5 "Things change" so that the algorithm has the flexibility to adjust to those changes.