At my past few companies, there has been heavy investment into live A/B tests on the customer population. However, for bigger companies that have more exposure, I have a worry that we may not be able to have the proper ways to measure the real cost of an A/B test that may 1) Have negative PR or branding effects 2) Conflict with the objectives of other features.<p>Sure we can measure customer drop off or engagement, but given the number of tests and projects at any given time, there are too many confounding factors to isolate those measures to a particular test. I'd love to hear from Hackernews suggestions! Thanks!
For #1 if customers are assigned to groups that get a certain set of variants for each test and you have enough groups to run the full combination of tests that are active, you should be able to correlate drop off with certain tests.<p>For #2: That's much harder. How do you handle this for larger products that you don't A/B test? I'd bet this isn't a unique issue.
How do you measure PR and branding effects?<p>If you aren't controlling for variables, it isn't very scientific. It's possible to run multiple tests at the same time and test different things, but you need to be able to isolate what you're measuring and be able to measure something that's statistically significant.