This is philosophically interesting, but little of this really is about _running_ a good experiment. Perhaps it's about choosing which ones to run?<p>Kohavi's book will probably provide much more value than this kind of abstract post. See <a href="https://experimentguide.com/" rel="nofollow">https://experimentguide.com/</a> for more details.
> Good experiments define success up-front.<p>This is absolutely critical. If you're not defining success up-front, you're not running an experiment. You're just doing a staged rollout. You can use the data to craft whatever story you want for most changes.
tl;dr former Slack director of product outlines twelve principles for a/b type experiments with features, products, whatever. (ex Good experiments involve analyzing the results as opposed to 'ship/kill?') Each principle has a couple of sentences of elaboration and calls out the opposite style to avoid.