Show HN: Open-source A/B testing framework

Wow this looks very polished. (Out of frustration with Optimizely) I created and maintain a couple of A&#x2F;B test open source projects[0][1] but the statistical analysis was always the hardest part so I’m keen to see what you are doing. We’re currently relying on a commercial tool called Analytics Toolkit[2] for this part alone and have been quite happy with it though. The owner is very knowledgable and responsive (no affiliation just happy customers). I wonder if you can adopt similar ideas&#x2F;algorithms into the open source tool. That can be useful I imagine.<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;Alephbet&#x2F;alephbet" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Alephbet&#x2F;alephbet</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;Alephbet&#x2F;lamed" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Alephbet&#x2F;lamed</a><p>[2] <a href="https:&#x2F;&#x2F;www.analytics-toolkit.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.analytics-toolkit.com&#x2F;</a>

Hi. Regular A&#x2F;B experimenter here.<p>This looks like a great tool!<p>Does your system store the stats, or does it trust the stats to be stored in, eg. GA, and then just allow you to analyse them?<p>Is it appropriate to send email alerts when &quot;significance&quot; is reached? Without adhering to minimum sample sizes calculated in advance won&#x27;t this result in a bunch of Type 1 errors?<p>Are the changes to the pages made client side or server side? I think clientside but I&#x27;m not sure. If so are they sync or asynchronous?<p>Thanks!

From the looks of it, it doesn&#x27;t look like the configuration can be stored in the code repository itself. This is one of the key things to do - treating configuration as a code and properly version it&#x2F;blame it etc.<p>Otherwise, this looks great.

I&#x27;m kind of curious how they&#x27;ve tackled the multi-armed bandits and early stopping problems inherent to A&#x2F;B testing but so far I&#x27;ve only found that they use some form of Bayesian statistics using unknown priors and likelihoods (except when you pick binomial I suppose, though the prior is still unknown).<p>They seem to allow filtering drilling down by various categories, which would make statistical significance even more of a concern.

Awesome! Can you comment on choice of mongodb? I admit I have a negative association with it but Im sure there are reasons.

I think you guys really did a good job with something that&#x27;s hard to do well. Kudos

This reminds me of <a href="https:&#x2F;&#x2F;github.com&#x2F;sixpack&#x2F;sixpack" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;sixpack&#x2F;sixpack</a> which I&#x27;ve been eyeballing for years.

Congratulations, that&#x27;s very cool! Do you intend to add new features to expand the scope (i.e. visitor personalization)?

Coupling metrics to experimentation is a huge red flag.<p>You should offload that onto DBT or some other data modeling tools.

Looks promising.<p>Any plan to support Matomo (formerly Piwik) analytics as a data source?

Would this be useful for A&#x2F;B testing mobile app features as well?

How does this compare to Facebook Ax?