A/B Testing to Increase User Engagement

The post has good questions for people to ask, but not specific advice about how to analyze results and decide when you have a conclusion. So I added a link to my tutorial in the comments section. ;-)

You guys suggest using Mixpanel to measure user engagement via split tests, but Mixpanel doesn't have a good way to split test!<p>I've resorted to hacking something together using properties. I use the property name to represent a segment/test name and the values to represent the splits. This leads to having a ton of properties, which the UI isn't designed for.<p>ex:<p><pre><code> {"event":"signup","properties":{"techcrunchreaders/signupbutton":"blue"}} {"event":"signup","properties":{"techcrunchreaders/signupbutton":"red"}}</code></pre>

How do you stub out a community?<p>If the community is engaging, people will be more likely to comment (like HN), but if you are a site, you really don't have two separate communities to swap in/out in the user's mind. Additionally, even if you did optimize your site design/functionality to increase comments, is that really desirable without an engaging community in the first place? Would everything end up being nothing that adds to the conversation? (see Youtube)