I recently implemented A/B testing on a client's site using one of these Javascript-based A/B testing tools (but not this one).<p>I hadn't used one before, so wanted to verify the data would actually be accurate.<p>I did an A/A test, basically testing the same exact page––expecting the results would be the same.<p>Not only were the results not the same, but they were off by a wide margin.<p>Given this, I don't know how I'm supposed to trust any of the data.<p>Has anyone else had experiences like this? Is A/B testing in Javascript just not as reliable?
8) Have a hypothesis of what you're testing and control for variables. Run a MVT test if you're changing a lot of things. If the test wins and it's implemented, everyone is happy and people don't ask too many questions. If it loses, what have you learned? Test a hypothesis.<p>If a client looks at a comp for a test and asks to change something, I always ask them, "What hypothesis are we testing with that change?"