Experimentation (like A/B) testing is a big part of product development in the industry. I see plenty of articles about the goodness of experimentation and was wondering what are the cons of experimentation driven development to keep in mind.
Experimentation has become a cop out in many companies I've seen. I've seen it become an easy way for product managers to show 'they're doing something' and no one would ever blame them for it.
I've seen it used as an excuse for not being creative, hard working, knowing the market and competitors and making big bold bets, but rather to just throw random stuff on a jira board. Later in meetings the experiment 'working or not working' is being discussed without any repercussions for having wasted a whole engineering team's time for weeks.<p>I'm not against experimenting in general, but it has a place. Experiment what you truly don't know. Don't experiment what you know. If you're good at your job you know things that need to happen. Get those done.
It's too tempting to optimize for what you can measure, not what's actually valuable (for your users). "User engagement" measured in "time spent on page" may just mean the page is necessary/useful enough that the users suffer through a miserable layout.<p>See also Goodhart's Law: <a href="https://en.wikipedia.org/wiki/Goodhart%27s_law" rel="nofollow">https://en.wikipedia.org/wiki/Goodhart%27s_law</a>
If you do A/B testing wrong it can be bad (search around that and you’ll find stuff)<p>If you maximize signups you may be fishing with too large a net and catch customers who aren’t appropriate. A/B won’t tell you that.<p>You may end up at a local maxima optimizing things when you should be fixing major real issues.<p>Experimentation should have a hypothesis and a test, not throwing stuff at the wall to see what sticks.
Say you have N + M users and you offer feature A to N users and B to M users and collect data like churn rate on them. If N or M is too small to be statistically significant (meaning either the average is too small in either segment, or the t-test is too close to either average) then the benefit of A/B testing N vs M users was a waste of resources.<p>So when user count is small you need to be story driven. That is to say, you have to actually speak to users (current, future, potential) about why they are using the product, why they walked away, and why they stayed.<p>Basically A/B testing is only useful if you have a large enough user base to be confident in its success. That can be hundreds or thousands of users. A lot of products don't have that critical mass, so you can't effectively be data driven.
Many a/b tests are false positives and fail A/A/B test<p>A/b test completely miss long term effects i.e. more ads= more money and no losses in engagement, but user is getting fed up and in 6 months resentment is going to boil over then all new tests will start showing negative outcomes but cause that triggered it occurred half a year ago.. only way to counter is have users on all possible variations all the time and have enough of them. Not feasible.<p>Only evolutionary ideas and efforts are optimized for not revolutionary. You will just get a faster and faster horse. Low hanging fruit type of mentality. Get caught in vot of mediocrity
><i>"You get what you measure."</i><p>E.g., Facebook optimized for user engagement. Eyeballs on the screen. What they got was a lot of advertising revenue and a lot of extremist hate filled troll clickbait. Meanwhile, the entire world turned against them. So much that Zuck bet his fortune on "Meta" which is looking to be a full fledged dumpster fire.<p>A/B is about fine tuning an existing product. What product you should focus on, I don't believe there is an algorithm for that.