This article is really weakened by its failure to resolve the initial example.<p>What was it in the initial A/B test that resulted in your 10% improvement in conversion rate to be non-statistically significant?<p>Would applying a simple statistical test to your decision making process have resulted in a different conclusion?<p>There might be a deeper, useful insight in answering those questions, but as it stands now I really don't see anything valuable in this article. If anything it leads me to believe that the author is susceptible to p-hacking (see: <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791" rel="nofollow">https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791</a>)
>Notice how ChatGPT didn’t put its thumb in the air and see which way the wind was blowing? Using a proper tool for analysis not only saves time but also adds precision. This example shows how statistical testing is no joke. Don’t rely on gut feeling or surface-level data.<p>Hold on - you can't just say ChatGPT "adds precision" with nothing to back it up! Acting as if ChatGPT is capable of thinking about your results better than you seems extremely unwise.<p>The real way that companies are doing Data-Drived Decisions wrong is by only measuring one thing without correctly thinking of other factors. For example, a website might find that having a pop-up that says "SUBSCRIBE TO OUR MAILING LIST!!!" might increase mailing list subscriptions, but you are forgetting the effects that has on other parts of your business like the website being slower, customers being less happy, etc.
This is a pretty thin article - as far as I can tell entire article boils down to "apply a statistical test" to ab test results?? The four step process laid out doesn't magically protect you from confirmation bias, for example there are ample opportunities for it to manifest while "identifying the ideal data".<p>The toy example provided is actually a perfect example of why tools (like chat gpt) can't protect you from fucking this up. Why is the view count so different between versions a and b? Was the test set up as a 60-40 split, or had you intended it to be closer to 50-50? Does the massive increase in conversion rate pass the smell test? If anything seems suspicious, you probably should do some further investigations...
It is often said that we have two (or more) decision making centers in the human body. Mainly, one is in the head and one is in the stomach, in the gut. There is a good case to be made that many of our decisions happen instinctively, intuitively, subconsciously, in advance and under the hood, and our brains come up with logical justifications after the fact and tell us what the decision was.<p>Given any model, a researcher usually doesn't just run it once and accept the output (unless they happen to agree with the output of the first run). If they get an "unexpected" result, they will either "reassess", meaning that they genuinely have learned something new, or "develop the model", re-tweaking the model structure and parameters until they get an "expected" output.<p>Most of the value in these data-driven exercises doesn't seem to be deciding between A and B. Any executive can do that (and ask the data scientists to justify the decisions after the fact, as others have said). It's in finding C, which you never even considered and takes you off in a new direction. For that to happen, the human mind and the organization behind it needs to be flexible. And this generation of generative AI certainly seems capable of providing or helping to reach novel insights. Or is it? How can we go off of the beaten track using a tool which is built on top of following the track as much as possible?<p>I like what you said down lower in your comments about looking at data, "Update your gut with an updated map of the state". That's a great description and I think it's exactly what you want from whatever data tool, AI or whatever.
Can you say more about how your example copy change experiment had a confirmation bias? I’m guessing the issue was it was inconsequential and you happened to see an uptick in the initial hours but it would have evened out over time. But hearing more about that would complete the case study.
I'm not sure I understand your point. Are you saying that in the absence of statistical significance, you should ignore the data you do have?<p>I tend to agree that data-driven decision making is often deeply confounded and more association-based than causation-based, but I'm not really sure statistical significance is the right way to solve that in every case.
The post could be more informative, but I 100% agree with the sentiment. Confirmation bias and "rain making" are prevalent in the tech industry.<p>"Rationality is not just about knowing facts. It's about knowing which facts are the most relevant." - Epictetus
I used to know a guy that had a math phd and worked as a data scientist in a hedge fund in nyc (a big one). When I asked him about how his profession works, he often found that management folks that employ him often come to a business conclusion and then ask him to make that data support that conclusion.<p>Given that he was paid stupid money, I would be inclined to do the same.
It's very hard to do anything in a data driven way. Where do you start? Where do you stop?<p>Is it signups that matter? Or DAUs? Or MAUs? Or (and it actually is this one) revenue?<p>But how do you decide the relationship between each of these? Does an easier on-board mean that you get less "sticky" users? How will you tell?
You mean by saying "we do data-driven decision making", and then the procurement manager agreeing to a multi-million dollar CRM investment because the head of sales says so.
Is a Chi-squared statistical test really appropriate in this situation? I’ve never seen one used in this way; I would’ve used a normal distribution to test the hypothesis. At least, that’s what my college statistics memory tells me to do.
For Data Driven Decisions, you must first have the correct model of the universe (or at least the subset of the universe that you are interested in) that you can feed the data in.<p>A vast majority of orgs that claim to be data-driven don't have the correct model