Excellent point. Blindly following A/B testing can lead to a good local maximum, but odds are it's not the market maximum. Radical redesigns and new business directions can lead to even higher (or lower) conversions.<p>In my experience, the trick is finding the balance between optimization and strategy. Doing continuous testing along the way helps you find out what's working and what's not.
I think the point "blindly following A/B tests is bad" is valid, but I don't completely follow the path this article paves to lead you there.<p>For part one, regarding forecasting, you were A/B testing within the segment you were targeting, whether you realized you were targeting that segment or not. You saw improvements within this segment, but were further alienating a blind spot. Strict, purely statistical A/B testing isn't a great mechanism for testing out segmentation strategies, business models, etc. A/B testing isn't a strategy to get you more oranges, it's a tactic to get more juice from the oranges you already have.<p>For the second part, regarding the map, I'm still not sure if this is a good candidate for A/B testing, or if the implementation of the A/B test really isolated enough variables to reach a business conclusion. I suspect that may be difficult. Critical portions of your user interface are most likely another area where A/B testing is a bad idea.
Well done; A/B testing is valuable, but under a very specific set of circumstances. The true purpose of AB testing is to say, "Does my conversion go up or down if I make that button green?" If it goes up you keep it. You put in some variables, and A/B testing can tell you which varables will produce the best result. But it's important not to forget that the results of these tests were (intentionally) limited to certain variables.<p>The reason being a founder is so hard and the reason your intuition as a founder is so valuable is because answers that require finding and solving pains can't (yet) be mathematically found. To us a very tired analogy, A/B testing will tell you people want a faster horse instead of a car.
I suspect many startups follow this pattern:
1. Launch with flawed hypothesis.
2. Confused or disappointed users say what they really want.
3. Pivot that leads to product market fit (in Seatgeek's case, launching stadium maps, Columbus, etc).<p>We went through that at GiftRocket. We launched with the premise of geo-located gift cards. Of those who tried, many had bad experiences. But they gave us enough direction to kill the GPS component and turn GR into a well-packaged online way to send money as a gift.<p>Funny enough- PB told us to do this before we launched (the same way PG identified the issue for Seatgeek).
Consumers don't want to think about fluctuating prices. Especially if it means there is a chance they bought too early. They want a price today that's better than the market.<p>Scenario 1: The Broker<p>I want to see Swedish House Mafia but don't want to pay more than $400. Inversely, I bought Beyoncé tickets, sobered up, remembered that I hate Beyoncé, and would like to get at least $300. Your give the customer a probability of the ticket being sold. Perhaps also the option of resetting the price if the probability falls below X.<p>Scenario 2: The Dealer<p>Consumers buy and sell a small number of tickets. If you bought(sold) tickets when your model said they were likely to go up(down) by a margin in excess of your error you could diversify across multiple event types, venues, artists, and dates. You would be exposed to model (transformed basis) risk and would need to finance inventory.<p>Scenario 3: The Market Maker<p>Derivatives! :D The simplest way to "do arbitrage without ever holding the tickets" would be to sell tickets as forwards. To illustrate, let's suppose prices are at $500 and you believe they will fall to $200. You sell a "ticket" at $250 to a customer and receive those funds today. The night before the event you buy the ticket at $200 and deliver it to the customer, pocketing the $50 plus interest as your spread. You're still exposed to model risk, but with the benefit of float, i.e. cash today for a deliverable tomorrow. Bonus: easy shorting.
A/B testing does not tell you what to do. It measures what you have done. HUGE difference, but a difference that very few people actually seem to understand.<p>The last paragraph of this article made me shake my head though. If you do A/B tests and then use the losing version anyway - because it feels better in your gut - you're an idiot.<p>Choose a different path if you want, but if done properly, your A/B results are fact (among the options you tested).
I think most folks building websites <i>should</i> blindly follow A/B test results. There are no doubt situations where it's best not to, but introducing that possibility means there are too many opportunities to make judgement mistakes.
If you go through the trouble of setting up an A/B test and determining a statistically significant result, you're not <i>blindly following</i> it when you implement the winning variant.<p>It's totally cool to disregard the result an A/B test gives you. But don't justify that decision by saying that if you do follow the results of well-run A/B test, that it's somehow blind.
Remember that A/B testing is a tool for refining the local maximum of a specific design. It works wonders when you are at scale (with some sense of product market fit) but is not a cure all at a small startup.<p>Going out and talking to 15 of your target customers in person will get you more learning at an early stage than a month long A/B test
Notice how nobody was interested in whether Marissa Mayer had any feedback for SeatGeek on the video of their pitch. I suspect that question session would work out quite differently today.
I watched your pitch video, and it indeed was painful. You didn't have the answer to an important question, and you still don't. "We don't want to be a hedge fund" is not an answer. If your prediction software works, you are sitting on a gold mine, and you DON'T need a huge amount of cash.<p>If ticket prices fluctuate 40%, and you're right 80% of the time, and half the time the prices are going up, your potential gain is<p><pre><code> 0.4 * 0.8 * 0.5 = 0.16 = 16%
</code></pre>
Which is twice as good as your fee margin of 7%.<p>EDIT: And what's more important, your volume is not limited by the percentage of your site visitors who make purchases, you are limited by your money, which will grow exponentially, more or less. 32% average profit (minus fees) on each buy/sell is fucking HUGE, it's unbelievable you're not doing it.