I went though sendwithus zero as an unknowing participant in the experiment, and it sucked.<p>At the time, the company I was working at was looking for a service exactly like this - a service where our app could just fling events, and then we'd do all the logic to control what emails got emitted on the basis of those events on the external service (and A/B test them). One of our "growth hacking" guys found SendWithUs somehow, and introduced us. We had a few calls with them - the service seemed like it had potential, and we got beta access to it...which was really sendwithus zero, but sending basic email.<p>We got most of the way though setting up an integration with them, but every time we asked when the A/B testing framework - which was what sold us on the product - was coming online, we were told "it's coming soon, it's what we're working on right now." So we held on for a few more weeks, and gave them some time...but nothing came of it. I was skeptical at the time that they were actually building it, even moreso having read this.<p>The positive side is that while we were being strung along by sendwithus, we found Vero (<a href="https://www.getvero.com/" rel="nofollow">https://www.getvero.com/</a>). They're not perfect either, but we've been on them for 6-8 months, and they've done well for us. They're responsive to our issues, are willing to Skype or chat when we try and develop some sophisticated campaigns on their platform, and have (or so it seems) rejigged their roadmap a few times to add in small features that we ask for. ExactTarget is still a more full-featured package, but I'm happy with the price/value of Vero.<p>So well done, guys. You've got an interesting story about how you bootstrapped your business, and you're evidently doing OK. I wish you the best, but I can tell you that you won't be getting my business (or my recommendation) - at the time I thought you were small and were actually on the cusp of delivering what you had promised us worked; now I see that we were just one more data point for a feature set that you never planned on having ready when you claimed.
> We spent the next 24 hours getting as many people as possible to view and interact with the experiment<p>OP - could you elaborate on how you got these 803 unique visitors within 24 hours?
Does anyone else find this kind of pre-development marketing kind of scummy and deceptive?<p>It's basically lying to potential customers: You're pretending to offer to sell a product that doesn't exist.
- Just because some people clicked on features they want, it doesnt mean they will pay for it once you launch.<p>- Just because some people are using your free features, it doesnt mean they will pay for it once it is paid.<p>- Just because you have 5 paying customers, doesnt mean it will be easy to find out 500 more.<p>- Just because you have 500 paying customers, it doesnt mean what you did to acquire these 500 will scale to the next 5000.<p>- Just because you have 5000 paying customers, it doesnt mean that there won't be a new upstart which will dramatically undercut your offering and acquire most of these.<p>There is a name for this validation process - its called a 'startup'.
Are you sure the value you got from the metrics exceeds the bad will generated by annoying your potential customers?<p>To me it's a lack of respect for your users and their time.
"We knew that transactional email was a problem worth solving" A story worth telling in it's own right I believe, when clamoring on about lean methodologies. Nice article would love the pre-phase lean story leading up to this stage.
Relevant similar story: I had this site for years, <a href="http://paycheck-stub.com" rel="nofollow">http://paycheck-stub.com</a><p>Developed it back when I was doing microsites, and I wasn't really sure what to do with it. My main question: why do people <i>visit</i>, anyway? I figured I'd know what to do with it once I figured out why people were there.<p>So I did just what these guys did. I created a list of buttons down the page with all the possible things people visiting a site called paycheck stub might want. Hooked them all into Google Analytics. Paycheck templates? Fake paychecks? Loans? Payroll services? A job? Sample paychecks for school or presentations?<p>Very interesting results. First, about 90% didn't want any of that. God knows why they were there. But out of the other 10%, I got a couple pieces of useful information. They were looking for payroll services. So I put up some banner ads. (I think I took down the Adsense. Can't remember)<p>That, in turn, provided more ad revenue than simply using Adsense on the site. It also provided me with a list of topics to develop and some search terms to try to hit.<p>So I developed those topics and tried to do better on those search terms. At one point I was almost ready to jump into online payroll. Fun times!<p>Then Google killed my search ranking -- I wasn't paying for links or spamming. Beats me why. I got so frustrated with them I gave up.<p>Still get quite a bit of traffic, though. But now, I'm sure, the traffic pattern is completely different. I wonder if, for static sites with a lot of content, this is something you should repeat every year or so?<p>Like these guys, I found it a rewarding experience, both for my understanding of the site and for my readers. The reason I'm posting is that many times we view this as a step on the way to creating SaaS, but it can be just a great way to connect with your readers. I bet it's extremely applicable for a lot of blogs.<p>Wish I could have figured out why those other 90% were there.
> We knew that transactional email was a problem worth solving and we were determined to build a great solution.<p>I have a problem with this phrase. Seems to me it needs justification maybe just a verb? I don't immediately think of "transactional email" as a problem...
I did that with my father's computer store website back in 1998. We had computer descriptions, a price, and a 'buy online' button. The button just had a hit counter on the page, with directions to the store. Every hit meant someone wanted to buy a computer through our website and couldn't. I showed him the data that we could double our sales just by making an online store, but he was not convinced. In any case, the cash flows of that business wouldn't have worked out for him. Still, I thought this was a widely known technique.
The apparent origin of the idea, the "button to nowhere" that they linked to in the story is also a good read: <a href="https://medium.com/design-ux/77d911517318" rel="nofollow">https://medium.com/design-ux/77d911517318</a><p>From a user perspective, I think there are both good and bad ways how this technique can be used:<p>I actually like the way Sendwithus used it - as part of a very very early "pre-alpha" version where you're not yet interested in providing a reliant service and just want to get a feel for the demand. As a user you're informed early-on that you're looking at a version in development - so you will not be surprised by missing or incomplete features and will maybe even motivated to provide additional feedback.<p>The downside of this approach is of course that the data you're gathering is really only good for that - an early overview. Your test group will likely be very different from the actual people who will later use the service. The usage patterns will be as well.<p>The second way to use the technique is how Nick Kishfy describes it in the "button to nowhere" article. In that approach, "buttons to nowhere" are not just inserted into an early preview/development version, but into the actual product as a continuous part of the development process.<p>This would solve all of the above problems and would give you a much richer set of data over a much longer time span - but from a user perspective that would be the worst thing possible.
<p><pre><code> We knew that transactional email was a problem worth solving
</code></pre>
Really? Hasn't this been solved already, by many businesses (SendGrid, Mandrill, SES, MailJet) – I'm happy you've managed to build something and are seeing some success, but I don't think you're solving a problem that hasn't yet found a solution.
What's the story with using logos for integration (SendGrid, AWS etc.) without actually supporting them? Is it not possible to get into legal issues there as it's quite an effective boost to the site to make it _look_ like you're associated with very reputable sites?<p>Or.. just something to worry about when people come knocking? :)
We are going through the same approach with our current beta customers (and will be making our demos available to a wide audience in the future too) and I can attest to the validity of this approach. We make it very clear to our customers that this is a demo and their changes will go away if they refresh the page. We don't have pop-ups integrated since we collect emails on another page. I think event tracking should be sufficient but we'll have to see what the data says in a little while.<p>We actually also use these semi-live demos instead of mock-ups (since we'll be implementing the same pages afterward anyway) since only one of us is proficient with Photoshop but everyone is comfortable with Bootstrap. It serves as a testing ground for various approaches for JS code too.
I'm wondering if the A/B testing score was skewed due to it being the first non-working link in the menu. Someone going through the menu to see what's possible will probably take a look at the top item first.<p>A random order for the menu items might have been better. Or maybe not.
Awesome way to design a product.
I wonder though, how the users reacted when they saw "dead end features". Were there any frustrated users when they saw it was mainly a "mock-up" web site ?
Obviously smart and handy way of getting metrics for a wannabe startup but as some comments states I would feel cheated as no real product are offered, but offers are pretended to be real. Thus wasting my valuable time.
I bet this tactic will backfire soon as more and more try to do the same. I would be more positive if some incentive were offered, e.g. Thx for your interest get 3 month for free when we launch ...
Can't decide how I feel about this. Does anyone feel that this type of testing is kind of dishonest? On one hand, listing a product as available with a price is a great way to gauge interest. On the other hand, telling someone it is "sold out" when it was never available feels wrong somehow. What do you think?