Ask HN: How to price a service that saves hundreds of engineering hours?

53 点作者 punkpeye将近 3 年前

I've built a Cypress test runner that completes all tests (no matter how many you've got) in less than a minute. In contrast, based on my experience, a medium-size Cypress test suite takes about 15 minutes to complete.The only catch is that individual tests have to be under 30 seconds.The idea came out of frustration running Cypress in my previous company, where it was taking 10 minutes+ to run all Cypress tests while parallelizing across 30 VMs and costing us in excess of USD 2k/month.In order to achieve this, I have effectively built a new Cypress test runner from the ground up. It understands Cypress syntax, but otherwise have nothing in common with how Cypress works. The way it achieves this performance is by splitting each spec into individual tests and starting all of the tests at once, i.e. if you have 10 specs with 5 tests each, this program will start 50 VMs to run your tests.I have two companies trialing this at the moment and the feedback has been incredibly positive, saying that it is saving _hundreds_ of engineering hours. I am trying to establish how to price this. The challenge is that this model is profitable at scale, but losing money if there is not high density of clients. This is because it costs me USD ~0.15 to run a VM for 1 hour and I need to spin up enough VMs to complete all tests, and I am charged in increments of an hour.My thinking is to charge 2 cents per a test-minute, i.e. Using previous example of 10 specs with 5 tests each, it would cost USD 1 to run all tests once. If you run integration tests 100 times per day, that's USD 100/day.This may sound much. However, if prior to using this you were waiting 15 minutes to run all tests, that is 14 minutes saved. If avg. engineer in your company is earning USD 60/hour, you are saving USD 14 by having engineers get immediate results rather than waiting for them. If positioned that way, it doesn't sound expensive.I am currently targeting companies with 30-50 engineers (existing customers are series A and series B companies). At this size, they don't have crazy amount of tests, so I can deliver on the 1 minute promise, and they care a lot about moving fast.What sounds reasonable?

36 条评论

codingdave将近 3 年前

It is an interesting idea, but your approach to pricing seems off:100/day is not reasonable, at least not for small/medium teams. Trying to do arithmetic on run times and hourly rates to convince people it is reasonable is not only overly simplistic, but actively distracts from the conversation you really want to have - how does your solution solve their problem? People buy when they believe you can solve a problem better than they could without you. So keep the conversation in that space and don't set up pricing that invites nitpicky arguments about hourly costs, etc.Instead, I'd just price based on your own needs - figure out your actual operating cost per test-minute, tack on a reasonable profit margin and operating costs, and bundle up some packages that have a quota of test-minutes per month in sizes ranging from 20/month to 100/month. Make sure that you are profitable at every package, and see if you get takers at that price point. Then let prices increase naturally when they run out of minutes and need to ask for more.

评论 #32453462 未加载

评论 #32453887 未加载

评论 #32453332 未加载

sanjayio将近 3 年前

I think your pricing math is a tad off. I don't think it's harmful to use that for marketing, like you're saving engineering hours, but when you use it for pricing it falls apart. The reason is that you're selling to engineers, and we all know as CI is running, we aren't blocked. So, it isn't a 1:1 minute saved, although, I wouldn't conflate that with zero saved. In my experience a 1-5 minute test run isn't much different, but 3mins vs 30mins is a huge difference. I wouldn't force an engineering team to chisel 6mins to 2mins for example. I would roadmap how to reduce 30mins to less than 10mins. That being said, I think your equation needs more nuance to sell to an engineering manager or leader.

评论 #32453572 未加载

issa将近 3 年前

Just to use your example, you would have a VERY hard time convincing me to spend $3,000 a month to speed up testing time. Running tests isn't exactly dead time. It gives engineers a break, they can think about other things while running them, get coffee, etc. There is DEFINITELY value there, but not $3k a month worth.That said, pricing correctly is hard and requires a lot of experimentation. Growth and adoption might be more important at this stage.

评论 #32453635 未加载

3np将近 3 年前

Do you have a market strategy otherwise?What is your customer base and target market? Both pricing models and numbers will come out very differently for the same underlying technical work depending on if you're aiming for fewer how-much-could-a-banana-cost-biggies (these need service agreements and someone who they can talk to on a first-name basis) or a turnkey system for every startup and corporate using Cypress out there (they just register a credit card and e-mail and start integrating).While you are considering your margins, you're going by value-based pricing here - you focus less on how much it costs you to build and run and your bottom line and more on how valuable it is to the potential customer. Some told you hundreds of hours were saved.It sounds like you have the pricing model for a mass-market SaaS with the pricing for the high-end.One risk with such high value-based pricing is that assuming initial success, some happy customers might get restless after a few weeks/months/years and realize that for the price they're paying you they could actually save money by building and rolling something similar in-house. The customers who are ready to pay those prices are precisely the ones likely to do it, I think. In any case, unless your market is super niche or you have some crazy edge, you will get competitors if you're successful enough. Maybe even straight-up copycats. This is nothing bad at all but it can be good to keep in mind that you may lose customers if they're able to slaughter you on cost-performance.Also consider that these higher-end clients likely care a lot about availability and stability. If you don't have brand-recognition or a personal reference, it can be hard to convince them to take a shot. Free trials will never be enough for them to be confident either. Initial underpricing (not too much obv) is a way to compensate for that initially.But again, I think we need more context before we can make a fair assessment of if the price is reasonable or not.

jacknews将近 3 年前

Your pricing sounds ambitious considering something like Autocad, which is a primary tool, costs about $2000/year, and you plan to charge $3000/month?Since this is essentially a cost-saving thing, I don't think it's realistic to try to charge the cost saved. If I as a customer can save $100/day by using your software, but your software costs $100/day, why bother?

评论 #32455333 未加载

评论 #32454669 未加载

patio11将近 3 年前

At Series A / Series B, they can’t think about serious commitments (and this sounds like one) in increments smaller than $10k. I’d try that and, if you get any pushback, ask what it would have to do to be worth $10k.(A month. Bill quarterly or yearly at their option; raise prices annually to capture increase in use.)Remember that their alternative is not “spin up VPSes” it is “staff an engineering rotation to maintain this service” which will cost them minimally a million annually.

评论 #32453782 未加载

评论 #32453801 未加载

评论 #32453674 未加载

x0x0将近 3 年前

You can price based on eng time saved, but the rejoinder will be that the devs use test time to answer emails, respond in slack/jira, etc. And to some extent that's probably true.I do have some very strong advice on how not to charge: (1) do not charge based on your costs (customers won't care), and (2) do not charge based on fine-grained usage. The latter is important because if a VPE signs a contract for $3k/mo, he/she doesn't want to sit there worrying that devs run a couple extra tests or add a few tests, etc, and boom: all of a sudden the VPE is going to the cfo to explain $10k in overage charges.So: price in broad tiers, and price based on value. If you save hundreds of eng hours/mo, charge for the first 20 hours saved. At $150 per, that's $3k/mo. Multiply by 5 to have a functional company (you can't do sales for under $10k; every successful sale -- which will be 20% if you're very good and lucky -- must pay for all failed sales, all marketing, all dev, etc. Oh, and you're doing sales.) So $15k/mo. That's within the approval authority for a vpe with 50 reports.And accept that bigger companies don't worry about $2k-$3k/mo. So maybe your target should also be smaller shops that can't accept the infra overhead to build this out, but will buy a cheap solution that saves their 1-4 devs a ton of time. Dunno.edit: and you need a strong cross-tenant security story.

评论 #32453489 未加载

Forge36将近 3 年前

Ask your customers. Propose your current amount and be honest "do you think this is fair?" If they say yes great! If not, ask what they think is reasonable. Set yourself a lower limit to keep it profitable for yourself."Sorry I can't price it below X". See how they respond.

评论 #32453556 未加载

评论 #32453593 未加载

holistio将近 3 年前

I think if you find companies who do run integration tests 100 times per day, paying 2-3k per month to save around 500 hours per month does make sense.You really have to work out your ideal customer, though. And find a way to scale it down - you can potentially use the capacity of more VMs spread across many smaller companies.Maybe your business model is having a few clients pay you a couple grand a month, maybe it is having a couple hundred paying you 20-50 or so a month.You do seem to have solved a problem, which is a great first step.

PragmaticPulp将近 3 年前

You need to determine which type of customer you’re targeting.At the high end, some enterprise companies won’t blink at the prices you’re suggesting. However, they will expect a level of customer support and service agreements that are going to make you work hard to deliver what they ask. You’re also going to need to be heavily involved in sales cycles and support, which will start costing you many, many hours just to get the sale.Are you prepared to handle this and stay responsive? If not, you will have mixed results pushing high priced offerings, especially if they have deal-breaker limitations (like your 30 second test limit) that could saddle the customer with even more work to integrate and use your tool. You have to acknowledge that your service isn’t purely savings for the customer. It’s work to implement and maintain on their end, as well as handle billing and so on. You cannot make sweeping calculations about savings without ignoring the costs it brings to a company to add your tool.On the other end of the spectrum, you could target the self-serve market with lower priced services. These offerings can get away with a “take it or leave it” type of service where customers are mostly on their own to evaluate, pay for, and self-support the software. If the tool is good, engineers will try to make it work. However, I suspect you’re massively overestimating the way that test times are a blocker for most engineer workflows. Even with long text suites, we don’t just sit and stare at the monitor while we wait. That’s time for responding to communications, checking other things, taking a short break and so on. You could have a hard time selling engineers on the idea that they can do more work and take fewer breaks while paying for the privilege. Choose your marketing angle wisely.

评论 #32453512 未加载

评论 #32453485 未加载

skmurphy将近 3 年前

Pricing decisions can be subtle. I like to try and calculate the value I delivered to the customer. Here are a couple of questions you may find helpful.What are you replacing? What else could they pay for to achieve an equivalent speed up?What are your enabling? How much time are you freeing up that can be used for other purposes?What is the impact on time to market / release inter-arrival time?What is the shadow backlog of tests that are not being run but now could be?What is the impact on quality: for example time to fix (MTTR) or production problems prevented because of increased testing?You are welcome to schedule a no cost office hours session to walk around your situation: <a href="https://www.skmurphy.com/blog/2016/08/01/skmurphy-office-hours-set-your-own-agenda/" rel="nofollow">https://www.skmurphy.com/blog/2016/08/01/skmurphy-office-hou...</a>

s1k3将近 3 年前

The only way to figure this out is to go to your clients and give them a price and see how they react. You want to get to the point where they are angry about the price but are still paying.This is going to be a lot of trial and error and no amount of analyzing is going to give you a good first answer. So start with something you find reasonable (the pricing you mentioned) and see how it goes with your current clients.If they are totally ok with that then you’re priced too low and on the next sales cycle change it up. If they all balk and threaten to walk away then adjust your price down, until they begrudgingly pay.

clearcarbon将近 3 年前

If you want to target the sme's I might suggest just separating out your value, set a few simple tiers and make it easy to plug a cloud backend in (only needs one provider at first). This way your value and cost is direct and obvious and you're not left managing a fleet of vm's. This probably increases the total cost a little as there's less sharing, but a lot of medium org's can manage that internally and it's much easier to convince a manager to spend $xx/$xxx a month rather than $xxxx or more

jmartin2683将近 3 年前

My biggest concern would be that this is all predicated on the idea that engineers are doing nothing useful (and thus burning time/money) when tests are running, which isn’t usually true.

algo_trader将近 3 年前

> and they care a lot about moving fast.You are thinking too much. Just tell them $5K/month. If they balk re-think your product.> existing customers are series A and series B companiesWith 2 VC funded customers, try getting a referral for cloud credits. If you cant get $10K credits try to find a business-side co-founder.Finally, would you be interested in some sort of experimental "guaranteed minimum activity" swap contract? No cost to you, but you may need to provide analytics. Email my puppet email which i dont check often enough

OJFord将近 3 年前

> completes all tests (no matter how many you've got) in less than a minute.> My thinking is to charge 2 cents per a test-minuteDoesn't that reduce to 2c/run?That aside, my gut reaction was don't charge for time if your selling point is how much time you save.Charge for #tests, or something even better targetted at what makes it slow usually (I'm not familiar) - you charge some (<100) percentage of the reason tests are expensive and make that go away, and it seems like a sound choice to use your service doesn't it?

评论 #32453877 未加载

peyton将近 3 年前

Cost-plus with a low cap and a “call me” for anybody who exceeds.

darepublic将近 3 年前

The tech sounds great. I'm regards to pricing, people aren't machines; you save them 9 min a day per engineer, doesn't mean you get 9 min / workday extra productivity * engineer. And that's not about laziness or anything, good programmers also need time to think about things abstractly etc. I wish you luck in getting properly compensated though what you did sounds great.

Fire-Dragon-DoL将近 3 年前

There is a tool from Nx in similar space, where they cache test results based on code paths.They charge by hours saved, for example they calculate how long it would have taken to run those tests and when they serve the cache instead, they charge money for it.Not sure if it could work in your case, but if it's saving hundreds of engineering hours, charge based on the saved time (and not on the normal runtime)

bluelightning2k将近 3 年前

Note: when running untrusted user code don't underestimate how much of a liability this is.It's so hard to be truly sure that there's no compromise path. Particularly when optimized for speed like you're saying.Security vs performance is often an explicit trade off in these scenarios when it's tempting to reuse processes or volumes or VMs.

vivegi将近 3 年前

1. Why can't you use something like AWS Fargate and spin up the required number of VMs as needed to keep your running costs completely variable? 2. Similar to many other comments here, you should price your service based on value offered to the customer (i.e., how much $ they are likely to save based on some engineering cost model) as opposed to a cost-plus-margin model. 3. Once you have a critical mass of clients and tests running, you can then optimize your Fargate pool with spot instances.Starting with #1 will give you a floor for your running costs and if you tack your other costs (overheads) and a desired margin on top of that you will have a floor price. Then you just need to figure out the value you are providing (and you can A/B test that with prices higher than the floor price you obtain).

nojvek将近 3 年前

We pay around 1500 per year for 8 cores (~10 tests on puppeteer). If your service offers not only test parallelization but also isolation of tests then that is powerful.Charging per test runtime in seconds is also a good idea.As for pricing, look at Browserstack, browserless, Saucelabs. They are well established players.

bluelightning2k将近 3 年前

Why do you need to spin up VMs? You could use a service like fly.io and have these running instantly.This is an interesting project.Is your project so technologically advanced that cypress could never get there? If so maybe you can join forces with them or sponsor them.

colechristensen将近 3 年前

Your minimum price is what it actually costs you, your maximum price is whatever you can get. You can convince customers to pay your price by demonstrating how much your product will save them and charging a discount on that amount.

twobitshifter将近 3 年前

Different pricing strategies:Market Pricing: what are your competitors charging.Cost plus: What is your investment to date and what do you see ongoing marginal costs as, and what type of pricing do you need to make this worthwhile for you financially?nearest substitute pricing: what’s the financial savings of your product versus the competitor? Charge a price that captures some of this value (what you’re proposing)Also be mindful of Cypress possible taking steps to improve their own run times. What will you do to stay ahead and keep your marketshare? include this in your considerations as ongoing r&d.

VoidWhisperer将近 3 年前

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite overall. Does this help improve that?

评论 #32453297 未加载

评论 #32453279 未加载

评论 #32453409 未加载

encoderer将近 3 年前

Sauce Labs is a good comparison. They charge for more concurrency, which really strikes directly at the value prop you are discussing - saving time running a large test suite.

ostenbom将近 3 年前

We've built a similar thing for playwright tests but on cheaper lambda functions if anyones interested! <a href="https://github.com/mentimeter/play-lambda" rel="nofollow">https://github.com/mentimeter/play-lambda</a>

verdverm将近 3 年前

I would expect to be charged per minute, like I do with my current cloud. (Actually they charge per second)You might consider migrating to cloud that supports per second billing.

yapret将近 3 年前

Regarding your scale/customer density problem, you may want to check AWS’s EC2 pricing, which bills by the second rather than the hour.

bb88将近 3 年前

Have you thought about approaching cypress and selling it for $1-2M dollars (after patenting it first of course)

评论 #32454683 未加载

jtchang将近 3 年前

I'm interested in trialing this service. Cypress takes forever. Feel free to reach out.

评论 #32453976 未加载

TradingPlaces将近 3 年前

My $0.02: Price high with generous free trial, and promote in niche places (like HN)

punkpeye将近 3 年前

Inspired by all of you, I've secured an _amazing_ project name. It is going to be a while before I launch it to the public, but this is exciting.

julienreszka将近 3 年前

Sell this to cypress.

joshxyz将近 3 年前

apex.sh is a nice example. it's not for everyone, but it's the best fit for those who need it.maybe give them knobs on the amount of maximum concurrent tasks?