Good article, and I generally agree that people are too quick to say “Ahh, this looks like a two week project” and ignore broader complexity. I’ve definitely seen a fair amount of that.<p>My question is, how do I know that the OTS solution would actually be much better? For example, OP describes some tricky migrations from one type of billing to another, or complex grandfathering schemes - how would you have any guarantee that your third party solution would be able to support those? At least if it’s in-house you can implement the needed functionality <i>eventually</i> - if it’s third-party it might just be flat-out impossible.
I've built several billing systems and managed a team that built a much larger one, and it's not <i>that</i> hard, but it does require an attitude and attention to detail that a lot of people lack.<p>E.g. one thing that worked well in the systems I've worked on was to break things into <i>small</i> state transitions, <i>very</i> firmly document the allowed states and their transitions, and <i>log every transition</i> to the database. Wherever possible (anywhere that didn't interact with external API's, and sometimes even when they did) we'd aim for state transitions to be idempotent. When they weren't, we'd seek to reduce the non-idempotent call to an external dependency to <i>just</i> that call as a separate state transition.<p>When something very occasionally went wrong, we could trace things in detail, and pinpoint it, and most of the time we could safely replay transitions around whatever went wrong until we could see what had happened.<p>It wasn't <i>difficult</i> to build these systems that way, but it was <i>tedious</i>, and something where it's easy for people to be tempted into shortcuts.<p>As for scaling, you spit out events for rollups, and can shard invoice generation. We handled millions of revenue on a machine far less powerful than my laptop today. Scaling really would not have been up there for me.<p>To #3, maintaining old pricing isn't generally <i>that</i> hard. Very few people drastically change how they account for usage, which tends to be the most complicated billing scenario. They tend to change amounts and periods and thresholds, which should be data, not code.<p>With respect to a team, I agree. One of the places I worked on billing was Yahoo. At the time it wasn't just one team, but several - my team (responsible for Europe) existed largely because the European finance and product teams didn't trust the US payment services team to take their requirements seriously enough and so wouldn't let them near the European premium services... Billing isn't something you want to do yourself unless it's either a core competency or you're big enough to have to deal with that kind of bullshit.<p>(As for size of teams, I've built a billing system with 2 people, but I've also worked on billing systems managed by 50... You really want to have a good idea whether you're likely to end up towards the former or latter before you go ahead...)
Having gone through /exactly/ what this article is about, I know the pain. But I will point those that don't read the article to the last paragraph:<p>> We considered implementing an off-the-shelf billing solution but there was nothing flexible enough and the switching costs were too high. Algolia also tried to migrate to Zuora before backing out and rebuilding their billing system for the fourth time.<p>Most (all? I have yet to find one) off-the-shelf systems are not geared towards everything businesses want to do with their billing. If you take the author's advice and choose one of these systems early on, you will have the exact same headaches. Either the system simply won't let you do what you want to do or the system /will/ let you do what you want to do with custom or hack-ish solutions that reproduce the custom-solution problem, but in someone else's system. Those that implement broad swathes of billing functionality are so complex, they make /everything/, even the most basic stuff, hard.<p>BTW, when I say I know exactly how the author feels, all of what they described we encountered and implemented. We even looked at a migration to Zuora and came away with the same conclusion (also: really freaking expensive). We even had a new product that we setup on a third-party billing system, and we ran into the flip-side of the problems; we were not able to implement some billing functions we needed and we had to migrate off.
Billing in general is very hard. Especially subscription billing. Just from my experience in building billing and also later on using a third party system for another projects, I can mention so many edge cases:<p>1. Grandfathering<p>2. Upgrading across different length memberships (monthly tier 1 to yearly tier 2)<p>3. Crossgrading across different lengths (monthly tier 1 to quarterly tier 1)<p>4. Offering free trial to tier 2 when user has tier 1.<p>5. Promotional offers to non-paying users (half off first month)<p>6. Downgrades<p>7. Applying coupons to accounts (customer support wants to offer a specific user a free month of service when user has a quarterly subscription: billing schedule change)<p>8. Differences in tax regulations across countries (VAT vs whole value tax)<p>9. Differences in display prices (in some countries displayed prices should include tax)<p>10. Handling currency exchange rate changes.
Why do I have the feeling that many people starting projects like this get in to situations where functionality is identified and they're beat down with "YAGNI!"? I've been in situations where 'build v buy' comes up - accounting, inventory, and similar domains. Identifying "hey, we need to keep XYZ data to allow for future reporting..." has been met with chants of "YAGNI" from people blind to the complexities.<p>It's just one of the things that might lead me to opt for 'buy' in 'build v buy', but... it's hard to trust the sales people involved in the process as well. And... taking the time to do <i>real</i> and <i>full</i> analysis... is time people often don't want to pay for.<p>And... once you're "in" to a COTS system... you're almost never leaving. The companies know that, the salesfolks know that, and have a lot of incentive to handwave away concerns, or just outright lie. Once you discover the lies... it's likely too late to switch.
Like with everything, wrap your billing in a layer and when you grow enough you put your billing under it if you need it or switch to a different provider without changing your codebase.<p>It's not wise to outsource parts that are essential for your business, because:
1. you depend on 3rd party service for a getting paid
2. your operations cost may rise or the service may change terms/functionality
I would instead say - commit properly to building your billing systems as a major and ongoing project, don't dismiss it as a 2 week "fun" side thing.<p>Not convinced any 3rd party product is going to have the flexibility to support ongoing innovation. If anything it's more of a case for in-house expertise and resource. Find some folks who <i>want</i> to be the billing guys!<p>Disclaimer: I built a large in-house billing system :-)
In my experience this happens frequently enough that there ought to be an article like this for every problem domain that software developers and executives under-estimate the complexity of.<p>I work on card transaction processing and it's absolutely more complicated than you can imagine looking at it from the outside. It's a vast, distributed peer-to-peer system where trust is built into liabilities outside of the network itself. You want to build a system that is correct with regards to some specification in order to protect peoples' money... but the problem is that there is no specification that enumerates every possible transaction state because while there are "typical" sequences of messages to handle, your system also has to handle unexpected sequences that can't be specified: there are no guarantees that systems on the other end are emitting properly formatted events let alone behaving as expected. Card payment handling systems have to have the ability to manually correct and adjust state by humans; reconcile with external systems (because yay, payment systems in the US aren't based on instantaneous settlement until the roll-out of FedNow is complete), etc.<p>I've watched many businesses walk straight into, <i>it's just X, how hard can it be?</i> Only to watch their ARR shrink and stress levels rise when a project they thought would take a month turns into a year-long journey of discovery, reflection, and increased head-count.
The point the article makes to me (as a leader) is “keep the billing model simple, honest and stick with it”. The cost of change is high, and the value isn’t to the people getting the work done. My peers would likely see this as “naive” and extoll the need to “sell to the buyer”. I see it as being value focused, and keeping the “business needs” rational, and in line.
I'm amazed anyone would ever want to DIY something that touches money.<p>People tend to get pissy about money. Almost as much as water plant scada systems and railway safety. Maybe moreso.<p>And if you're dealing with money, you're already inherently doing cloud stuff, because you're talking to the payment services, plus you already by definition have money to pay for a real billing system...<p>It seems as crazy as people on r/wallstreetbets advising their parents on how to put their retirement money into hand picked stocks or something.
I've worked on telco billing systems in the past and the generally accepted metric in the industry is 70% of billing systems projects fail. Not fail as in "did not completely meet requirements or expectations", fail as in "did not deliver anything whatsoever".
I built my billing system on top of Django and it's used in 2 online services for a few thousand users and hundreds of invoices a month.<p>You have to be mindful of some edge cases, but if the scope is kept narrow, it's doable. The added benefit is the flexibility to fit it to your use case. And with Python, you get many high-level libraries, e.g. for decimal calculations and PDF generation.
I've been working on an simple invoicing app for small businesses for over 20 years. It has become a somewhat complex project and my app does not do much of what's cited in this article.<p>>> Now, when someone asks for advice about their billing system, my answer is clear: DO NOT build it yourself.<p>Moving from using CGI.pm to CouchDB/PouchDB about 10 years ago made it much easier to manage users and their data, and implement the UI. It also moved most of the workload to the user's web browser and made the app much faster for users.<p>CouchDB was practically designed for this. Early on their developers used "invoicing" as an example for using CouchDB.<p>But... if you have a marketing team that keeps moving the goals it wouldn't matter what you built the app with, it's going to take time to build and debug it. Could be that CouchDB/PouchDB could make that easier, but my experience is developers who've only been using SQL may get frustrated with it.
Worked in mobile Telecoms billing back in the relatively early days.
Remember discovering that Orange at the time made a good feature of competing (and acquired lots of new customers) by offering "competitor match" billing options. They made this as a business choice, but it cost them a team of over 100 contractors developing that system.
Other telcos had packages but were constrained by what the packages offered as to what their marketing could offer.
You pays your money and makes your choice!
p.s. things learned the hard way in those early days - customers realised that a call held open for over 24 hours resulted in a reset of billing due to counter wrap around!
I wrote a billing system myself, and my first impression was that it couldn't get past 1000 lines of code. However, it had become a ~5000 lines monster and keeps on growing. Never underestimate details and edge cases.
Billing is one of those problems that seem deceivingly easy. Just talk to someone who has worked in telco billing and learn about the 4 horsemen of billpocalypse: metering, mediation, accounting, billing.
I worked for a mobile phone operator many years ago. I was there when the company was bootstrapped. We built our own billing system because we would be billing for stuff that incumbents didn't bill for, because they didn't offer those services: we were the first 3G operator of the country and we also had to create and sell contents (example: YouTube was not born yet.)<p>I was not in the billing team but we definitely followed rule #4 of the post: "You must be prepared to staff an entire team".<p>And yes, #1 "Pricing changes all the time and billing needs to follow".<p>About #2 "Your billing system needs to scale with your user base", we had to go from 0 to 3 million customers in 9 months to be viable. We were not typical, we made it.<p>No idea about #3 "Grandfathering causes headaches" but there were probably many headaches, that one and others.
It looks like one core issue was designing the system around pre-determined bounds (monthly, yearly, etc). This happens in lots of systems and isn't specific to billing. A job scheduling system will have the same problem (we built it to track seconds and now they want us to track calendar dates!).
When we launched in 2017, we went "in-house" with our billing. We found an open-source project and uses that for all billing as it had a recurring option and a billing portal. Fast forward a number of years, we now sync invoice data in the app and have a tiny billing portal there. We can still adjust invoices, keep old pricing, setup new pricing, auto-bill, check balances to see if a customer hasn't paid yet. Total Lines of Code here are like under 2000 for this integration and it's great. Also zero vendor lock in as well. Just another way to approach a billing system.
Now take your system and make it multi tenant. Imagine the horror of doing that. This is what you often get with "off the shelf". There's a lot of broken dreams either way you go.
Should have a (2022) label <a href="https://web.archive.org/web/20220715000000*/https://www.getlago.com/blog/the-4-biggest-problems-with-homemade-billing-systems" rel="nofollow noreferrer">https://web.archive.org/web/20220715000000*/https://www.getl...</a>
Why are we against modifying our processes to align with a particular software platform? Maybe it is much easier to follow such guidelines during the early stages of a company, but much harder later on. Do standards exist in billing systems?
I was waiting for them to discover the rabbit hole that is 'day counters'. (I wrote a lease payments calculator for the financial services industry. Many gotcha's lurk there.)
That's why you hire Product and UX designers so the engineers don't run wild into endless attempts to find an initial solution through the rabbit's hole.
The number backend engineers with competency in recurring revenue accounting is too small. Strange in our world that is more and more subscription driven.
Another problem with billing systems I've seen repeatedly is that the billing demands flows one way; marketing or product management basically scribbles on a napkin how they'd like to bill the customer, and that napkin gets passed to the billing team as a specification carved into stone. Basically, in practice it gets run as Waterfall, actual factual Waterfall, except the design phase is skipped. It goes straight from features to implementation.<p>Now, on a business level I acknowledge that frequently the monetary costs of the billing system are negligible compared to the business advantages of the "correct" billing system. My complaint is more that the scribbled note often ignores what the rest of the company is doing for billing, e.g, "yes usage based billing is great but do you <i>really</i> need to bill on furlongs per fortnight when the rest of the company is billing based on seats or some basic, easy-to-understand resource allocation?", ignores that maybe there's an easy thing that's <i>close</i> to what you said but since it reuses all the existing stuff you can have it in two weeks instead of four months, etc. Do you really need to issue a discount coupon that takes off a percentage proportional to the length of the customer's first name, unless the customer is in a jurisdiction where that's illegal in which case we roll a die, unless the customer is in a jurisdiction where <i>that's</i> illegal in which case we give them $10 off their first $100 and take off every prime-numbered dollar after that? Because I've got code that just takes 10% off their first month right here. Do you really need to bill on the first of the month unconditionally when the rest of the company bills based on subscription start date, or vice versa? Even if you're not too worried about the costs of paying 6 months of developers, I bet you <i>are</i> worried about those 6 months of opportunity cost.<p>I can't even count the months of delays caused by treating napkin scribbles as carved stone I've personally witnessed, and I'm only tangentially involved in billing over all.<p>And somehow managers who are deeply familiar with costs and benefits and tradeoffs and make sensible decisions all the time just become the most obstinate customers when it comes to billing. No matter how small the tweak proposed and how many <i>months</i> forward it may bring your release date to do something just <i>slightly</i> different (and consistent with the rest of the company's existing policies), they will go to the wall for their billing deviation, even when it was frankly clearly nothing more than a whim or a transient thought at some point by somebody somewhere of no strategic or marketing consequence.<p>It isn't just engineers who underestimate the complexity of billing until it's too late; it's everyone, really. "Just" print an invoice turns out not to be so simple.