Learnings from 5 years of tech startup code audits

That&#x27;s a very interesting set of findings. What is important to realize when reading this that it is a case of survivorship bias. The startups that were audited were obviously still alive, and any that suffered from such flaws that they were fatal had most likely already left the pool.<p>In 15 years of doing technical due diligence (200+ jobs) I have yet to come across a company where the tech was what eventually killed them. But business case problems, for instance being unaware of the true cost of fielding a product and losing money on every transaction are extremely common. Improper cost allocation, product market mismatch, wishful thinking, founder conflicts, founder-investor conflicts, relying on non-existent technology while faking it for the time being and so on have all killed quite a few companies.<p>Tech can be fixed, and if everything else is working fine there will be budget to do so. These other issues usually can&#x27;t be fixed, no matter what the budget.

&gt; Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering.<p>I wrote about this before that as an industry, we have made writing software complex for complexity&#x27;s sake.<p>&gt; imagine, in a world where there wasn&#x27;t a concept of clean code but to write code as simply as possible. not having thousands of classes with indirection.<p>what if your code logic was simple functions and a bunch of if statements. not clever right, but it would work.<p>what if your hiring process was not optimizing for algorithm efficiency but that something simply works reliably.<p>imagine a world where the tooling used by software engineers wasn&#x27;t fragile but simple to use and learn. oh the world would be a wonderful place, but the thing is most people don&#x27;t know how to craft software. but here we&#x27;re building software on a house of cards [0]<p>:[0] - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30166677" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30166677</a>

I&#x27;ve seen most of the architectural problems in consulting - it&#x27;s amazing how a team of clever engineers can take a simple thing and make it sooo convoluted.<p>Microservices, craptons of cloud-only dependencies, no way to easily create environments, ORMs and tooling that wraps databases creating crazy schemas... The list goes on and on; if you&#x27;re early, you could do a lot worse than create a sensible monolith in a monorepo that uses just postgres, redis, and nginx and deploys using &#x27;git pull&#x27; ...

&gt;&gt;&gt; Business logic flaws were rare, but when we found one they tended to be epically bad.<p>oh yes ...<p>I always bang on to my junior staff that their job was known as &quot;analyst programmer&quot; for a reason. The analyst part matters probably even more than the programmer part. In large companies just <i>discovering what needs to be coded</i> is 90% of the job, (the securely coding it in the constraints of the enterprise the other 90% while the final 90% is marketing your solution internally)<p>Anyway .. yes

&gt; Generally, the major foot-gun that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs.<p>Finally, someone said it

Its kind of discouraging to see the part where he says almost no one gets web tokens right the first time. Working on projects as someone entering the industry, its pretty clear that security is the most important part of a web app, and its so seemingly easy to get woefully wrong, especially if you’re learning this stuff on your own and trying to build working crud projects

This is an interesting write up!<p>The only question I have is around your point on monorepos - every monorepo I’ve seen has been a discoverability nightmare with bespoke configurations and archaic incantations (and sometimes installing Java!) necessary to even figure out what plugs in to what.<p>How do you reason about a mono repo with new eyeballs? Do you get read in on things from an existing engineer? I struggle to understand how they’d make the job of auditing a software stack easier, except for maybe 3rd party dependency versions if they’re pulled in and shared.

Once I worked on a team that none of the engineers knew that jwt payload was readable on the frontend. They were in shock when I extracted the payload and started asking questions about the data structure.

&gt; For example because it’s so fast, [MD5 is] often used in automated testing to quickly generate a whole lot of sudo-random GUIDs.<p>Actually, it’s because programmers are lazy. GUIDs or UUIDs are 128-bits and MD5 produces 128-bits. A string like “not-valid” is not a valid UUID, but MD5(“not-valid”) is both possible to format like a UUID when output as hex (with dashes) but also self-descriptive - so you can name the token when generating it in a fixture function and know how to regenerate it later in a test, for example.<p>All the normal ways of generating UUIDs, including v6 and v7, are about trying to make them unique and collision resistant. But that’s nonsense when you want deterministic, reproducible tests. Hard-coding 32 characters is too much work, ain’t nobody got time for that. Magic numbers? Pfft. Just MD5 and write your own text…<p>Pro tip: have data model creator helper functions include a counter that resets every test (every time the database resets) and then assign a UUID like MD5(`InsertTableName-${counter}`) that way you have a unique ID that’s also easy to predict&#x2F;regenerate.<p>That said… I’ve always personally preferred simple database IDs to be generally preferable over using UUIDs. It’s easier to understand THING 20 as an ID than 32-odd characters. But UUIDs are an industry standard, so they end up in your test code everywhere anyway…

At what point is there something beyond a framework - a SaaS in a box perhaps, that just avoids many of these basic problems (oh and the HR, legal, etc problems) of starting a startup. Startups are not snowflakes apart from that one little core competency. In short most serial founders say the second one was easier, simply because they followed the template ground out in the first.<p>Would it be easier to start with that template?

Does <i>learnings</i> ever mean anything different than <i>lessons</i>? How did this enter corporate-speak?

Nit: &quot;...or example because it’s so fast, it’s often used in automated testing to quickly generate a whole lot of sudo-random GUIDs.&quot;<p>ITYM: &quot;pseudo-random&quot;<p>Although I do like the mash-up concept of &quot;sudo random&quot;-ness.

&gt; <i>All the really bad security vulnerabilities were obvious.</i><p>All the really bad security vulnerabilities <i>that were found</i> were obvious?<p>One is more likely to find things that are obvious?

Although I strongly agree with it in principle, I&#x27;m growing seriously tired of the &quot;simpler is better&quot; argument. It hides all the nuance, hard work and, guess what, complexity, that goes into making something simple.<p>Simplicity is different to each person. What seems like unnecessary abstractions with complex inner workings often exist to actually hide other complexity away.<p>Know the in and outs of Kubernetes? Maybe it&#x27;s easier (simpler) for you than directly provisioning different pieces of infra.<p>Have a team of over 10 [1] working on the same monolithic codebase? Productivity while maintaining sane separation of concerns might increase going for a more domain-service-oriented architecture [2].<p>How can we teach what simplicity is instead of just calling it better or saying arrogant platitudes like KISS?<p>[1] yes, the number is that low, and often lower [2] yes, &quot;micro&quot; services does seem like a mistake in most cases

Ah, yes, I’ve felt the pain of an unnecessary microservices migration. It ate time for years and the core was still a mess

&gt; Custom fuzzing was surprisingly effective. A couple years into our code auditing, I started requiring all our code audits to include making a custom fuzzers to test product APIs, authentication, etc.<p>Any recommendations for a good fuzzing tool for testing both web-based APIs and language specific APIs (C and Java in my case)?

&gt; Surprisingly, sometimes the most impressive products with the broadest scope of features were built by the smaller teams.<p>Would probably not be surprising to Fred Brooks author of the _The Mythical Man-Month_, but as much as we think that book is famous&#x2F;impactful, it still surprises us!

&quot;All the really bad security vulnerabilities were obvious.&quot;<p>I used to work for a company that did a <i>lot</i> of acquisitions and I often involved in working with teams at newly acquired companies - although it wasn&#x27;t my main focus I did used to ask some simple security questions and it was remarkable what these uncovered. I literally had people run from the meeting to fix services after I had asked a simple question....

Interesting that he feels the default state of software security has improved a lot in the last few years.<p>Anecdotally I&#x27;d also agree with that. Certainly better defaults and more secure libraries is a major factor. I haven&#x27;t noticed a huge increase in developer security awareness, although I&#x27;d say it is also better than 10 years ago.

yes. When I get called in as a senior consultant for some business app, it&#x27;s always for the same reason: development speed has crawled to an almost stop. And it is always caused by unnecessary complexity.<p>I blame the fact that design patterns and specific architectures are being taught to people who don&#x27;t understand the problem those things are trying to solve and just apply them everywhere.<p>Any senior dev or architect should always live by this maxim: make it as simple as possible.

Great article.<p>To expand a little on why “Keep It Simple” is so powerful: less code = less bugs and less security issues. Less code = easier to change.

Thanks for writing this down @Ken. You&#x27;re another example that learning the failure modes is the main benefit of being a consultant for many clients. Since I&#x27;m sure you began each audit meeting with the CTO&#x2F;VPE and possibly others like senior devs&#x2F;architects, how much of what you ended up finding in the audits was predictable based on those meetings? (I&#x27;m guessing almost everything).<p>My follow-up question is that once you heard about their snazzy microservices architecture, were you ever surprised by it being a good decision based on the product type and how well it was engineered?

I&#x27;ve been a part of 3 startups, 2 of which failed and are no longer around. What they all could have benefited from was a business audit.

Interesting to see the JWT issue. I have recently found a vulnerability in a publicly traded CRM SaaS that was also about JWT claims validation. It’s also quite amazing that popular Auth SaaS rely so heavily on JWTs with 1 hour expiry times, making it impossible to log users as you can’t invalidate the token for the next hour.

&gt; Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering.<p>My hunch is that this is related to the nature of product-market fit. If a company is very successful, there&#x27;s a decent shot that market demand became overwhelming at some point early on. That demand, in turn, becomes a strong motivator to keep things simple and ship quickly, instead of writing code The Right Way.<p>Facebook using PHP might be one of the best examples of this: if their user base didn&#x27;t explode, maybe they would&#x27;ve taken the time to carefully rewrite their code in Java or Python. But the fact that they would&#x27;ve had time to do that would&#x27;ve made it less likely that they&#x27;d become $500b company today.

Guilty as charged. I bootstrapped a our startup with just myself and 2 junior engineers in the last year during Covid. Junior in the sense that they are young. But actually they outperform many older engineers I&#x27;ve worked with. We are starting to close some pretty big deals and I&#x27;m dreading the moment where I have to turn this into a normal development team. In my experience velocity drops when you do that and you lose a lot of momentum. 3 people can do a lot. 6 people don&#x27;t do that much more. I&#x27;m not so young myself and quite experienced. But I lean heavily on my team for doing the work. I&#x27;m the CTO, the CPO, and I need to worry about team management, sales, and a few other things. So, less than half of my time is spent coding. This it the reality of startups. You have to do all of it.<p>I made some technology choices early on. We use docker but not Kubernetes. There is one server, it&#x27;s a monolith. There is one language, it is Kotlin. And we even use it on our frontend (web only).<p>The latter is not something I would do normally or recommend. But both my junior engineers only knew Kotlin and we just went with it and never actually ended up regretting this. This surprised me and at this point I don&#x27;t feel React&#x2F;typescript have anything that I need or want. We&#x27;re doing a lot of asynchronous stuff, websockets, maps (via libremaps), etc. And it all runs smoothly and responsively. Kotlin is kind of awesome for this actually.<p>Originally our frontend was Android only. We ditched that app in favor of a Kotlin-js based web app that started out as a proof of concept that just more than proved the concept and became the actual thing. At the time we had a demand for IOS and web and no IOS or web developers on the team. Hence Kotlin for the web. When this looked like it was workable we actually lost our Android developer. So the decision to forget about that app was pretty easy. At that point it was half working and full of bugs and technical debt. Fixing that would only half fix our problem because we&#x27;d still need IOS and Web. So we did web first. And we are packaging it up with Cordova for those people that want something from an app store.<p>It&#x27;s a good lesson on prototyping. If it works, do more of it. At the same time, I normally recommend minimizing risk and not building too many things in parallel. Like building 3 apps for 3 platforms instead of just a web app.<p>Our server is Kotlin&#x2F;Spring boot and we use a lot of Elasticsearch because that&#x27;s what I&#x27;ve been using for the last decade. A little bit of Redis and I&#x27;ve so far found no execuse to use a relational database. But I&#x27;d probably end up with mysql if that ever comes up. Done right, Elasticsearch makes for a nice key value store without transactions but with optimistic locking on documents. If I get some time, I add a database for safety at some point. But less moving parts means less headaches. Having just one language means the distinction between backend and frontend is a bit blurry. We have an api client that we use in our spring tests that also compiles to kotlin-js. That library contains a lot of code that we use in our front-end. Model classes, caching layers, functions that call various of our APIs, etc. And it&#x27;s all covered in tests. All the business logic basically. If we ever need to do native apps, we&#x27;ll use that there as well.<p>On the devops front I&#x27;m a combination of very pragmatic but also focused. We use stuff that works that doesn&#x27;t distract us. So, no terraform for a setup we only create once; in 20 minutes. Not worth spending weeks automating but worth documenting. But we do have CI&#x2F;CD via github actions. So we don&#x27;t manually deploy anything. And we have lots of API integration tests. If it builds, it ships. No rollbacks; roll forward only. Keeps things simple.<p>We use Google Cloud and keep our cost low. A couple of VMs, a loadbalancer, a managed redis, and a managed elastic cloud cluster. That&#x27;s it. Nice and simple.

If people would just admit, and adapt to, the fact that the browser won over native, and Oracle won over Sun, we would avoid this situation with armies of Java developers making rube goldberg variations of basic relational, sysadmin, and gui programming tasks. But then again, how would we employ all the people with these extreme productivity multipliers, as long as our politics and economic system is still pretending that we just had the industrial revolution, and need to man the assembly lines, then Parkinson&#x27;s law will apply for tech work just like everything else.

I assume this isn&#x27;t content marketing because PKC doesn&#x27;t seem to exist anymore. But this post made me really want to get Ken to audit our code.<p>Are there any vendors that do similar work that people here recommend?

&gt; <i>the major foot-gun that got a lot of places in trouble was the premature move to microservices</i><p>I sometimes wonder if the move to microservices isn&#x27;t just a weird consequence of Conway&#x27;s law in reverse: make a department of each developer, let them have their thing.<p>(See also this amazing video about Conway&#x27;s law: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=5IUj1EZwpJY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=5IUj1EZwpJY</a> )

At what point should I push for a code audit?<p>I don&#x27;t think any of the codebases I worked on ever had a &quot;real&quot; audit. Best case was reviews pre&#x2F;post acquisitions. An external audit seems like a good thing, but I have no idea how to argue for such a thing.

Point 2. I think this is a common misunderstanding in what engineering is. To come up with good simple solutions to complex problems often takes a lot of experience and domain knowledge.

&gt; the major foot-gun (which I talk about more in a previous post on foot-guns) that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs.<p>Preach! Micro services are a solution to a problem that affects effectively 0 startup sized systems. That problem is scale. Micro services are <i>hard</i>. WAY harder than monoliths. They become necessary only once your physical hardware can no longer keep up in a monolithic fashion and parts of your system need dedicated compute and&#x2F;or storage.<p>And no, they are not automatically necessary once your engineering team reaches N size either. Introducing network boundaries as a way to scale your engineering organization is a bad idea.

This is a great list.<p>One minor criticism on...<p>&gt; <i>Monorepos are easier to audit.</i><p>&gt; <i>Speaking from the perspective of security researcher ergonomics, it was easier to audit a monorepo than a series of services split up into different code bases. There was no need to write wrapper scripts around the various tools we had. It was easier to determine if a given piece of code was used elsewhere. And best of all, there was no need to worry about a common library version being different on another repo.</i><p>This is much more dependent on the auditor&#x27;s personal workflows (as well as the relative hygiene of any team&#x27;s monorepos), rather than being universal. I&#x27;ve found the opposite to be true for e.g. the current orgs that I am auditing: individually split up repos tend to be idiomatically structured, and &quot;just work&quot; as expected more often than monorepos, which more often than not have a lot of custom glue or unusual monorepo-management init scripts.<p>Comments on the other (generally very good) points in the list:<p>&gt; <i>Writing secure software has gotten remarkably easier in the last 10 years. I don’t have statistically sound evidence to back this up</i><p>I suspect compiling such statistical evidence would also be impossible as <i>detection</i> of security issues has also improved, so any data would never be comparable over time.<p>&gt; <i>The counterargument to this is that heavily weighting discoverability perpetuates ”Security by Obscurity,” since it relies so heavily on guessing what an attacker can or should know. But again, personal experience strongly suggests that in practice, discoverability is a great predictor of actual exploitation.</i><p>This is a tough circle to square because security by obscurity <i>works</i>. It&#x27;s probably the best security measure you can have in place. But it&#x27;s bad for two reasons:<p>(1) The process of obscuring often (doesn&#x27;t need to, but very often) obscures auditing, which means you end up relying upon obscurity solely. It&#x27;s not a worthwhile trade-off.<p>(2) In a simplistic marketing world, the idea of obscurity as a standalone measure is so tempting to non-technical decision makers that I believe it requires a bit of innocent dishonesty about it&#x27;s effectiveness to dissuade.<p>&gt; <i>(on auditing dependencies) Node and npm were absolutely terrifying in this regard—the dependency chains were just not auditable.</i><p>I agree with the overarching bullet point this is said within, but I see this point about NPM said a lot, and I&#x27;m not sure how people are going about auditing or how many language ecosystems they&#x27;re looking at. I have found Node&#x2F;NPM to be the best &#x2F; second best popular system for auditing dependency chains. I have significant experience in this area: the relative consistency of package management config across the JS&#x2F;TS ecosystem is enormously helpful for software composition analysis - the only package manager configs I&#x27;ve found that may be slightly better is Composer, but the inconsistent usage of Composer by many PHP devs still makes it a little worse than NPM in practice. PIP &#x2F;PyPi &#x2F; setuptools is an inconsistent moving target of requirements.txt (is it a lockfile?), pipfile.lock, setup.cfg -vs- setup.py, pyproject.toml, and whatever else. Maven is a nightmare of multiple registry endpoints, and issues parsing custom &lt;dependencyManagement&gt; directives, extensions (without even starting on maven wrappers and pom.xml templating strings). Don&#x27;t get me started on Gradle. Go&#x27;s idea of package management is: just pull it from Git; good luck automating it if you&#x27;ve got private repos with any kind of secure ssh auth. I have less personal experience with Rust&#x2F;Cargo.<p>&gt; <i>for some reason, PHP developers love to serialize&#x2F;deserialize objects instead of using JSON</i><p>PHP serialize&#x2F;deserialize predates the existence of the JSON spec., so that might have something to do with it. A lot of PHP code is <i>old</i>.<p>&gt; <i>Almost no one got JWT tokens and webhooks right on the first try.</i><p>Nor the second try...

&gt; All the really bad security vulnerabilities were obvious.<p>Isn&#x27;t that just tautological? They are bad because they are obvious?