Everyone is still terrible at creating software at scale

Sometimes I feel like there are two worlds in IT: the one you encounter on a daily basis, and the one you read about on HN. The only thing consistently &quot;terrible&quot; about both of them is the often expressed lack of humility towards solutions you don&#x27;t &quot;like&quot;, or more often, actually, solutions you don&#x27;t understand.<p>Gottfried Leibniz suffered no small amount of flak for his conclusion that the reason the world looks the way it does is because it&#x27;s already &quot;the best of all possible worlds&quot;: you may think you see an improvement, but you lack divine understanding and therefore you don&#x27;t see how it would actually make things worse.<p>While this is in my opinion a silly idea, I think most people who work in tech could use a bit of humility influenced by this line of thought: if a solution exists today, and it has yet to be replaced, it&#x27;s at least <i>possible</i> that it&#x27;s over all the best solution, and the reason you&#x27;re not seeing that is because of a lack of understanding.<p>Of course this is not always true, but if nothing else it would lead to more interesting discussions than those stemming from someone saying &quot;if you&#x27;re not using Kubernetes you&#x27;re a moron&quot; and another replying &quot;learn how to sysadmin and you&#x27;ll realize you don&#x27;t need Kubernetes in the first place&quot;.

At scale software stops being a technology problem and becomes a people problem. And groups of humans don&#x27;t scale.<p>We naturally organize into hierarchies of small subgroups with limited interaction boundaries. Each subgroup will adopt slightly different methodologies, have different tooling preferences, will have integration and communication overhead with others, and so on.<p>This can not be preventend, only mitigated. Which is perfectly fine. The important step is recognizing that software development is really not special [1], and that the limiting factor is the human element. Then you can build your organizational structures around these restrictions.<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26713139" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26713139</a>

The tooling is simply not there, so every software project keeps pushing the boundary of what is possible in its own unique fragile way.<p>People don&#x27;t want solutions to yesterday&#x27;s problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.<p>By tooling I mean programming languages, frameworks, libraries and operating systems. All of whuch have been designed for a single machine operation with random access memory model.<p>This no longer holds true. In order to scale software today you need multi machine operations and no such OS exists for it yet. Only immature attempts such as Kubernetes, whose ergonomics are far from the simolicity that you&#x27;d expect from a unix like system.<p>And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.<p>I don&#x27;t think we&#x27;ll push the boundary further if we keep piling crap on top of existing crap. The foundations have to change and for that we&#x27;d better go back to the 70s when these ideas were being explored.

Software is indeed a lot like cities.<p>But cities ain&#x27;t pretty. Dig the ground and nothing looks like what&#x27;s on the map. To create a new building, you need to cut a lot of plumbing and re-route it somehow.<p>Stuff gets old and breaks all the time, and maintenance teams are on the ground 24&#x2F;7 to fix it. NYT subway is the mainframe of the city. Look at the central steam heating that&#x27;s still pumping water into high rises.<p>Sure, you can sell a shop, and people will be able to repurpose that space to suit their needs very efficiently. But isn&#x27;t that what Docker does? Provide a cookie cutter space to run an app?<p>But in cities, there are places where trucks can&#x27;t go, where you can&#x27;t park, where unplanned work happens, where the sewer floods. That&#x27;s when the city &#x27;bugs&#x27;, and people need to manually intervene to fix it...<p>Trying to find an ideal way of running software at scale is just as utopian as building the perfect city using &#x27;super structures&#x27;. It&#x27;s a dream of the 60s that&#x27;s never going to happen.

My own highly simplified opinion why software development &quot;doesn&#x27;t scale&quot;: because software development is all research and zero production - with the notable exception of asset production in game development, which actually is quite easily scalable once you&#x27;ve figured out a workflow that works for a specific (type of) game. If you think about how to scale software development, don&#x27;t think about an assembly line, but about a research institute.

Very often, even tough I only ever write very short programs (100 to 1000 lines) in Python, C++ (and learning Rust), I look at my finished program and feel like there is a million things to improve and a billion other ways to do it. Each variable could have a better name, each abstraction could be abstracted more, the whole thing could be rewritten in a different style or language.<p>Whereas if I design something in CAD, I send it to the printer, try it, improve it once, print again and never ever again think of it as long as it does it&#x27;s job.

&gt;How is it that we’ve found ways to organize the work around so many other creative disciplines but writing software is still hard?<p>I think OP heavily overestimates the organisational praxis in other disciplines. Nearly every creative discipline I was ever in was largely purely ad-hoc with very little explicitly stated organisational approaches to the craft. Academia, movie and music production, and creative writing for instance have much less readily-available principles than software.<p>Software is probably the most thought about creative industry I can think off in modern history.

Alan Kay said at one point:<p>&gt; I have always done my work in the context of a group that is set up to maximize a wide spread of talents and abilities. Just as “science” is “a better scientist than a scientist”, such a group is “a better programmer and systems designer than any individual”. Really learning to program — etc. — is really learning about how to be part of a scientific&#x2F;engineering&#x2F;design team. As with a sport like basketball, there are lots of fundamentals that all need to do extremely well, and there are also “special abilities” that not every member will have, but which the whole team will have when it is functioning well as a team.<p>I fear that very few people really spend the time needed to really learn the fundamentals and very few people learn a scientific mindset that would allow them to cooperate effectively with others without getting their ego in the way.

I thought somebody might mention it, but since nobody did: a best boy grip is the chief assistant to the person in charge of building rigs and frameworks for things like cameras and lights.<p>Best boy basically means &#x27;chief assistant&#x27;, second in command kind of thing.<p>Grips build the structures which cameras and lights are hung on. They don&#x27;t touch the electrics, gaffers and electricians do that: grips make the towers, stands, tracks: the physical stuff on which you put things.

There is no &quot;scale&quot; so to say. To create MS Windows which has lets say 50 mloc, versus MS Calc which has maybe 1&#x2F;1000 lines of code compared to Windows you don&#x27;t &quot;just&quot; write x1000 times more code and done. Writing one Windows is many orders of magnitude harder than writing a thousand Calc sized apps. That&#x27;s why &quot;scale&quot; don&#x27;t work, it&#x27;s not scale, it&#x27;s difficulty spike. If you&#x27;ll hire large enough corp, with several hundred expert programmers (no trainees , no juniors), and do what they think they want - don&#x27;t interrupt them from flow, schedule almost no meetings, communicate strictly async via email, allow then self organize and self manage - I think that such a project will be an epic sized fail.

Yesterday I watched a video about how Netflix scaled its API with GraphQL Federation.[0] The video contains some neat visualizations that helps you see how complex data accees problems at scale can get. And this is just the services level they talk about.<p>No mentioning of the underlying infra with all its complexities needed to acheive the goals of flexibility, reliability, speed and cost cutting.<p>You don&#x27;t have to be of Netflix size - when you start getting tens of thousands of users, complexity hits you real fast.<p>[0] <a href="https:&#x2F;&#x2F;youtu.be&#x2F;QrEOvHdH2Cg" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;QrEOvHdH2Cg</a>

Torvalds once said [1] that the only scalable software development methodology is open source, and I tend to agree for two reasons:<p>1. The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. In Coasian terms, the transaction costs around development are minimized.<p>2. It enforces hard interfaces and it clearly separates development from operations. Lack of discipline around these issues is a source of much accidental complexity.<p>[1] I can’t seem to find the quote, I read it in an interview a few years ago and stuck with me since.

This post seems like the author is early in their career and is realizing everything in the world is half broken and not perfect. What stood out to me is when he said “you could build what takes weeks in an enterprise in a weekend.” I don’t know about everyone else but I’ve become a much much better engineer since college and I still have to carefully handle every edge case for production and write a lot of tests before shipping something and I really don’t think code I write in a weekend will be production ready at a large company. Truth is by any metric, the massive systems in place by Amazon are pretty amazing examples of scalable software. At my day job at least managers have seemed to figure out that they set the high level goals&#x2F;milestones and then let the engineers figure out how we want to get to those goals. This system has worked pretty well (of course the deadlines can get tight and we accumulate technical debt) but once we hit a milestone we deliberately decide to take some small amount of time to fix technical debt, as long as we kept track of it as we scrambled which also requires some discipline. The way things distill up to management is for my VP he doesn’t know what details you did on x day, he wants to know did you help implement y one of our org’s high-level goals. This system works to some degree and I haven’t thought of a better one yet.

The city metaphor is evocative and new to me. I like it.<p>&gt; Conversely, if we built our cities the way we build our software, you would need to enter the shop through the special garage, and exit through the roof to walk a wire to get to another custom made building from scrapped containers to do the checkout. And some of the windows are just painted on because they’re an MVP.

Software crafted carefully for the beauty and structure is one thing. Another is software for customers where delays of shipping the functionality has financial impact. Will you perform a change with cost of 1 day or do it right and spend 5 days? What will you be rewarded for? Short-term thinking is right until it&#x27;s wrong because it stalls the whole company&#x2F;system.<p>Another challenge in software comes from the added cost due to decisions that are tricky to reverse. Worst thing is that they only show at certain scale, meaning that in many cases they were the right decisions in the past (business context).<p>If you only operate in a single market, you can process prices with a single currency, so you get away with bad modelling - without the currency. To scale to another market, you can just deploy your application twice. If the company succeeds in new markets, you will replicate the solution. Fast forward a few years and you have a migration project to add currency to your software and data to optimize deployment.<p>Organizational complexity is another domain. Understanding what places to adjust is key. For example: how complex is it to change the tax rate in your software? How do you find out which applications to change? How do you know who needs to perform the change? Do you have to broadcast a question to the whole company and ask them for performing the change until X? How do you know you adjusted all places? Do you have to deploy the change manually or will the change be automatic based on time?

This is one of the reasons I’m in the erlang derived system of languages. You never have to use the distributed tools if you don’t want to but they’re really powerful and really elegant when you start to need them.<p>This is only one part of the problem though because as someone stated below even though server systems have networking stacks they are not entirely designed around the concept of seamless computation over multiple nodes so you wind up with abstractions that solve the problem but do so fighting against some occasionally nasty gotchas that make it a lot harder than it needs to be.<p>This is all to say I’m greatful for those who maintain things like erlang, elixir, kube, go etc. They’re somewhat fighting against the stream so we have an easier time when we do need to scale.

That&#x27;s a great essay! I think he describes the issues quite well, and I enjoyed his metaphors (I like the city metaphor).<p>I also -like him- have no answers for &quot;the big picture.&quot; I ran a small cog on a big wheel for a long time. I feel the basic <i>quality</i> (as in &quot;few bugs&quot;) of our work was good, but I also feel that it took too long, and failed to flex, so it often fell flat with our customers. Our software was developed, using a hardware process, and there are vast differences between the two disciplines.<p>I am fortunate, in that I don&#x27;t need to play in anyone else&#x27;s sandbox. It means that my scope&#x2F;scale is quite limited, but I am pretty happy with what I am able to do. It&#x27;s actually fairly ambitious for a one-man &quot;team.&quot;<p>Since striking out on my own, I have experimented with what I term &quot;ultra-agile&quot; techniques; with some success. Hard to codify, though, as they depend on a high degree of <i>experience</i>, as well as the fact that I&#x27;m a bit &quot;spectrumish.&quot;<p>The closest to success that I have had, is to design <i>all</i> of my software as an extensible infrastructure. I don&#x27;t know, exactly, how it will be extended, but I write &quot;hooks&quot; into it. I often end up using these &quot;hooks,&quot; myself, in future work.

&gt; At least, building a new project within a company should be easier than starting from scratch, but my hunch is that many companies fail that test.<p>This observation seems both true and important.

&quot;Ironically, in my experience, just bouncing off ideas of one another is not the way this works well. What you can end up with is people throwing their ideas into the ring just for others to find faults in it. In the worst case, this can turn into a competition who knows most and who is smartest.&quot;<p>This right here is exactly why I learned to say nothing in brainstorming sessions and try to dodge then outright if I can.<p>My experience is these are typically for someone higher up in the pecking order needing their own ideas validated.<p>My usual work around is if something needs to be done, quietly work on a POC on your own and solicit feedback or even bring in stakeholders.

I see another person using Fred Brooks &quot;accidental complexity&quot; in a wrong way.<p>Software development is hard because &quot;essential complexity&quot; is hard. Whatever people want to do is hard. You can make it easier to write software as much as you want, but you are not making &quot;world&quot; easier. That is main point of Fred Brooks essay.<p>I see a lot of devs are making statements that now development for them is essential complexity. While no, software and code never will be essential complexity, software only helps us to solve essential things faster.

IMO, technical leadership isn’t allowed to build at scale or the leadership doesn’t exist.<p>The race to the bottom incentivises profit over principle. Boards don’t care if something is built to scale if the competitor gets to market with a cheaper solution. Scaling is a day two problem.<p>I’m bearish on my future employment as someone who questions the motivations for profit driven development.<p>I naively thought the market would reward the best product, instead I see the cheapest product being rewarded. What happened to innovation and doing what’s best for society?

&gt; The Team Topologies book suggests to favor teams that are end-to-end, that fully own a problem to be solved, supported by platform teams and teams that manage a very complex piece of technology.<p>Does anyone have first hand experience with this working really well? It seems sensible but in my experience it does not work. The platform teams, I think, should be collaborations between a small number of members from each end-to-end team. The platform would be allowed to grow only as needed by the use cases. As soon as you put a platform team in charge with a mandate, what they build and what is needed starts diverging in the name of &quot;Long term planning&quot;. Instead of fixing the struggles of today, they think they have the capacity and formula for fixing the problems of a year from now. In my experience, they do not. Would love to hear from others

Computing <i>is</i> a lot like American cities, in the that the infrastructure (yak shaves) that would make everything easier are almost invariably under-invested.<p>One of my favorite examples would <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Capsicum_(Unix)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Capsicum_(Unix)</a>. It&#x27;s incremental which is good. But people wrote the kernel part and declared mission accomplished like Bush in 2003, ludicrous! One has to do that and <i>then</i> overhaul the userland <i>and</i> either get those patches upstreamed. And these days &quot;userland&quot; isn&#x27;t a collection of init, shitty scripts and Gtk&#x2F;Qt, but a bunch of libraries, especially programming language standard libraries.<p>This would dramatically change the security <i>and</i> ergonomics landscape, because so much global state programmers<p>&quot;Containers&quot; was always the wrong metaphor because software is about <i>composition</i>, but containers are inert and only their mass and volume composes (very crude). Better to think about plumbing or rail.<p>Another perhaps more opinionated example is getting everyone to use Nix (or fine, something like it.) Whether with container style virtual global state, or nicer capsicum, we need to make it trivial to install <i>and develop</i> the entire commons. All &quot;builds on my machine but you can use it&quot; just leads to a lack of integration so no one can help the original author smooth over the gaps. It will also allow machines to be demystified, allowing people to toy with all the software on the system, which will help reduce the programmer alienation which allows so much accidental complexity to occur in the first place.<p>But yeah, almost nobody is doing these things at the scale they deserve, and even the megacorps drown in their own technical debt like sunbelt cities that are just metastasized suburbs. Eveyrone is in the &quot;well I&#x27;ll eat shit and shut up as long as my competitor is too&quot; mind site. It&#x27;s disgusting.

Scale up vs. Scale out.<p>Those require different architectural considerations.<p>E.g. with scale out your chache has to work suddenly in a distributed way.<p>And as we all know scale-up is expensive, can quickly become very expensive, and more importantly it has upper limits. In most cases scale-out is actually the only option, even if scale-up would work for some while. Many times I have seen architects go the easy route and scale up. They either get promoted or switch companies until their solution either gets too expensive or can&#x27;t scale any more.<p>In the meanwhile other systems are interacting with this system and would need to make changes to adjust to a scale-out system.<p>E.g.: A distributed cache can result in lower write performance, or someone else could have overwritten a certain entry because locking failed (e.g. SELECT FOR UPDATE).<p>In many cases those systems join the list of legacy systems.

&gt;<i>Everyone is still terrible at creating software at scale</i><p>Terrible is a relative term. Terrible compared to what?<p>Who said&#x2F;showed (much less proved) that there&#x27;s a better way and we just don&#x27;t follow it to achieve it some optimum?<p>There&#x27;s also a semantic confusion here. Compared to e.g. the car industry we&#x27;re infinitely better at &quot;creating software at scale&quot;. We can create a billion copies of a software and distribute it our the world, with marginal cost close to zero.<p>But the author doesn&#x27;t mean &quot;creating software at scale&quot; like when they say &quot;car production at scale&quot;. They mean production of &quot;large software&quot;.<p>Well, let&#x27;s see the car industry make a car with the scope, flux requirements, shifting environments, etc that large scale software has...

Rather meandering and over-verbose, this post never really seems to reach its point.

Software is predominantly a fashion business, not a technology business.<p>Just allowing the market winners to drive that fashion means we aren’t able to progress the state of the organisational art scientifically.

The problem is we stop pursuing answers on this topic, thus stop making progress. It&#x27;s basically like what Bret Victor has described in &#x27;The Future of Programming&#x27;. [0]<p>There were a lot of language zealots at the end of the last century, especially on evangelizing Object-Oriented Programming. Nowadays everybody can easily counter those arguments with &#x27;No Silver Bullet&#x27; without further thinking, it&#x27;s arrogance in disguise of humility. There are still a huge amount of accidental complexities to deal with in most tech stacks. Most businesses would die fast and leave nothing behind anyway, while the progression of the industry would accumulate and benefit the whole industry itself.<p>Java looks slightly better for creating software at scale than C. C looks slightly better than FORTRAN. FORTRAN looks slightly better than machine code. Say there&#x27;s a language that looks like Haskell but has tooling and ecosystems as good as Java, I believe it would also slightly better than Java.<p>[0]: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;8pTEmbeENF4" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;8pTEmbeENF4</a>

Two quotes stood out to me:<p>&gt; What if software were built in the same way? What if the core parts of our business would be like streets, and all that newfangled stuff is something we could build on top, experiment, tear it down if it does not work?<p>To me, it kind of already is. The foundation on what we build most software on today; TCP&#x2F;IP, HTTP, Win32, POSIX APIs, the C-runtime library, threads &amp; processes, etc. Much of that is about 50 years old now.<p>&gt; Somehow, the code in front of you is just the tip of the iceberg of a lot of mental representation of what is happening<p>I think this is a key point. When we look at code, the comfort with it is relative to how well we understand that abstraction, and have a good model for what it is doing &quot;under-the-hood&quot;. I think a lot of the burn-out and &quot;fatigue&quot; some people feel is where these abstractions are regularly reinvented (I&#x27;m looking at you, JavaScript frameworks), and thus you have to spend time learning the details of a new abstraction before you can just read&#x2F;write a piece of code comfortably.

The problem is that there are perverse incentives for people to enrich themselves as quickly as possible and make problems like performance and scalability a liability for someone else.<p>I can create a startup, get a massive amount of funding, sell the company and make millions of dollars... all without ever seriously caring about performance and scalability.

I prefer single-duty one liner pieces of code&#x2F;commands over large and complex programs that tie all these code snippets into one large abstraction.<p>I would not be hired because I only enjoy executing &#x27;one-thing-well&#x27; snippets of code&#x2F;commands. My minimalism would be a turn-off for most software companies or code shops.

Bad software (therefore all software) is like the recurrent laryngeal nerve in giraffes [0] (a freak of evolution). It is unlikely to be the result of a singular minds &quot;design&quot;. It is more of a result of evolution through countless independent decisions, each of which are individually an improvement, but put together results in weird inefficiencies that are impossible to fix without starting from scratch. And when was the last time you were allowed to &quot;redesign&quot; something from scratch at work?<p>[0] <a href="https:&#x2F;&#x2F;bioone.org&#x2F;journals&#x2F;acta-palaeontologica-polonica&#x2F;volume-57&#x2F;issue-2&#x2F;app.2011.0019&#x2F;A-Monument-of-Inefficiency--The-Presumed-Course-of-the&#x2F;10.4202&#x2F;app.2011.0019.full" rel="nofollow">https:&#x2F;&#x2F;bioone.org&#x2F;journals&#x2F;acta-palaeontologica-polonica&#x2F;vo...</a>

If you think of all of the decisions that are made between having no software&#x2F;users&#x2F;&quot;scale&quot;, to being something like Netflix, you can imagine how many opportunities there are for things to become &quot;terrible&quot; (or really, for a system to be made a small percentage worse, which obviously accumulates over time). I would challenge the assertion that most sufficiently scaled systems are &quot;terrible&quot;, more that a person did not like them for one reason or another. If a system accommodates the demands of its users and maintainers, I don&#x27;t really think you can label it terrible. Scaled software isn&#x27;t art, it is either good enough or it isn&#x27;t.<p>I would argue that it is becoming easier for &quot;terrible&quot; software to scale well. I would say that&#x27;s a much bigger win.

Some reasons that I don&#x27;t think have been covered yet:<p>- experienced programmers are promoted out of the tech track - it causes all sort of problems, people keep reinventing the wheel, they are forced to relearn through their own mistakes, transfer of knowledge and skills is hampered<p>- (probably as a consequence of the first point) people with 3 years of experience are believed to be senior programmers<p>- worse even - experienced programmers are promoted to non-coding roles aka architects. Over time they increasingly disconnect from the tangible artifact (code) while still hanging onto a false belief that they can function just fine by embracing it through a metaphor (diagrams, etc.) (&quot;Simulacra and simulation&quot;?.. but possibly I&#x27;m digressing)

Cities are a good metaphor. Now consider we want to move some buildings around. Teardown some buildings. Create some new ones. Move a bunch of roads around. We need to think about water supply. Electricity. Where to put the people. Which roads can be closed. How traffic should be diverted.<p>Luckily with software this can be done more easily. We only need to design the new architecture. Design how everything should be executed. Finally we hand over the blueprints and let the computers do all the work.<p>But it&#x27;s still going to take some time. A process which is largely invisible. The construction yard is there. But everything is code. You can&#x27;t really take your stakeholders for a tour around all the work that&#x27;s happening.

&gt; I find myself coming back to the film business,<p>The difference I think is that film-industry has separated the creative parts from the production parts and the creative part is not really complicated at all. Write the screenplay. That can be innovative but it is a single author typically who will do it. No complications because everybody&#x27;s screenplay has to work together. Because there is only one.<p>In other words making movies is more like making buildings than making software. Buildings can be huge and very expensive but their design is still quite simple and based on designs the architects have created in the past.

TFA mentions “Notes on the Synthesis of Form” by Christopher Alexander but doesn&#x27;t really do the book justice: <i>&quot;things get exponentially easier if you take problems one piece at a time&quot;</i>.<p>That&#x27;s <i>sort of</i> what the book says but it&#x27;s missing the critical part. The whole book is about <i>how</i> to split the solution space into pieces that can be taken &quot;one piece at a time&quot;, as it&#x27;s obviously not the case that just any split will do.<p>It goes through some real world examples of design and goes on to build a more general semi-formal theory of design, components and independence.

One thing that might make software fundamentally different from other stuff, is that before being able to make a meaningful contribution a person needs to invest X hours just to understand.<p>In most other creative fields, this can be hand-waved away. No other engineering field has anywhere near the complexity.<p>Therefore, all important people on a project are stakeholders of some sort, and anyone else is not able to make real improvements.<p>The project will move in whatever direction satisfies the incentives of the important stakeholders.

Maybe it&#x27;s time to finally differentiate the tools for software as in functions, algorithms, classes, libraries, etc. and software as in e-shops, databases, sso services, streaming services, etc.<p>We have good science and steady (albeit slow) progress in the former while it feels like the latter is more or less subject to stagnation. For instance, when I need to setup a DB cluster, why is there no tool that takes my requirements and generates deployment scripts, monitoring, migration tools, etc.?

I’ve come to think about this more and more in terms of politics. The core problem is that not all people come to work with the intent to build high quality software in a rational and efficient manner. Also, what do you think film production would look like if you just hired a lot of “senior film developers” and let them hash out who should be the director and who should hold the microphone bar?

I agree with the post that software at scale is a big open question.<p>What gives me hope is that there were successful mega projects though not about software. Für example, I read about the Manhattan Project over Project Atlas to the Apollo Project. Those were all mega projects with high innovation and uncertainty like software development. I have no good theory what makes it successful though.

&gt; There is something peculiar about software that makes it different from other crafts.<p>What distinguishes software from other <i>techne</i> is a lack of <i>physics</i>. There is no &#x27;solid ground of reality&#x27;. All other forms of making involve <i>discovering</i> and then <i>applying</i> the governing laws.<p>Hardware architectures, operating system, and programming languages, to an extent, do furnish a <i>phenomenal context</i> and where these characteristics are stable and well established (e.g. Memory Models), <i>science</i> appears [1].<p>But clearly, this ground of reality is indeed &#x27;soft&#x27; and the practitioners usually re-invent not just the wheel, but the ground that it rolls on as well.<p>The dilemma -- which nearly means facing a choice of a multiplicity of lemmas -- that confronts the designers of hardware, OS, and languages is precisely that tension between generality (and corresponding weak &quot;world&quot; semantics) and specificity (with robust but constrained semantics).<p>[1]: <a href="https:&#x2F;&#x2F;ocw.mit.edu&#x2F;courses&#x2F;electrical-engineering-and-computer-science&#x2F;6-851-advanced-data-structures-spring-2012&#x2F;lecture-videos&#x2F;session-22-history-of-memory-models&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ocw.mit.edu&#x2F;courses&#x2F;electrical-engineering-and-compu...</a>

I know this comes with experience but imagining how to scale or how to actually implement it is daunting. Obviously not everyone is going to have to need, but there&#x27;s just so many tools and so many things to keep in mind to scale. What would you reckon is the best way to learn how to scale? Possibly one step at a time.

This is a competitive advantage for companies who can do this better than others. Different technology stacks and different development mentalities make this easier or harder. If it were easier, than smaller &quot;startups&quot; wouldn&#x27;t beat companies like FANG.

There is at least one massively scaled system that relies heavily on software, that doesn&#x27;t often get a mention because it just works. When was the last time you picked up a land-line phone and it didn&#x27;t work because of a fault on the system?

Reading about Neuralink yesterday makes me wonder if in 50 years it will be possible to connect a bunch of minds together to achieve a task none could do on their own. Could be programming or some other large scale task with a lot of interdependence.

This isn&#x27;t a software specific problem. 90% of work in every business is inefficient.

Every system requires sooner or later another system on top. Entropy is increasing. <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Chaos_theory" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Chaos_theory</a>

I have often thought about the comparison to film, and film has been around for over 100 years. So in like 2080 software will be like film. In the year 4000 software will be like houses. In the year 10,000 software will be like nature.

I mean, I think amazon does a good job. Google too, Microsoft gets some points too.

We have actually degenerated back to CLI. &quot;You don&#x27;t need UI&quot; lol.

I&#x27;m surprised this article didn&#x27;t mention test driven development for building large pieces of software!<p>I thought the test-driven development idea was impractical and pedantic when I first heard about it. I gave it a shot, though, and the thing it allowed me to do was to prevent massive systems from becoming impossible to modify. When a system gets so big that it is impossible to fit it in one&#x27;s mind, most developers get scared to do new releases because there&#x27;s probably some critical part of the system they forgot. They forgot all the requirements and QA test cases for at least one obscure part of the system. Even if all the QA test cases are documented, the process to release becomes increasingly difficult and time consuming.<p>With test-driven development, you can just run all the tests, and if they pass, say ship it. The key is not to get too many tests, especially integration tests which take a long time to run and like to break when designers make UI changes. Usually, I start with the happy path integration test and then write tests as I&#x27;m developing for things that don&#x27;t work right. About 40% of the stuff works the first time, and I never have to write a test for it. 45%, I write the test and fix it, and it passes, and I can forget about it. The other 15% is usually some tricky algorithm with many corner cases, and that&#x27;s where most of the testing goes. I typically write one happy path integration test and then fill in the lower level tests as needed. When the happy path works, most of the system is working. That said, spending time on integration tests is usually a massive waste of time. It either works the first time or something lower down broke, and one should write a test for the lower level part that runs in a shorter amount of time than the integration test.<p>I was able to port a large enterprise app from an Oracle backend to Postgres because there were tests for everything. The port just amounted to getting all the tests to run on the new database and the necessary data migration. This migration was by no means a trivial feat, but it was at least possible, and it ended up working and saving the company millions in license fees.<p>The point being. A system with millions of lines of code is approachable if it has good tests. A new developer can work on it, and if all the tests pass, that developer didn&#x27;t break anything. I can go back to the code I wrote years ago and still use it if I run the tests, and it works. I can then see how it&#x27;s supposed to work, add features, and so forth. Without tests, this is very difficult because as systems become bigger, they break more and more if they don&#x27;t have tests.

A lot of guessing and hunching and try to find one answer to a couple of problems where the question isn’t clear. So I don‘t know what point this article wants to make.

Infrastructure as code services is actually matching the City metaphor quite well. With the huge difference you can rescale and reshape your &quot;city&quot; at will

Software development is research work.<p>If you take processes from people who build machines and apply them to software, it will fail. Software means <i>inventing</i> machines.

We are now seeing the limits of everything, USB2&#x2F;3, Gb network (pushing beyond increases complexity to a ridicoulus amount), Gflops&#x2F;W (2 is CPU peak for electronic transistors) and persistence (the most durable drives where made 10 years ago on 50nm).<p>So now we know that one man can build, host and improve&#x2F;maintain a product himself with the entire earths population as customers.<p>This is a short window, before thermodynamics takes charge and ruin the prospects, enjoy it while it lasts!<p>The first one to do Minecraft without hiring anyone or selling it wins. :P

Software is more art than science. Show me the F=MA of software. What is the industry accepted fizz buzz?

The world is terrible at creating software because the developers are the last to be paid.

I sometimes suspect that this is a consequence of the anti-engineering mindset that a lot of the software industry embraces. Things like YAGNI, programmers don&#x27;t need to study CS, not hiring testers, etc.<p>Maybe if software &quot;engineers&quot; were treated more like actual engineers then we would start seeing better results.

Does the old adage &quot;80% of all software projects fail&quot; still hold?

Here&#x27;s why:<p>First, consider a car. Imagine if somebody told you that a random company would be changing the software that runs your car, every day, as you drive it. Freaked out? You should be. That&#x27;s what a modern tech company is. They just don&#x27;t have the same risks, so they don&#x27;t care so much if they fuck up, so the work isn&#x27;t organized very well.<p>Next consider cities. Do you want to build something? Great. I&#x27;ll ignore all the costs (there are many).<p>First you&#x27;re gonna need a design by an accredited expert. That design needs a dozen permits approved by the government before you can even touch a trowel or hammer. Then a team of experts guess to town, changing grade, pouring foundation, waiting to cure, installing basic plumbing, and then getting an initial inspection. Then comes framing, inspection, plumbing, inspection, electrical, inspection, HVAC, insulation, drywall, fixtures, trim, walkways, driveways, flooring, landscaping, grading, and inspection.<p>So far we have used specialist teams to build sections of the building and only continue when a strict inspection by authorities says we can continue. No customer has used the building yet. Also consider that this building will not scale. If you reach capacity, go make another building.<p>Building software at scale is a combination of monkeying around with a car&#x27;s internals while someone is driving it, and building a building. Only the risks to human life are lower, and we do not have teams of very specific contractors strictly following government approved codes and zoning laws to build one thing that meets one set of criteria.<p>The people doing the job(s) in tech aren&#x27;t good enough at it to do the things we&#x27;re asking of them. And they aren&#x27;t just building one piece and walking away, they are constantly fiddling. And very often, they have no master plan approved by inspectors according to well known and inviolable codes.<p>On top of that, they have never sat down and figured out how to do all of this really efficiently amd reliably. Scrum&#x2F;Kannan are general processes; they do not tell you specifically how to build a website in an extremely efficient and error-free way. But we&#x27;ve done that a million times by now, poorly. It&#x27;s because we haven&#x27;t yet codified it as an engineering discipline and stripped out the fat. And we haven&#x27;t done that because there&#x27;s no requirement to, because nobody&#x27;s life is on the line, the way it is with cars and buildings.<p>And it&#x27;s hard. It&#x27;s hard because we still hand-make components rather than buy them off the shelf. Every company I&#x27;ve worked at has re-created the same god damn thing 1000 times, by hand, because for some bizarre reason they thought it was a better idea to forge steel than to connect plumbing. Imagine if your plumber hand-crafted her own custom fittings for each job.<p>Really, we do a <i>marvelous</i> job today considering how completely undisciplined, unregulated, haphazard, and dangerous what we do is. There are many approaches we could take to simplify the actual process of it all, and make it efficient. But the complexity and difficulty will be there for a very long time. We can also move to pre-fab, but you&#x27;d have to convince people that writing new software just to build a tech product is a bad idea. Good luck with that.

Conversely, writing software for a <i>small</i> scale is absurdly inefficient.<p>I consult for government departments that often have legally mandated requirements for making something available online. That could be a form submission process, some geospatial information presented on a map, or whatever.<p>The problem is that when you might have 500-10K user transactions <i>annually</i>, it becomes crazy expensive to write bespoke software, even with the most agile process and the lowest overhead tooling.<p>Take cloud deployments, for instance. Sure, you can just &quot;click through some wizard&quot; pressing next-next-next-finish and be up and running, but the security team won&#x27;t allow your infrequently-maintained web server on the Internet without a web application firewall. Setting one of those up is days of tweaking to avoid false positives.<p>Need to send mail? Azure bans outbound port 25 connections, you have to use Sendgrid, or something like it. Time to read up on yet another unique and special API!<p>Collecting fees and penalties or making payments to citizens? Woah there... there&#x27;s a massive API surface you have to learn. Security on top of security is needed. Whitelisted IP addresses. Client certificates. That have to be rotated, <i>manually!</i><p>You&#x27;ll forget some essential maintenance, of course, and then you&#x27;ll have to set up triggers and alarms so you don&#x27;t get burned the second time. Which entails mailing lists that <i>change dynamically</i> because the team of contractors has a turnover rate faster than the typical certificate expiry time. Send too many alarms and all recipients will configure an Outlook rule to ignore them. Not enough alarms and you&#x27;ll miss issues. Just setting this up semi-reliably is an exercise in itself.<p>Really basic stuff becomes difficult, when you realise that 99% of the alerting and monitoring features in Azure and AWS are designed for systems at <i>their scale</i>. It&#x27;s all about analysing beautifully smooth curves of graphs aggregating millions of points of data, where deviations are glaringly obvious. These tools are utterly useless when you get one real transaction per day, swamped by a thousand bots. The <i>load balancer health check</i> is 99.99% of the traffic for some of these sites!<p>Then there&#x27;s the human element:<p>Have you tried justifying the time to set alarms for a system where a week-long outage might only affect a dozen customers?<p>How about the budget to upgrade to a newer operating system for something that is not <i>technically</i> broken -- merely hard to support now?<p>Or have you tried doing any sort of maintenance on a system that was built by contractors hired for a fixed term, all of whom are now gone?<p>Meanwhile departments are renamed every couple of years to suit the whims of the latest batch of politicians, so <i>everything</i> has to rebranded. Even tiny little sites use by practically nobody.<p>I&#x27;ve seen sites up for 10 years, where I estimated that they cost $2,000 per citizen that actually used the site! Madness.

Every system requires sooner or later another system on top. <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Chaos_theory" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Chaos_theory</a>

TLDR: Complex systems are complicated to build

This the template for a clickbait headline on HN.

I enjoyed this.. fun sort of intentional-meandering style.<p>I propose that scaling (as defined here) is a general problem, not just a software problem. I realize this is a hard case to make. Technical debt, refactoring hells and such don&#x27;t plague film sets or factories like they do code shops. But.... Consider this:<p>- Consider factories. A factory is just a huge pile of physical capital organised to efficiently do a specific, defined task. Everything is optimised to reduce marginal costs. Capital efficiency is strictly enforced by capital money markets. Marginal cost efficiency is strictly enforced by real markets.<p>... Either of these two constraints (marginal costs too high or not enough capital) usually come into play before engineers have had a chance to redesign, refactor, repurposed and abstract a factory to the point where these are a problem.<p>- Everything that is in a movie is in a shot. Movies have a finite number of shots and a director can direct each one. A film may have a whole team replacing Toby Philpott with Jabba the Hutt, an army of costume designers, set designers, actor psychologists, etc. But, everything that gets into the movie gets there in a shot, and the director can work shot-by-shot and wield god-like control of lots of labour that way.<p>Software is unique. Software is free from most physical limitations. There are no material costs. There are no capital costs. The only economic resource that a software enterprise wields are the engineers&#x2F;engineering itself.<p>When a Ford Company scales, it raise money. It uses that money to to build factories, buy materials, etc. These are all scarce&#x2F;finite resources needed to start&#x2F;continue making additional cars.<p>Google, MSFT, FB, post-AWS amazon... When they scale up, they just scale up. They hire more engineers. They produce more software. The only &quot;resource&quot; being scaled up is the people making the software.... <i>Something</i> has to be the limiting factor.<p>In any case, the &quot;scaling creative work&quot; problem <i>does</i> exist in factories too. The difficulty 50s era auto manufacturers faced competing with Toyota are sort of evidence. They struggled with &quot;technical debt&quot; in the form of car models, factory design, company culture and such that couldn&#x27;t adapt flexibly.<p>A lot of tesla&#x27;s wins (besides marketing, fundraising and software) have come from their recent blank state start. Auto manufacturing today is highly caught up in the &quot;Toyota Way&quot; of doing things. It&#x27;s been that way for decades and parts of it are explicit assumptions of international trade deals. It&#x27;s very compartmentalized. Flexible within compartments. Rigid without compartments, whether they are outside the department or outside the company. It&#x27;s a lot like the city metaphor this article mentions.<p>The problem is... sometimes you need to design the factory and the car simultaneously. At times that has screwed Tesla. The quality control benefits of the Toyota Way are not to be taken lightly. At other times though, it works. The factory designer has been banned from designing the car and the inverse for 40 years. I think their version of technical debt is the reason.

I really hate titles like this.

I&#x27;m not.

+1 for Design Thinking