Fragile narrow laggy asynchronous mismatched pipes kill productivity

So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools&#x2F;solutions exist to &quot;build houses&quot; when most of us are just throwing together lean-to sheds and dog houses.<p>Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago.<p>Then there are some folks&#x2F;organizations building cathedrals, and the associated tooling (react, angular, maven, etc) and all the rest of us look up in awe and thing &quot;well I guess if I want to be that good I need to use those tools on this dog house.&quot;<p>But your dog house doesn&#x27;t have the need to host parties, provide security, or even real weather protection other than a roof to keep the rain and sun out. Yet we all try to build our dog houses in ways that might be better if they are one day converted to a proper living quarters but likely will never have a need for running Water or windows.

Failure is such a fun thing to think about, and it gets handwaved away so often. So many devs, architects, product owners, etc, just focus on happy path, and leave failure unspecced, unhandled, and just hope it never happens. And then boast about 99% uptime, but once you start questioning them you find out they get weekly pages they have to go investigate (and really the system is behaving weirdly a solid 10% of the time, but they don&#x27;t know what to do about it and it eventually resolves itself, and they don&#x27;t count &quot;pageable weirdness&quot; in their failure metric).<p>It&#x27;s actually one of the things I love about Erlang, and how it&#x27;s changed my thinking. Think about failures. Or rather; don&#x27;t. Assume they&#x27;ll happen, in ways you can&#x27;t plan for. Instead think about what acceptable degraded behavior looks like, how to best ensure it in the event of failure, and how to automatically recover.

I think git is a good model for what would otherwise be &quot;laggy async and mismatched&quot; distributed systems.<p>It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you&#x27;re hitting the network, rather than hitting it ALL THE TIME.<p>-----<p>I would like to use something like git to store the source code to every piece of software I use, and the binaries. That is, most of a whole Linux distro.<p>I have been loosely following some &quot;git for binary data&quot; projects for a number of years. I looked at IPFS like 5 years ago but it seems to have gone off the rails. The dat project seems to have morphed into something else?<p>Are there any new storage projects in that vein? I think the OP is identifying a real problem -- distributed systems are unreliable, and you can get a lot done on a single machine. But we are missing some primitives that would enable that. Every application is littered with buggy and difficut network logic, rather than having a single tool like git (or rsync) which would handle the problem in a focused and fast way.<p>It would be like if Vim&#x2F;Emacs and GCC&#x2F;Clang all were &quot;network-enabled&quot;... that doesn&#x27;t really make sense. Instead they all use the file system, and the file system can be sync&#x27;d as an orthogonal issue.<p>Sort of related is a fast distro here I&#x27;m looking at: <a href="https:&#x2F;&#x2F;michael.stapelberg.ch&#x2F;posts&#x2F;2019-08-17-introducing-distri&#x2F;" rel="nofollow">https:&#x2F;&#x2F;michael.stapelberg.ch&#x2F;posts&#x2F;2019-08-17-introducing-d...</a>

See also Peter Deutsch&#x27;s &quot;Fallacies of Distributed Computing&quot; list (<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Fallacies_of_distributed_computing" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Fallacies_of_distributed_compu...</a>).<p>There&#x27;s some overlap, but also some new stuff. In particular, &quot;pipes&quot; isn&#x27;t covered by the Fallacies list and is consistently a pain point and&#x2F;or issue you always face in some way. Also &quot;asynchronous&quot; isn&#x27;t covered by the Fallacies list.

&gt; Sometimes a distributed system is unavoidable, such as if you want extreme availability or computing power, but other times it’s totally avoidable.<p>But so much of our sales pitch involves these shiny cloud systems.<p>Who ever sold business by telling the customer: &quot;Your use-case really isn&#x27;t exciting, and a boring batch-driven process is completely appropriate&quot;?

I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It&#x27;s how you outsource expertise.<p>The vision of the service mesh is to make unreliability and security no longer the job if the application binary. Even without a service mesh, you can put a lot of common functionality into a reverse proxy. Personally I am loving OpenResty for the simplicity of writing adapters and oauth at the proxy layer with good performance.

This article has a point.<p>But, as in all things software, &quot;it depends.&quot;<p>It depends on what the tools are, and what we are writing.<p>In my own case, I have the luxury of writing fully native Swift code for Apple devices. I don&#x27;t need to work with anything off the device, except for fairly direct interfaces, like USB, TCP&#x2F;IP or Bluetooth.<p>Usually.<p>I have written &quot;full stack&quot; systems that included self-authored SDKs in Swift, and self-authored servers in PHP&#x2F;JS.<p>I avoid dependencies like the plague. Some of them are excellent, and well worth the effort, but I have encountered very few that really make my life as an Apple device programmer that much easier. The rare ones I do use (like SOAPEngine, or ffmpeg, for instance), are local to the development environment, and usually quite well-written and supported.<p>If I were writing an app that reflected server-provided utility on a local device, then there&#x27;s a really good chance that I&#x27;d use an SDK&#x2F;dependency with network connectivity, like GraphQL, or MapBox. These are great services, but ones that I don&#x27;t use (at the moment).<p>I&#x27;m skeptical of a lot of &quot;Big Social Media&quot; SDKs. I believe that we just had an issue with the FB SDK.<p>That said, if I were writing an app that leveraged FB services, I don&#x27;t see how I could avoid their SDK.<p>So I write fully native software with Swift, and avoid dependencies. That seems to make my life easier.<p>But Xcode is still a really crashy toolbox.

Just reading the title I assumed it was a post about business processes and communication between teams. Because this is how working for a big corp sometimes feel.

Really like the curiosity and thought behind this article. Couple of thoughts:<p>&gt;probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed.<p>Sure. Do it on one giant machine. Then you&#x27;ll be spending 80% of your time doing things you wouldn&#x27;t need to do if it weren&#x27;t monolithic.<p>At the end of the day, if your customer is on the other end of the internet, then all of those complaints apply. If you solve that by running an app on their device, then oh boy are you going to have fun testing.<p>I prefer scaling out. The stackoverflow peeps prefer scaling up. There are some great write-ups about how they scale. I found this [1] one after some quick googling, but I am certain there are more. So it&#x27;s really about choose your poison.<p>&gt;I think people should be more willing to try and write performance-sensitive code as a (potentially multi-threaded) process on one machine in a fast language if it’ll fit rather than try and distribute a slower implementation over multiple machines.<p>Sure. I once replaced a system that ran on 10 32-core machines with one that ran on one with four cores on one machine and did the work in the same time. Another time I had 96 cores, more threads, and I replaced it with one that had three threads and was faster.<p>But both of those solutions were evolutionary dead-ends. The tasks were very specific, and not subject to change. The first one was a single C file. The latter was actually java, but with hand-rolled hash tables and optimistic locks. The first one I doubt I could follow it now.<p>My point is, you can have understandable systems that good people (as opposed to geniuses) can work on, evolve and adapt, and that have well understood failure modes and scaling cliffs. Or you can have bonkers code that everyone is afraid to touch, and which fails in production when it hit a cliff you didn&#x27;t know about and now your site is dead for eight days.<p>If you can strike a good balance, then you&#x27;ll probably have some combination of distributed, and brute force.<p>[1] <a href="http:&#x2F;&#x2F;highscalability.com&#x2F;stack-overflow-architecture" rel="nofollow">http:&#x2F;&#x2F;highscalability.com&#x2F;stack-overflow-architecture</a>

Do you have a moment to talk about our Lord and Saviour, Erlang&#x2F;OTP?

Of course they do. But there&#x27;s no alternative.<p>No matter how fast or beefy your server is, these days if your product becomes a success, 99% of the time it will outgrow what&#x27;s possible on a single server. (Not to mention needs for redundancy, geographic latency, etc.) And by the time you see the trend heading upwards so you can predict what day that will happen, you already won&#x27;t have the time for the massive rewrite necessary.<p>So yes, it&#x27;s tons slower to write distributed servers&#x2F;systems. But what other choice do you usually have?<p>Though, as much as possible, you can <i>try</i> to avoid the microservices route, and integrate everything as much as possible into monolithic replicable &quot;full-stack servers&quot; that never talk to <i>each other</i>, but rather rely entirely on things like cloud storage and cloud database. Where you&#x27;re paying your cloud provider $$$ to not fail, rather than handle it yourself. Sometimes this will work for you, sometimes it won&#x27;t.

I&#x27;ve found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades or maintenance.<p>Common example is data being pushed between systems with HTTP. Take the simplest case of propagating a boolean value. You toggle some setting in the UI, and it sends an update to another system with an HTTP request, retrying on a delay if it can&#x27;t connect. This has two problems. The first is that if the user toggles a setting on and then off, you can have two sets of retries going, producing a random result when the far end can be connected to again. The second, is that the machine doing the retries might get rebooted, and people often fail to persist the fact that a change needs to be pushed to the other system.<p>I&#x27;ve seen this issue between two processes on the same machine, so technically you don&#x27;t even need a network.

&gt; Untrusted: If you don’t want everything to be taken down by one malfunction you need to defend against invalid inputs and being overwhelmed. Sometimes you also need to defend against actual attackers.<p>Sorry, but unless your centralized alternative is only used internally by troglodytes you have to at least defend against invalid inputs.

The title reminded me of the turboencabulator. <a href="https:&#x2F;&#x2F;www.thechiefstoryteller.com&#x2F;2014&#x2F;07&#x2F;16&#x2F;turbo-encabulator-best-worst-jargon&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.thechiefstoryteller.com&#x2F;2014&#x2F;07&#x2F;16&#x2F;turbo-encabul...</a>

Networking and distributed software are well understood nowadays, the lower productivity of web companies vs. SpaceX etc. comes from all the unstable (both ever-changing and buggy) software they need to use. Most modern software is affected by this, but the web has it worse because of security issues and because the way browsers are evolving (on purpose, one has to add, because it&#x27;s a cartel of large players on the web trying to stifle competition). SpaceX doesn&#x27;t get some innovative new alloy they didn&#x27;t order every couple of weeks and even the games industry has fewer obstructing external dependencies (hardware vendors and their drivers being one).<p>At least that&#x27;s my experience from about 20 years of web development (15 professionally).

Yes, I have a little Python library for managing network shares in Windows.<p>It has things like automatic retries that &quot;back off&quot; slowly, switching to cached IPs in case DNS is down, and checking to see if all of the drive-letters are full and either re-using a letter or creating a &quot;letter-less&quot; share. I had to develop it during a period of great instability within our network. It&#x27;s ... large and over-engineered, but it just keeps on truckin&#x27;.<p>On the other hand, it has been quite useful going forward, so that&#x27;s a plus.<p>I tend to program fairly defensively, in layers, right down to the much maligned Pokemon exception handling. The results don&#x27;t have the, ah, velocity that is so often praised but they&#x27;ll be there ticking along years later.

Hah, I saw the title &quot;Fragile narrow laggy asynchronous mismatched pipes kill productivity&quot; and thought it was about the pitfalls of trying to coordinate remote teams across disparate time zones.

&gt; I hope this leads you to think about the ways that your work could be more productive if you had better tools to deal with distributed systems, and what those might be.<p>We have tools. Promula&#x2F;SPIN model checker is one just off the top of my head.

Just some fun with the English language:<p>&gt; probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed.<p>If you don&#x27;t ever plan on distributing your software you can save a _lot_ of time :)

Funnily, I thought the headline was talking about development process, as that also describes how a lot of places (mis-)handle the flow of what get worked on.

&gt; I also think all these costs mean you should try really hard to avoid making your system distributed if you don’t have to.<p>There&#x27;s a point here about microservices.

It&#x27;s perfectly applicable to people too; one stickler for the rules or slow working in a critical role, say security officer, or change board chair can kill productivity.

I feel so attacked right now.

TLDR he means microservices. We now have a generation that doesn&#x27;t even recall a non-MS world.<p>Tristan, the generation between us has created the IT world you describe. You&#x27;ll probably spend the next 20 years of your career dealing with that mess. Sorry about that.

doh! I thought this was going to be about remote work.