Linear logic has been known since 1987. The first release of Coq (dependent types for functional programming and writing proofs) was in 1989. The HoTTBook came out in 2013. Ada/SPARK 2014 came out the same year as Java 8 did. We also witnessed the Software Foundations series, the CompCert C compiler, the Sel4 microkernel, and the SPARKNaCl cryptographic library.<p>Instead of learning about those achievements and aiming to program for the same reliability, clarity, and sophistication, we see an abundance of software that cannot clearly describe their own behavior nor misbehavior.<p>Instead of incorporating the full functionality of XML/HTML/CSS/SVG/JS/WebGL into the development experience and providing ways to control them at the fundamental level, we reinvent crude approximations like the various web frameworks.<p>YAML and JSON often trumps XML/XSD until things get out of control, and even then, people still don't learn the lesson. Protobuf, flatbuffer, capnproto, and the like keep reinventing ASN.1.<p>Naive microservices partially reimplements Erlang's BEAM VM while ignoring all the hard parts that BEAM VM got right. Many people riding the microservice bandwagon have never even heard of Paxos, not to mention TLA+.<p>Many programmers keep learning new shining frameworks but are reluctant to learn about the crucial fundamentals, e.g., Introduction to Parallel Algorithms and Architectures, nor how to think clearly and unambiguously in the spirit of Coq/Agda/Lean.<p>No wonder ChatGPT exposes how shallow most of programming is and how lacking most programmers are in actual understanding. Linear logic and dependent types are there to help us design and think with clarity at a high level, but people would rather fumble around with OOP class hierarchies (participate in the pointless is-a/has-a arguments) and "architecture" design that only complicate things.<p>What is this madness? This doesn't sound like engineering.
> Why is there so much useless and unreliable software?<p><a href="https://twitter.com/maxkreminski/status/887815522061926400" rel="nofollow">https://twitter.com/maxkreminski/status/887815522061926400</a><p>"a reminder: if inexperienced creators are using your tool to churn out loads of half-baked garbage, your tool is a phenomenal success"<p>Software is such a powerful concept - it basically imbues physical objects with magic - that even bad software is hugely empowering to its users and takes off very quickly. The demand is staggering.<p>I appreciate that when you see yourself as the most intelligent person in the world it becomes intolerable to be surrounded by unthinking muggles scratching in the dirt, but after a while you realise that life is more complicated than that, people usually have good reasons for doing the things they do, and that perfection is neither attainable nor necessary for most of that.
As a software agency owner I can tell you that barely anyone wants that. We maybe get 1-2 requests for quality software every year. Most of our clients want their apps built as fast as possible, with little to no consideration about the tech quality. We have the choice of doing it that way or going out of business.<p>This also affects our people (usually our top engineers) - which is why I want to start developing our own products this year.
Engineering is about tradeoffs.<p>When you're shooting a multi-hundred million satellite into orbit it's worth the extra few million expense of formal verification because otherwise you might lose a gigantic investment and even kill people in the process.<p>When you're working on something without such extreme constraints, with vague SLAs, and limited to no business risk, then regular unit/integration/etc tests are good enough and exceedingly cheaper.
For context: I come from a mathematical background. I didn't particularly appreciate that most programming languages use the word "function" for something that is not a mathematical function (e.g. has side effects). Well, then I got used to it that sometimes <i>it is practical</i>.<p>> This doesn't sound like engineering.<p>It is precisely engineering. As opposed to pure science and art. (I consider mathematics to be, above all, art.)<p>Based on the post itself, you come with a theoretical mindset.
You may consider purity to be more important than practical applications. Yet, if people write software to be used, they focus on the latter. Sometimes it results in hacky code even within an already hacky language. There are no extra points for purity.<p>Purity itself is a double-edged sword. Sometimes it makes the code more reliable. Other times - it generates a lot of abstract nonsense, which makes it harder to reason about the piece of software or change it.<p>On the positive side, look at the Rust language (and community!). While it has lovely abstractions and safety guarantees, it is a practical language - performant for writing and execution.
Why? Because of the barely competent programmer, like myself. A DIY army.<p>Is XML better than JSON? For longterm stability sure. A quick config file? Nope. You see, to understand XML you kind of have to know how to work with trees. Bread-and-butter for compsci grads, a nuisance for all others.<p>Recently I needed to query a SOAP-based API. It took me 3 days because I had no idea how a namespace in XML worked, because I was not sure which lib in Python to use (lxml was the solution), and because their API had some quirks. I read forum posts from the late 00's, the documentation is really scarce.<p>I would love to learn the fundamentals, I don't like half-assing things. But a) I'm not going back to uni for it, b) I'm not bright enough to be excellent in two fields. Learning the fundamentals takes a very, very long time. Where to start even? A complete math program? Programming fundamentals as such? Theoretical compsci?<p>At the end of the day, things need to work. If they fall apart next year I can tell my boss "ooohhh big problem give money". Using XML instead of JSON has no payoff today.
Nice rant but the answer is simple - incentives.<p>Seems like there's not that much demand for what you would like to see.<p>Crappy, barely working, software now is better then 1 year delayed perfect one.
In construction, or most other fields of "traditional" engineering, projects will take years from start to finish. That's completely normal.<p>In the world of software - especially for startups - it is completely unacceptable to spend 5 years on building a MVP. There are some protected sectors where things move slower (defense, aerospace, for example), but if you're in the consumer field, you just can't spend too much time. You want to push out a MVP as soon as possible, and build on that.<p>And most software companies do not get penalized on shipping buggy software. Big game studios ship broken games, and spend a couple of years patching them up to their final form. People bitch and moan, but still throw money at 'em.<p>If you don't want bloated, broken, slow, and unreliable software - use your pocketbook.
The madness is insisting that applications, instead of operating systems, enforce security. In the desktop era, the OS was designed to blindly trust applications to do the right thing, and everyone went along with it.<p>That was the seed of the madness.<p>Instead of supplying dialog boxes(open, save, etc) for applications to use and then open files directly, the OS could have supplied handles (capabilities) from those dialog boxes for the applications to use. This would have allowed the user interface to be almost identical, and only require a few lines of code in the applications to change, in exchange for an environment which was almost immune to confused/rogue programs.<p>Because expectations were so corrupted in the desktop days, the situation now is effectively hopeless.<p>Applications should NEVER be trusted, especially not by the operating system.
Many possible reasons, including:<p>- no engineering standards in computer science, including especially in education of computer science<p>- some corporate finance views which see technology as a cost center rather than a business enabler<p>- some corporate strategies where marketing decides what is possible and when it must be ready (tomorrow. or yesterday.)<p>- computing and software development as a "fun", creative endeavor - as opposed to a rigorous, formal process<p>I'm sure there are dozens more reasons.
It's simple. Hardly anyone wants to pay the premium price for quality software. Software is not physical goods and its real value comes from the things it can do it does not matter how it can do them.
Pareto principle. The last 20% take most of the work. People are not willing to pay for it (or at least that's what sales/marketing believe), so the effort is not being spent. It's as simple as that.
I feel your pain, but you really aren’t gonna be able to change it.<p>As long as people are willing to pay for dross, people will continue to produce it. Just look at Hollywood, for a classic example.<p>For myself, I found a company that was interested in producing Quality, and spent most of my career, there. It had its trade-offs.<p>It certainly didn’t get me much respect from today’s software development community. If anything, it gets me scorn and outright disrespect.<p>I had to retire and work on my own (for free), in order to finally get to write software the way that I want. I doubt it would be considered commercially viable.<p>In any case, “doing it right” is in the eye of the beholder. The fact that I'm not writing all my native Apple software in Haskell, is "doing it wrong," to some folks. Don't even get me started on OOP or using UIKit/AppKit.<p>Cant' please everyone.<p>I’m best off, putting my values into practice, in my own context, and not being an old man, yelling at the sky.<p>But I really want to. It just won’t do any good.
JSON is used because it is much simpler to generate and parse in browsers than xml.<p>It is also "good enough" for most use cases. So it is reasonable to start out with JSON and only upgrade to XML for complex documents/strutures.
Also a different take on the problem is so much software is unreliable from a data perspective. We trust these software providers with GBs of our data, yet they all have different backup strategies, are using different consistency and ACID guarantees, and are not at all experts at managing and backing up data.<p>If we could abstract away the storage so that it is user owned, or at least a "utility" like electrical power lines where everyone is playing by the same rules, we would be able to trust these platforms significantly more, and rest assured our data is safe even if the software isn't.
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."<p>( Melvin E. Conway, <a href="https://en.wikipedia.org/wiki/Conway%27s_law" rel="nofollow">https://en.wikipedia.org/wiki/Conway%27s_law</a> )<p>Thus, we may conclude the world has some people that choose to be "useless and unreliable". The software part is just an artifact of this ideology. =)
Engineering is not always implementing the best solution, but implementing a solution that completely or partially solves the problem at hand. In engineering, the fact that the solution is suboptimal is immaterial compared to the outcome. It may have ramifications to future problems that are possible following that, but that is a separate engineering problem that can be considered separately and decided upon accordingly.<p>Transferring ideas and software discovery are difficult problems. Expecting everyone to know that tech x is superior to tech y and singularly pursue development of tech x is unreasonable.<p>Most people prefer work with a solution that you are familiar with or caters to a particular technology stack. JSON is a first-class citizen of Javascript, but can be secondary in other programming languages. Anything that provides the least cost or effort to resolving or minimizing the extent of the product at hand is all that is needed. There is almost never a case where point B must follow or supersede point A. Point A continues to solve the problems it was designed for or turned out to be useful for by chance: the technology can continue to be used where it is known and regarded.
You don't have to use software, frameworks, tools that don't meet your standards.<p>I find that the learning curve to get on top of most abstractions is greater than what is required to understand the foundations. Knowing and applying foundational components tends to yield more performant and less bloated solutions. Debugging is easier because you don't need to unravel layers of abstractions which don't quite align with the domain you are working with.
Most companies don't do real cost/income analysis on any work. Most internal work is just approved by a department leader/tech lead. So most companies tend to overhire and do menial busywork, like constant refactoring of robust and well-working code, to catch up with the latest and greatest frameworks.
The latest offender is vite, offering minimal gains (faster dev builds) at a non-negligible dev cost (and source of new bugs).
There isn’t any external pressure to give people an incentive to do better. Until that arrives, we’re stuck in this crap software local minimum.<p>I’m hoping such pressure arrives in the form of legal or regulatory stuff, even if it chills the industry and slows/shrinks it. Until then we’re still in the computing equivalent of the auto industry when people died in mild fender benders that today are just mild annoyances.
My two cents is that you are optimizing for engineering excellence. A more cynical person would optimize for making as much money in their career as humanly possible, which is made easier by pitching new tech that seemingly "Solves all the problems that old stuff had". A non-techy business person will naturally want to win market, beat competition, and do things faster. If you push that really hard you get snake oil engineering and churn out garbage.<p>Also, I assume most people fall in the middle. Wants to make money but also wants to do a good job. It just turns out those with the purse strings have the incorrect assumptions about tech, and if you know anything about business people you know that a laundry list of facts and reasoning is not always a good way to convince some of them of what should be done. More than likely a majority would see an article about a company going big with technology X. Now they have it in their head they need that too.
The harsh truth of the matter is that in B2B software engineering, you can usually get away with providing a shitty product, then charging to fix the bugs you introduced. It's worse than 'clients aren't willing to pay for a premium product', it's that they are willing pay you to clean up your own mess, but not willing to pay a much smaller sum upfront.<p>Internally at my small business we have stopped using the word bug, and started calling it defect to change the thinking about this. Bugs sound innocent, like they just 'creep in' and there is almost nothing you can do about it as a software engineer. Defects can be prevented, mitigated, or fixed. It places the responsibility in our hands again to deal with it and prevent it. Zero defects is probably not a realistic target (same with any production line for hardware), but it does help to frame it in the same way and take proper responsibility.
The reason is complexity, not necessarily that the companies don't care. Most end-consumer software is highly complex, using a vast number of dependencies, programming has become similar to LEGO brick building. Libraries are written by normal programmers, too, and even carefully designed libraries contain errors. Since the interfacing between libraries also produces bugs, the end product will become buggy simply as a matter of combinatorics.<p>The only way to avoid this complexity is by keeping programs simple or by writing everything in house in a language like Ada/Spark very carefully, including formal verification and extensive testing. This is way too expensive for non-critical software. Btw, I don't think Erlang's concepts are general solutions, fault-tolerance by restarting services only eliminates certain kinds of problems and isn't suitable for all high-integrity demands.
Because it's not hard engineering where you can't rebuild the shitty foundations after you've poured the concrete. You can rebuild those multiple times a day in software, hence it's much more iterative.<p>I question what's your age and background, this view of the world is something I held when I was pretty young and thought everyone else were just stupid muggles. Until reality hits you in the face and you start to comprehend the constraints of the real world, software isn't some pure intellectual pursuit; it's a tool, a very very powerful tool that's used to enhance the real world somehow.<p>You can play around with pureness of theory if you want in academia. In the real world there are customers, they have needs and they want their needs fulfilled by a business, the business that achieves that earns profits, that's all there is to it.
When I think of bad software I think of software that works but has bad or hostile user experience. This is harder to solve than just learning linear logic and dependent types.
Making a useless and unreliable (say) toaster takes a <i>lot</i> of time and money. People notice fast that it's crap. Very few people are overly impressed by toaster styling and "features", or clueless about what a toaster is supposed to accomplish, or quietly intimidated by toasters, to overlook the fact that a crap toaster is crap, or quietly blame themselves.<p>Vs. with software, ~any clueless wanna-be kid can make some in an afternoon, at trivial cost. A PHB who could never manage to get a toaster designed & manufactured (let alone sold) can "lead" a team of crappy developers in creating some {cough} great {cough} new pieces of software.
The adoption curve of advanced technologies that solve it all. Is it just the cognitive burden of the tooling or the concepts, are the training methods different, a dearth of already trained talent and sufficient post-graduate or post-doctoral instructors, unclear signals between demand and a supply funnel?<p>The limited availability of Formal Methods and Formal Verification training.<p>The growing demand for safety critical software for things that are heavy and that move fast and that fly over our heads and our homes.<p>In order to train, you need to present URLs: CreativeWork(s) in an outline (a tree graph), identify competencies (typically according to existing curricula) and test for comprehension, and what else is necessary to boost retention for and of identified competencies?<p>There are many online courses, but so few on Computer Science Education. You can teach with autograded Jupyter notebooks and your own edX instance all hosted in OCI containers in your own Kubernetes cloud, for example. Containers and Ansible Playbooks help minimize cost and variance in that part of the learning stack.<p>We should all learn Coq and TLA+ and Lean, especially. What resources and traversal do you recommend for these possibly indeed core competencies? For which domains are no-code tools safe?<p>If we were to instead have our LLM (,ChatGPT,Codex,) filter expression trees in order to autocomplete from <i>only</i> Formally Verified code with associated Automated Tests, and e.g. Lean Mathlib, how would our output differ from that of an LLM training on code that may or may not have any tests?<p>Could that also implement POSIX and which other interfaces, please?
There are a lot of very good answers already in this topic:<p>- quality doesn't bring enough to the business value;<p>- high demand requires hiring people without enough knowledge/skill;<p>- other people have just different quality metrics than you;<p>- "know-how" transfer between generations is difficult...<p>IMHO the similar question may be asked about any domain of life: why not all the buildings are well-built? Why the political/social systems are not just? Why good art is so hard to spot nowadays? They have different answers, but there is one common theme: building "well" is difficult; creating a "just" political system is "difficult", creating a <i>good</i> art is difficult; also writing "useful and reliable" software is difficult.<p>It's only natural that not everybody is a perfect software engineer, therefore is not a surprise that not every software will be perfect. What's more, there are a lot more mediocre software engineers than the perfect ones, therefore the majority of software will be... mediocre. The same about art nowadays — the sheer amount of music produced (even the word "produce" suggests it) every day makes sure that, in overall, there is a lot more terrible music compared to good compositions. I believe that software engineering has the same issue — the rising number of software projects is an enough explanation for low quality of the software we are able to perceive.<p>In fact, I don't believe the quality of the software engineering degraded over the years, but it's just more difficult to spot the great projects in the avalanche of "normal" life.
> What is this madness? This doesn't sound like engineering.<p>To me this sounds exactly like engineering.
You craft the tools / apps you want, you build frameworks around complicated concepts that simplify their understanding and usage, at the expense of losing some fundamentalities.<p>You don't need to know the history and all the evolution of technology to apply it.
It is part of software science, for sure though. You can be good at software science and fundamental concepts, but this also does not imply you are a good software engineer.<p>Would software in general be better if mentality proposed to you would be standard?
I doubt, because the learning curve to enter the industry and even begin to start doing something would be immense, you would need to study as much as doctors now do, and only after 10 years you would be able to be "trusted" with your work.<p>I do agree though that there is a bare minimum that one who calls themselves "Software Engineer" should know and understand, like OS fundamentals, basics of compiler theory, etc., etc., but it is not so restrictive as you suggest.
I blame the desktop era. From the commercial standpoint, robustness is as good as reliability. And in the desktop paradigm, all the systems are robust enough because when something fails, there is always a user. A human part of the system that restarts the operation, or reinstalls the software, or, if nothing else works, buys an update (the best outcome for the software provider).
Because the cost of an error in different systems is different, and it is disproportionately lower than the cost of formal verification in 99.999999999% of cases. Where verification is needed and rational, it has long been used.<p>Because the only thing in common between the development of intangible products (software) and the engineering of tangible products is the cost of implementation.
Alan Kay has done a few lectures about exactly this phenomenon: his talk 'Normal Considered Harmful' in particular is worth a look as it pretty much goes straight to the heart of the issue: <a href="https://www.youtube.com/watch?v=FvmTSpJU-Xc">https://www.youtube.com/watch?v=FvmTSpJU-Xc</a>
Searching for any of the comparison examples will surface the reasons. Rather than rant about it, why not propose a theory that generalizes the observations?<p>Personally I would tend to agree, that using a sharper more capable tool is moving in the right direction. Where I disagree is that we can't all reach for the sharpest, most capable tools, as that would render many/most of us ineffective.<p>The next question is if only the people capable of using the sharpest, most capable tools can satisfy all production needs. That might be possible for core things such as energy production, but likely not possible for all things down to the mundane. Or to remove judgement/controversy of 'mundane', all things people are willing to pay for.
That <i>is</i> an excellent question! I have been asking that myself all my professional life and I can now make an educated guess:
lack of planning and mediocre specification.<p>I've read some comments here that I agree with; in a lot of cases the goals in dev are set to implement a solution to a problem either as fast as possible, or without a limited scope.<p>"We need a software that can fly rockets" - Yeah, but where to?<p>I studied computer science twice. Both times I have been taught to implement corner cases instead of working on clearly defined projects with scope and specification. "Write a small program that does X" yeah, but the solution space is unlimited, then, maybe? Can you elaborate? Put a scope on it? I mean, can do, may I order compute time on the campus super-computer?<p>Crap software was there ever since, and it got worse ever since.
Programmers are driven by the need to get stuff done and they migrate towards technologies that do that quick and that are sufficiently documented.<p>You want your "fundamentals" to be learned? Make knowing them useful to achieve something and document them in nice skimmable, searchable manner.
One pattern I noticed in the workplace: People are mocking things like LISP, Haskell, or Erlang, but get a hard-on when their bread language (e.g. Python) introduces new syntax for, say, pattern matching (rather a pattern-aware switch/case, because it isn't an expression).
Software is dysfunctional because everything is dysfunctional. Any human endeavor done at scale is going to be a complete clusterfuck of a trainwreck. It's just more obvious with software because we're closer to it.<p>Civilization is a goddamn miracle.
> What is this madness? This doesn't sound like engineering.<p>Did you hear that everyone can build a bridge that stands but it takes an engineer to build a bridge that barely stands?<p>I'd say software development has a lot to do with engineering then.
> Protobuf, flatbuffer, capnproto, and the like keep reinventing ASN.1.<p>This is a strange argument in the context of your other complaints (which I largely agree with fwiw). All of them are <i>much safer</i> to parse than ASN.1, which has been a decades long security nightmare, and add additional security concepts on top.<p>Never mind that they're also networked IPC protocols on top of binary serialisation formats, solving a host of security issues we've had with exchanging serialized Java/Python/PHP objects over the wire, DCOM, etc. pp.<p>Damned if you do, damned if you don't?
Craftsmanship is not valued the same way by everyone, and it costs a lot more. The people paying for software (either in salary or in license fees) have to choose their battle: quick and cheap is generally the default target.<p>Most likely, you also chose cheap over quality at some point in your life. I have furniture that would make a woodworker cry, sportswear that raise eyebrows, unreliable WiFi and bad cables. How can I judge people that don't even know what quality software is?
Like any other product, software has different levels of quality to it.<p>Most people are not prepared to pay for that level of excellence. They want the cheapest thing as quickly as possible.<p>The only way that this will improve is if the "baseline" quality of software improves so the average coder automatically produces better stuff because the popular tool of the day is providing it for "free" without them really thinking about it or having to opt-in to it.
Because of "Agile" planning practices. Get things barely working (MVP) then move onto the next project to make sure your manager's manager stays happy.
> Naive microservices partially reimplements Erlang's BEAM VM while ignoring all the hard parts that BEAM VM got right<p>Can someone elaborate or give hints more about this?
Because it is funnier to write new shiny things than boring, reliable things.<p>I have an education in computer science, I know how to write a Paxos setup, I am fluent in Erlang. I am still very much not interested in formal systems, proofs and verified computing, because there I will spend 10x much time for the same set of features. Bugs are normal. I will live with bugs. I won't live with 10x development time lost.
Welcome to normal life. Almost everything is crap. Almost nothing is like a high quality event, if you mean. There's a thing that maybe you have heard about called Normal distribution. You can only call something high quality because a little few extraordinary things happen. Otherwise it would all just be ordinary.
Welcome to normal life. Almost everything is crap. Almost nothing quite like a high-end event, if you will. There's something you may have heard of called the normal distribution. You can only call something high quality because some extraordinary things happen. Otherwise it would all just be ordinary
It all boils down to capitalism. Capitalism won't allow you to spend time and money building quality, when someone else can do the same job twice as fast and for half the time. Getting first to marked is more important for most than having software that work all of the time.<p>The only reason that other engineering professions does things properly is that bridge building is not free marked capitalism, but government mandated rules. Just look at how much GDPR rules in EU does for actually making companies do the bare minimum to respect privacy.
Unlike most fields, anyone with a computer can program<p>Ivory tower academics will keep insisting that there are ways to make foolproof software.<p>If it were so, they could do that and make millions and trillions.<p>Software is not a science, its a craft, it depends on the outside world, not some cute lab demos
Because people pay for it.<p>People don't pay for incorporating full functionality of "XML...Whatever else" they pay for ability to see cute kittens NOW not when dev will be done with reading and understanding ASN.1 well enough to use important parts.
1. There are much less really good programmers than people who think they are ones.<p>2. Business dont care much about quality, as long as the product is good enough to sell - real improvement and development stops there.
To some extent I agree - I do see a vein of anti-intellectualism in software. But “engineering” isn’t about using the most theoretically-perfect setup for any given job, it’s about navigating practical trade-offs.
The software in a typical enterprise has to cross a lot more domains then it used to, and has to work with independent organizations over the internet. No surprise then that it has more confounding elements.
There was a great white paper investigated how software gets bigger and bigger. They used the command line tool true as an example. I can't find it anymore but it was really an eye opener.