A Generation Lost in the Bazaar (2012)

This is always going to be a problem with dynamic linking. It happens with dynamic linked libraries. In order to use some function, you have to bundle and depend on a whole library. That library may have functionality you don&#x27;t need, which may make it reliant on other libraries, and so on. And if two of the libraries you depend on, in turn depend on different versions of some other library, you get dependency hell.<p>But libraries are just one level of fractal self-similarity. Packages bundle libraries and other supporting tools, so one package depends on others, containing tools and functionality you never use. And packages suffer the same dependency hell.<p>Then we invent even higher abstractions to make the dependency hell go away. Maybe Docker containers. But then what happens when you need assemblies of those? Pods of Containers?<p>This is a problem with the dynamic linkage model of code reuse. Not that static linkage solves the problem totally, but it doesn&#x27;t have the same dependency bloat, and dependency failures occur in the development phase, rather than during deployment (or worse, many months later).<p>So yeah, Cathedrals are more elegant, and better code.<p>But, so what? Cathedrals might be nice to go sightseeing at in a new city. But they took years to complete, were funded by government and rich benefactors, and are mostly empty. I&#x27;m sure God appreciates them; but they are largely irrelevant to most mortals.<p>I&#x27;ve been doing this for more than 20 years, and my old self cringes when I download a 4 megabyte package for anything, given what we used to achieve with 4Kb. But I can&#x27;t deny that I can get a product out much quicker now than I used to. It might not be as elegant, or perhaps as maintainable, but the market doesn&#x27;t care: I can worry about that when I get revenue. If I spend time building a cathedral, a bunch of others are going to come build a bazaar outside the door and clean up, well before I&#x27;ve put down the foundations.<p>I agree with everything he writes, but I don&#x27;t see any practical alternative.

I must admit to being a bit puzzled as to what his point is.<p>Yeah, there are a lot of dependencies if you want to compile an entire linux distro from source. But it&#x27;s just exposed because everything is right there. If you actually tried to figure out how to compile all the software on your <i>Windows machine</i>, would it be any better?<p>And to say that libtool, bloated and complex as it is, came about because someone was <i>incompetent</i> seems quite a bit insulting. It seems that confronted with a compatibility problem between different Unix-like systems, one has two choices:<p>(1) Coordinate changes to all Unix-like systems so the compatibility problem is removed, or<p>(2) Work around the problem with some sort of wrapper.<p>Now, in a perfect world, (1) is obviously the better alternative (but even that wouldn&#x27;t help existing installations.) But the world is <i>not</i> perfect, and my chances of accomplishing (1) are practically zero, even if it&#x27;s <i>&quot;just a single flag to the ld(1) command&quot;</i>. Hence, anyone who actually wants to get something working on all those systems would have to do whatever it takes.

I have always enjoyed that essay, it reminds me of something my Dad once said to me, &quot;Cannibal&#x27;s don&#x27;t realize they are cannibals.&quot; It wasn&#x27;t strictly true of course but it tried to capture the essence that if you&#x27;ve grown up in a society where X is the norm, you don&#x27;t know what living in a society where X is not the norm is like or can be like. Combine that with humans who like what they know more than what they don&#x27;t know, and you find communities moving to high entropy &#x2F; low energy states and staying there.<p>That said, there isn&#x27;t anything that prevents people from having standards. Both FreeBSD and MacOS are pretty coherent UNIX type OSes. But it is important to realize that a lot of really innovative and cool stuff comes out of the amazing bubbling pot that is Linux as well. I sometimes wish it were possible to do a better mashup.

The article author points out several problems in the open source world.<p>The first is that it&#x27;s too hard to converge things that have diverged. I pointed out an example in a Python library recently - the code for parsing ISO standard date&#x2F;time stamps exists in at least 11 different versions, most of them with known, but different, bugs. I&#x27;ve had an issue open for two years to get a single usable version into the standard Python library.<p>Some of this is a tooling problem. Few source control systems allow sharing a file between different projects. (Microsoft SourceSafe is a rare exception.) So code reuse implies a fork. As the author points out, this sort of thing has resulted in a huge number of slightly different copies of standard functions.<p>Github is helping a little; enough projects now use Github that it&#x27;s the repository of choice for open source, and Git supports pull requests from outsiders. On some projects, some of the time, they eventually get merged into the master. So at least there&#x27;s some machinery for convergence. But a library has to be a project of its own for this to work. That&#x27;s worth working on. A program which scanned Github for common code and proposed code merges would be useful.<p>Build tools remain a problem. &quot;.&#x2F;configure&quot; is rather dated, of course. The new approach is for each language has their own packaging&#x2F;build system. These tend to be kind of mysterious, with opaque caching and dependency systems that <i>almost</i> work. It still seems to be necessary to rebuild everything occasionally, because the dependency system isn&#x27;t airtight. (It could be, if it used hashes and information about what the compiler&#x2F;linker&#x2F;etc actually looked at to track dependencies. Usually, though, user created makefiles or manifest files are required. We&#x27;ve thus progressed, in 30 years, only from &quot;make clean; make&quot; to &quot;cargo update; cargo build&quot;.<p>The interest in shared libraries is perhaps misplaced. A shared library saves memory only when 1) there are several <i>different</i> programs on the same machine using the same library, and 2) a significant fraction of the library code is in use. For libraries above the level of &quot;libc&quot;, this is unlikely. Two copies of the same program on UNIX&#x2F;Linux share their code space even for static libraries. Invoking a shared library not only pulls the whole thing in, it may run the initialization code for everything in the library. This is a great way to make your program start slowly. Ask yourself &quot;is there really a win in making this a shared library?&quot;<p>Shared libraries which are really big shared objects with state are, in the Linux&#x2F;UNIX world, mostly a workaround for inadequate support for message passing and middleware. Linux&#x2F;UNIX still sucks at programs calling programs with performance comparable to subroutine calls. (It can be done; see QNX. When done on Linux, there tend to be too many layers involved, with the overhead of inter-machine communication for a local inter-process call.)

Earlier discussion, including a number of comments from the author, here:<p><a href="https://news.ycombinator.com/item?id=4407188" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=4407188</a>

I always understood this article as being about software design in open source.<p>Every project (library, application, you name it) needs to be designed, by one or more people thinking deeply about the problem space. You can&#x27;t expect software design to just happen, once the code has been written, it&#x27;s already too late, see you in v2. You can&#x27;t expect a 100 people to do it, there will either be a lot of conflicting visions, or no vision at all. And you can&#x27;t expect to do it in your 1 hour of &quot;free coding time&quot; a day, because it doesn&#x27;t give you enough time for deep thinking about the problem.<p>If you try to bypass design and solve it from &quot;another angle&quot;, you get libtool, a solution that is a hundred times more complex than the problem it solves.<p>Look at successful open source projects (Rails, Go, even Linux). They were all initially designed by someone, handed down to the community to improve. They still have strong minded architects leading the effort. Now compare it to those random &quot;Let&#x27;s clone X!&quot; threads that never produce anything.<p>So, there&#x27;s cathedral thinking even in the bazaar. And it&#x27;s the only thing preventing it from eating us alive.

Good point, very odd way to prove &#x2F; address it, using what is mostly a low level detail. I believe in the &quot;quality happens only when someone is responsible for it&quot; idea, however it&#x27;s not really a matter of automake&#x2F;autoconf, or at least this is an old thing to look at, and a detail. A more modern result of the idea that central authority is not needed in software projects is the bloatware created by the idea that software can be created like this &gt;&gt;&gt; &quot;Let&#x27;s merge this Pull Request, tests passes&quot;. There are three flaws in this apparently non-brainer action:<p>1) &quot;Let&#x27;s merge&quot;: adding features without stress in taking software self-contained and small.<p>2) &quot;This Pull Request&quot;: the idea that there is no need for a central design.<p>3) &quot;If test passes&quot;: the idea that you can really provide quality code <i>just</i> with tests and without manual code quality inspection and writing code in a clean and defensive way to start with.<p>So I believe in the postulate but the proof is rather strange in this post.

I think the core issue the author seems to be getting at is bigger than just the Bazaar (Open Source).<p>Even when it comes to closed source commercial software development I really miss the days of code ownership. When I first started working as a programmer back in the 90&#x27;s it was common for different members of the team to &quot;own&quot; sections of code in a larger system (obviously I don&#x27;t mean &quot;own&quot; in the copyright sense, just in the sense of having one clear person who knows that bit of code inside and out (and probably wrote most of it). Of course we&#x27;d (preferably) still be beholden to code review and such and couldn&#x27;t change things willy-nilly so as not to break the code of consumers of our code, but it was clear to all who to talk to if you needed some new functionality in that module.<p>The last few places I&#x27;ve worked for have been the exact opposite of this where everything is some form of &quot;agile&quot; and nobody &quot;owns&quot; anything and stories are assigned primarily based on scheduling availability as opposed to knowledge of a certain system. There is admittedly some management benefit to this -- easier to treat developers as cogs that can be moved around easily, etc, but my anecdotal belief is that this sort of setup results in far worse overall code quality for a number of reasons: lots of developer cache-misses when the developer is just bouncing around in a very large code base making changes to various systems day to day, lots of breadth of understanding of the system among all the developers, but very little depth of understanding of any individual component (which makes gnarly bugs really hard to find when they inevitably occur) and what should be strongly defined APIs between systems getting really leaky (if nobody &#x27;owns&#x27; any bit of the code it is easier to hack such leaks across the code than define them well, and when your non-technical managers who interpret &quot;agile&quot; in their own worldview force developers to try to maintain or increase some specific &quot;velocity&quot; shit like this happens often).<p>Granted, there are some cases in which such defined ownership falls apart (the person who owns some system is a prima donna asshole and then everyone has to work around them in painful ways), but there were&#x2F;are solutions to such cases, like don&#x27;t hire assholes and if you do fire them.

And yet it all still works.<p>Technology is more complicated than even a few years ago. It can do more. It is accessible to more people (and all of their unique needs and abilities). Computers have the ability to make an almost infinite number of interconnections with other computers.<p>The point is that a single person can&#x27;t possibly keep track of a sufficient quantity of information to direct a sufficiently complex system anymore. And with the communication and development tools available today we are able to build these complex layered solutions without always having to worry about all of the other details that we can&#x27;t possibly worry about.

FWIW, the 19000 lines of &quot;configure&quot; for Varnish also check whether stdlib.h exists. Perhaps it&#x27;s still useful today to do so in order to avoid obscure compilation issues or to catch problems on misconfigured systems early on?<p>As an old-timer with ~30 years of programming experience, I have similar sentiments as the author about complex projects today, yet I also often feel that too much knowledge, accumulated in sometimes cumbersome form, is being thrown away and reinvented badly. There has to be a compromise somewhere and it&#x27;s no surprise that projects in an old language like C, running on evolved systems like Unix, de facto standardized on Autoconf to make it a little easier for developers. Do I want to use it myself? Certainly not, I have the luxury of being able to choose a modern language that abstracts most (not all!) platform-specific issues away at compiler installation time, at the cost of having much fewer deployment options for my code.

I didn&#x27;t know about Coherent:<p><a href="https://en.wikipedia.org/wiki/Coherent_%28operating_system%29" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Coherent_%28operating_system%2...</a> <a href="https://en.wikipedia.org/wiki/Mark_Williams_Company" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Mark_Williams_Company</a>

His observations are sound, but his blaming it on the &quot;bazaar philosophy&quot; doesn&#x27;t really follow. The problem of unused dependencies that he points out with the Firefox port is a failure in packaging, either due to clumsy work with the port itself, or an inability to properly support soft dependencies.<p>I barely understand the voodoo magic behind libtool myself, but as PHK says, it &quot;tries to hide the fact that there is no standardized way to build a shared library in Unix&quot;. I&#x27;d wager dynamic linking inherently poses such quandaries that are easier solved through kludges.<p>Hey, it&#x27;s still probably better than WinSxS.

<p><pre><code> &gt; I updated my laptop. I have been running the development &gt; version of FreeBSD for 18 years straight now, and &gt; compiling even my Spartan work environment from source &gt; code takes a full day, because it involves trying to &gt; make sense and architecture out of Raymond&#x27;s &gt; anarchistic software bazaar. </code></pre> ahh, you should&#x27;ve tried Plan 9, phk. 130 seconds to compile all software and libraries, 6 seconds to compile a new kernel. no bazaar there...<p>of course, this didn&#x27;t appeal to you back then, did it? ;)

There are many related problems to the one pointed out by Kamp. But I&#x27;m left asking, does the cathedral scale? Does it handle evolutionary complexity well?<p>I&#x27;m a believer that a much simpler&#x2F;cleaner set of software tools could be created. But their wide-scale adoption would be more difficult.

There are a few good quality &quot;stalls&quot; in every bazaar, but discovering them, and maintaining some sort of order is difficult. Every package ecosystem suffers from this.

The author is very good at pointing out the problem yet offers no solution of his own. How else do you manage a mostly opensource ecosystem and not end up with different versions of everything? We don&#x27;t get an answer for that. Maybe I should write an article about articles that offer no solution to problems people know exist.

Ah, the good ol&#x27; days when all we needed was netcat and vi, none of this fancy graphics hoohah and kids these days don&#x27;t even know how to spell!! Mind rot, I tell ya! Autocomplete!!! Bah! Use a printed dictionary, you lazy brats!! In my day a web browser supported 3 kinds of images: JPG, GIF, and BITMAP, that was all we needed! ... and you had to draw it yourself with colored pencils, we were all artists ... Now it takes 3 damn hours to recompile my whole friggin&#x27; OS and this cruft and nonsense and doohickies clogging up my disk drive ... no one knows how this stuff works anymore!!! It&#x27;s all gonna fall on your head, and then you&#x27;ll be sorry!

&quot;detour&quot;, to reuse the word of the last paragraph, is how things happen. Look at how we&#x27;ll get our favourite programming languages in the browser: by compiling to javascript, which evolves to become a potent IR (e.g. browsers support asm.js).<p>Even if you look at the end result (the m4&#x2F;configure&#x2F;shell&#x2F;fortran example) and it is indeed twisted, to honestly say it is abnormal to reach such a state is to disregard any experience developing software. Any project, even brought to life in the cathedral style, will accumulate cruft in the long run that can disappear only with effort.

I&#x27;m a bit confused. Is this supposed to apply to software development in general or just package management &#x2F; software repository systems? The author is describing his ideas at a high level of abstraction but I can&#x27;t seem to make a concrete connection. For example, how would one design a web app in a cathedralesque way?

Unix was never much of a cathedral, was it? The original codebase took points of pride in its jank.<p>MULTICS on the other hand...

If a better build system were as easy as Kamp seems to think, he could have written it by now with 1&#x2F;3 of the effort he&#x27;s spent complaining about build systems over the years. Turns out the problems really are hard, and the layers of cruft are there for a reason.

Altenative to a bazaar is not a cathedral. It&#x27;s few huts on a parking lot of a shoping mall. Shoping mall that is superficially neater but becomes abandoned when economy shifts.

Of course, the Linux attempt at a cathedral is systemd.

Still don&#x27;t get it, but I recommend looking at the link someone posted to get the view of the author from when this was posted.

Definitively going to look for &quot;The Design of Design&quot;; Fred Brooks is a master.<p>Though I am relatively young - one of those clueless hacks who arrived at this industry when the dot com party was on its last legs - I can see the blight. Danced with the devil a couple of times, so to speak. It is really depressing to confirm that there&#x27;s no greener pasture to be had anywhere.<p>&quot;A million of replaceable moneys... building an ever collapsing tower... exploring implementation space...&quot; [1] sounds an awful lot like my career so far.<p>[1] Not the author&#x27;s words, but a comment in the original post.

So, a really great example of this is over in Node land. I was trying to install some basic boilerplate HAPI demo scaffolding, and I watched with horror as dependencies were pulled in (as third or fourth level deps!): isarray, isstring, three different versions of underscore and lodash, and so on and so forth.<p>I&#x27;ve <i>never</i> seen developers so lazy or just uneducated about their own language that they blantantly pull in libraries for such trivial operations. On the server even, no excuse about compatibility!