I was interviewed recently. I was asked to implement a solution in Java. It was essentially a stack-based computation engine. The interviewer posed a question and there were enough tacit hints for me to see what he really wanted: an "OO" approach using a type hierarchy + command pattern. He offered unsolicited prompts and I took them: I coupled the command objects to the engine. He liked that.<p>What we ended up with was O(n) _classes_ (where n = the number of computation operations).<p>I remained in Java, but took a functional and dynamic approach yielding a three-line implementation of the engine where each command was simply a function. All the tests passed.<p>I was kind of holding my breath at this point. What will he say? We just went from this huge implementation to three lines. I thought he was going to love it. The crazy simplicity of it all.<p>But to my draw-dropping surprise he muttered something about adding new types and how he liked that the commands were coupled to the engine.<p>Is this complexity worship? Or what is this? It's almost as if his brain was refusing to see the solution. Like it couldn't possibly believe that the solution could be that simple. So it just ejected it, put a big black censorship block over it.<p>I'm currently reading a book on brain hemispheres. Apparently experiments have shown that the left hemisphere will completely and blatantly reject obvious evidence even if the right hemisphere 'has a clear counter proof'. Sometimes I think our industry suffers from an abundance of left hemispheric dominance. Or something like this...
Honestly, I think we as programmers have the opposite problem. We value simplicity so much that we neglect planning for complexity, because we think our simple, elegant solutions will last forever.<p>It's really fascinating to study modern CPU design. Modern high-performance CPUs are horrendously complex, with the very notion of superscalar architectures and pipelining resulting in guaranteed complexity explosion. Yet, as far as we know, <i>there is no way around this</i>. You need pipelining and superscalar execution to get adequate instruction-level parallelism in real-world code. Unless you want to use microcontrollers for everything, that complexity must exist.<p>Compilers are another example. Many people who go to implement compilers read the Dragon Book and think that all the complexity in GCC and LLVM is needless bloat. Then they quickly discover that they can't compete with them in performance.<p>It is of course desirable to avoid complexity where possible. But all too often the response that we as engineers have to discovering that difficult problems require complex solutions is to become defensive, stick our head in the sand, and stand by our simple "solutions". This is how we ended up with crufty Unix APIs, the security problems of C and C++, the pain of shell scripts, and so forth. We need to learn how to accept when complexity is necessary and focus on <i>managing</i> that complexity.
It's interesting to me that Parkinson's Law wasn't originally conceived of in the context of software projects, yet that's the only place I hear it applied today.<p>I've been working outside of software recently, and noticed to my delight that this Law hasn't applied at all. In one case, I was hired for 3.5 days of work, and we got finished after 2.5 days so we were sent home early -- nobody was dragging their feet to make it last 3.5 days. In other cases (more common), we've temporarily had too many people for the job at hand, so the team lead said "Just wait", and we do nothing until there's more work ready for us.<p>Why have I never heard of any software team ever saying "There's nothing for you to do right now, so just wait"? My first thought was the endless supply of bugs, but that can't be right, because I've never heard of a team lead saying "We have no work for you today so go fix bugs for a while", either.<p>It really does seem like every software team manager thinks that the proper amount of complexity in a system is perennially $(current_complexity + 1). The cases where program complexity approaches a constant asymptote (like Redis and perhaps SQLite) are so rare as to be notable. They're also frequently mentioned as being developer favorites.<p>Maybe the field just needs another 50 years to mature.
People definitely fetishize complexity. I often mention that I like an intellectual challenge at interviews, and the interviewers often mention 'oh, but this project is <i>very</i> complex' like they are saying something naughty. (Once, the interviewer followed up with "we're using <i>algorithms</i>). I suspect they are usually right, and the codebase is a pile of unnecessary design patterns (because those are what educated developers use, right?).<p>Where I currently work we're using Azure to do a shitload of computations. At the same time, many modules don't even bother to throw away intermediate calculations. They literally have a <i>giant</i> array of them, and save every result, for each step, even if it's not necessary.<p>But hey, the project is seen as a huge success, because it's <i>so complex</i>.
When talking about software complexity I'm always reminded of the old quote by German general Kurt von Hammerstein-Equord but attributed to various military leaders:<p><pre><code> I divide my officers into four groups. There are clever, diligent,
stupid, and lazy officers. Usually two characteristics are combined. Some
are clever and diligent – their place is the General Staff. The next lot
are stupid and lazy – they make up 90 percent of every army and are
suited to routine duties. Anyone who is both clever and lazy is qualified
for the highest leadership duties, because he possesses the intellectual
clarity and the composure necessary for difficult decisions. One must
beware of anyone who is stupid and diligent – he must not be entrusted
with any responsibility because he will always cause only mischief.
</code></pre>
Clever and diligent developers devise complex solutions to complex problems, which may often be good enough.<p>Stupid and lazy developers can be entrusted to come up with simple solutions to simple problems.<p>Clever and lazy developers are able to find simple solutions to complex problems, a very desirable trait.<p>But stupid and diligent developers, given the chance, <i>manage to implement complex solutions to simple problems!</i>
I think it could be that rather than worshiping complexity some people find it hard to value or respect simple things, and that that leads them on a path to "accidental" complexity. Like when someone find out that a rainbow is just water refracting light, or the sun is just a big ball of gas... that it takes some of the coolness out of it. There could be a tendency like that to find simple systems disappointing.<p>Or it could just be that people have a certain mental budget for maximum complexity, and they'll try and make sure they spend it all in the belief that it will cover more uses.<p>Or it could be that complex systems tend to stick around because they are far more immune to random management changes because everyone goes "oooh, we'd better not touch that". Simple systems might becomes the victim of their own ease and get subsumed by a more complex monster.
We also worship simplicity. I like to think of things as quadrants. Imagine a 2x2 matrix with columns and rows titled as such: [simple, complex] [easy, complicated].<p>They are quite not the same since one dimension denotes the complexity of the thing whereas the other denotes the complexity of the act of building, maintaining and evolving the thing in question.<p>We should always strive for easiness, however complexity shouldn't always be avoided. Quite to the contrary, staying away from complexity locally often leads to that complexity being sprayed on a global level, and that's when it turns into something complicated.<p>I have too often seen "complex" code, i.e. code that works on "complicated" datastructures such as trees and graphs be discarded in favor of solving the problem "simply" and directly, which means by disseminating the problem's logic accross the codebase and with multiple, gradual bugfixes because the problem being intrasequely complex is underspecified and cannot be tested in isolation of the system.<p>Of course the same people that are baffled by "complex code" and think simplicity sums up to looking away and not anticipating future needs, are the same who advocate it. Actually, they like good design and typography, focus more on indentation than datastructures and generally have a taste for nitpicking with as many subtle details as they can come up with, not seeing past the filter of their own opinion about what simplicity and complexity entail, and of course that makes the social process of building code a slow nightmare that does not converge.<p>I sound harsh I know. What I want to point out here is that by denigrating complexity in favor of simplicy, we may get rid of medieval savants, but we open the door to plain idiots disguised as zen masters.
Engineers are supposed to simplify complexity, taking something complex and making it simple.<p>There are many engineers though that like to over-complicate and add complexity because it makes them look smart or they think they are expected to create complexity.<p>Complexity through simple parts is ok, but overall the job of an engineer is to take complexity and break it down to simplicity and simple parts.<p>In the game dev world for instance, Unreal used to be needlessly complex, still is a bit, then Unity came along and made things simple, so Unreal then looked to make things simple.<p>Or in the web dev world, a framework might abstract away the underlying standards and add complexity on top to seem simple, but actually make the domain more complex with more to learn and push for developer lock in to the framework over basic standards and simplicity. The first version of .NET with WebForms was an example of this, other web frameworks can be seen as this as well.<p>Lots of engineering people like to look smart by managing complexity but it doesn't always need to be so complex. Sometimes time pressure can create complexity and thus technical debt due to the complexity. Some simplification is misguided, a one liner that isn't understood 6 months later is not simplifying.<p>An engineer that takes something simple and makes it more complex for no reason other than job security or to look smart is the worst kind of engineer.
One word: "Plastics", no, actually it's "Job Security". There is an incentive in ANY profession to recommend or encourage more of your own labor. For example, a surgeon is more likely to recommend surgery to solve a problem than a non-surgeon.<p>This bias isn't necessarily intentional: human nature (competitive evolution) just naturally pushes us to interpret the world in a way that makes ourselves as valuable as possible.<p>In IT, complex solutions that we create or learn keep out competition. "Only Bob knows how to fix this monstrosity" is a common pattern.<p>Simplicity should be added to the employee evaluation process. This includes parsimony in both features (YAGNI) and in how the features are coded.<p>Further, avoid "eye candy" UI gimmicks that add complexity and fragility. End-users often love them, but they are often a longer-term maintenance headache. Beauty ain't free.
I was with him until the very end, where he said that the cloud, microservices, etc. were the simpler approach, that other companies (not Google or Amazon) are trying to avoid.<p>What I see is the opposite; companies that would do perfectly well with a server or three, and a simple monolith, trying to instead use Kubernetes and an Amazon service salad because that's what Amazon and Google say to do.
I think a massively underutilized core strength of developers is that we have a working thought process in front of us in the form of code, which can be used for more than just running a program.<p>In my experience, complexity in the code often reveals not only technical problems, but frequently also points to product and business issues. Exploding complexity and long iteration cycles are often a consequence of bad business decisions. Looking at the points of exploding complexity in the code can sometimes help identify these issues.<p>The key is to make the disciplines work together. Business and product decisions should not trickle "down" to developers. Instead, there should be a working feedback loop. For that to be achieved people need to be team players and talk to each other frequently. It also helps to have a 10% or so generalist in each team member's head to guarantee a shared understanding of a high level perspective.<p>I know, I know — this is all a given in agile methodologies. But in reality, it's unfortunately rarely executed that way.
One only has to look at any Node.JS codebase for the answer to your question. On the Java side, the Spring Guys love complexity as well: <a href="https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/servlet/support/AbstractAnnotationConfigDispatcherServletInitializer.html" rel="nofollow">https://docs.spring.io/spring/docs/current/javadoc-api/org/s...</a>
Everything is always at the edge of maximal complexity. If it were any more complex it would collapse under its own weight. If it were any simpler someone would say "it would be so easy to add..."
Simplification of systems should be a an actual college course, mandatory for all computer science grads.<p>Too many people these days <i>manage</i> complexity through ever more complex sets of tools instead of simply getting rid of it. They use complexity as an excuse to introduce more complexity and it just keeps growing.<p>One important thing I discovered for myself is the practice of starting with <i>minimum viable representation</i> of the problem. If I work on a system, I try to write down what it needs to keep track of. If I work on a component, I start with the minimal configuration it requires to instantiate. This helps to avoid anchoring myself with available tools and "common" solutions.
There is a kind of complexity worship I've noticed, but my accounting of it is that it comes from folks who aren't aware of the methods of simplification employed by those successfully wrangling complex subjects/systems; absent an alternate explanation, they assume that the mind of the complexity wrangler is fundamentally different from their own.<p>In other words, rather than imagining the complexity wrangler inventing layers of simplifying abstractions, they imagine him/her to just have a radically extended short-term memory or something on those lines.<p>The article points out the case of architects worshipping complexity in some cases by essentially over-engineering. I think this is a mistaken explanation. I think it's more of a fear-driven thing: the architect is like, "Oh shit, this problem I'm gonna be dealing with is gonna be <i>really</i> complicated; I'd better throw everything I've got at it!"
Primarily, people worship their own positive self-image. Complexity gives them the chance to view themselves as smart enough to handle this really complicated thing.
<i>After all, we all are looking for technical challenges and want to implement interesting projects.</i><p>In my experience everyone does, but the difference between the complexity-worshippers and (for lack of a better term) simplicity-worshippers is that the former like to take a simple problem and blow it up with complexity, while the latter like to take a complex problem and make it simple. In other words, some people like the challenge of making things more complex, while others like the challenge of making things simpler.<p>To make a concrete example, complexity-worship would be something like using half a dozen different new web frameworks/languages/libraries to set up a personal blog, while simplicity-worship would be more like an H.264 encoder in a single file[1] or a self-compiling C-subset JIT[2].<p>One thing that seems somewhat obvious about the complexity-simplicity divide, and which could also explain the prevalance of complexity-worship, is that it is relatively speaking much easier to generate complexity than to reduce it, and unfortunately it seems the majority of developers just don't have the skill to reduce complexity. It's easy to glue a bunch of existing code together without really understanding how it all works; it's much harder to take a spec that dozens of experts have worked on for many years, and condense it into a concise implementation.<p>[1] <a href="https://news.ycombinator.com/item?id=18045494" rel="nofollow">https://news.ycombinator.com/item?id=18045494</a><p>[2] <a href="https://news.ycombinator.com/item?id=8558822" rel="nofollow">https://news.ycombinator.com/item?id=8558822</a>
> The managers, that this paper describe, worship complexity without realizing it. They want as large a team as possible and thereby they make a problem complex because a large organization can cause the architecture to collapse.<p>Maybe we have different definitions of "worship", but this seems like accidental complexity, not reverence. If anything, people tend to worship simplicity.
Conway's insight, and the subsequent law that bears his name, was so laser-precise that it is uncanny.<p>Local incentives matter too. In any social system there's an incentive to "flexing your muscles" and showing off our abilities, since that helps to build prestige and respect.<p>When coding moves away from "impact", and "scaling" and "agility" and back to old fuddy-duddy engineering staples like better up-front design, partitioning problems properly and solving them in toto before shipping (i.e. doing it right the first time), allocating the right problems to the right people--difficult to do in a team with junior members that need space to grow and experts that can dash things out in minutes--then we'll hopefully be sober enough to see through cargo cult practices down to what works.
There is "complex", and then there is "complicated", I believe the author refers to the latter. A complex solution involves a lot of moving parts, but each of these parts serves a specific purpose, such that it is possible to break down the solution into relatively simple, understandable blocks. On the other hand a complicated solution will have inefficiencies, some of its building blocks may be too big or convoluted to be broken down further, causing implementations that are hard to understand and/or may be redundant, feeding back into the complications. I think Rube Goldberg's Inventions are an excellent example of the difference I'm trying to make.<p>Social posturing is certainly largely to blame for over-complications, but I think more deeply, a lot of people hate to let go, to be told that the very thing they were responsible for, that they worked hours/days/... on isn't necessary any more, or perhaps never was. And then in the other direction, there are what I refer to as "exponential requirements", for lack of a better term. You buy a large bag of coffee that you could simply use as you go and store in the fridge. Or, you could have a jar with some of the coffee in it, and the rest of the bag still in the fridge. Then a plate to put the jar on. Then a new shelf to put those plates and those jars. By the time you've followed this track long enough, you're looking at buying a new house, even though your essential requirements haven't changed from the beginning.
The ability to recognize and manage complexity is surely a strong survival trait, akin to wisdom. As true for hunting and gathering as for particle research. Complexity is also somehow deeply related to beauty, or at least awe. Such emotions motivate us to become more observant of the complex, possibly therefore out-replicating the more oblivious.<p>Worship of the complex, or at least a particular focus on it, may be built in by Chuck Darwin.
The fetishism of complexity in engineering (and our machine loving culture) is much like the fetishism of helplessness in some of our religions. Both intensely value surrender to an unknowable omnipotent power. A power addressable only by blind ritual.<p>You might say that in both cases there is a longing for a dissolution of self. Or death even.<p>Is the machine-lover a death-lover?
when i was younger I used to think complexity was some sort of example of ignorance. I work as an automotive engine mechanic, so my gold standard for "its easy and works" was stuff like old fox body mustangs and such.<p>Fast forward to 2018 and stuff like timing chains are buried under a mountain of engineering sins. I have a honda with the oil pan, air conditioner, exhaust, and oil drained and in various states of disassembly and for what? a chain.<p>As i get older, I start to worship it. its inevitable that engines come with a spinal cord of three or four dozen sensors you need to carefully disconnect before doing anything. But what I cant abide by is the inclusion of demonstrably poor quality parts. Companies hope the engineering complexity will just baffle people into accepting the idea of disposable plastic valve covers and plastic water pumps, but I remember when these things didnt get thrown out.
The intro on Conway's law was more fun to read than I expected going into this.<p>It is interesting, because I've seen Conway's law used as a weapon against modularizing something. I think the key is that you need to weigh what you are getting out of the modules. Even if it is, in some ways, more complicated, if you can more meaningfully delegate ownership of parts of the problem out because of the the modularization, it is a good chance it is still worth doing.<p>I don't think I can give a brief example. In my case, we had three teams, rather than trying to put everything on one team, I was trying to ask for what the three major components of the system were. If we couldn't agree on that, it seemed unlikely we were using three teams well.
The "Simplicity: Not Just For Beginners" talk does a great work of touching the problem from the opposite angle : <a href="https://www.youtube.com/watch?v=n0Ak6xtVXno" rel="nofollow">https://www.youtube.com/watch?v=n0Ak6xtVXno</a>
Surely at issue is that we do not know how to generally handle complex problems well. Particularly those <i>outside</i> the domain of computing.<p>The old fallback of functional decomposition into parts that can be reasoned about and developed independently, is problematic due to the need to constantly rework and extend the hierarchical decomposition as more about the domain under consideration is specified.<p>How to predictably build a payroll system, or air-traffic control system or a much simpler (at first glance) yet inevitably complex real-world system, remains out of reach.<p>Surely we don't worship complexity, but outside of computer science, just still don't know how to deal with it?
The more complex a system is, the more people it takes to maintain, write, test for it. It builds the manager's empire and also increases mindshare for the complexity-driven approach. I've seen it many times.
Any serious programmer simply can not worship complexity because then his/her work would never get done.<p>You can start with a trivial problem and grow it into a system so complex it's hard to even begin grasping what it actually does and how the software actually works.<p>If you start with a complex problem the only viable path is down to less complexity; often we won't reach simplicity but at least we're somewhat equipped to handle the underlying complications.
How many times have you stumbled upon something on Github that took 100x more lines of code and 20x more contributors than necessary? Probably every day.
Another law of complexity: the Law of Requisite Variety: If a system is to be stable, the number of states of its control mechanism must be greater than or equal to the number of states in the system being controlled.<p><a href="https://en.wikipedia.org/wiki/Variety_(cybernetics)#Law_of_Requisite_Variety" rel="nofollow">https://en.wikipedia.org/wiki/Variety_(cybernetics)#Law_of_R...</a>
I very much doubt this originated from me but whether I heard this somewhere and took a liking to it or thought of it myself, I became fond of saying this while I was at Google:<p>"How do you know if you're engineering if you're not overengineering?"<p>Now I don't direct this at Google specifically. This is more about the mental traps we, as engineers, can easily fall prey to.<p>Take interviewing and the FizzBuzz thing that was popularized (if it didn't originate with) Joel Spolsky. The beauty of FizzBuzz is that it's a quick and great _negative_ signal. This doesn't mean that if you ace it, you're a superb engineer but it does mean if you fail it, you're almost certainly not. Hiring (from the employer's perspective) is a numbers game. Every candidate costs you time (giving interviews, writing feedback, etc) and is a huge opportunity cost (other work that could be done, other candidates that could've been interviewed) so the goal is the filter out the "no"s as quickly as possible.<p>So the engineer trap when faced with something like FizzBuzz is to think "oh wait, that's too simple!" and they go on to change the problem to something you'll pass if you've heard of the particular obscure algorithm and will probably fail if you haven't. It's a easy fallacy to fall for. Harder is better. However now you're not optimizing to cull early candidates. Now you've designed a test that optimizes for people who do well coding under pressure (with some degree of luck) on a whiteboard. That is completely different from the original intent and (IMHO) almost entirely useless as a hiring signal.<p>Another example: another thing I was fond of saying at Google was that there was a hierarchy of engineers:<p>- At the top level (in their minds) were the C++ engineers. You're not engineering if you're not writing code in C++ (basically)<p>- Next tier were the Java engineers. They thought you weren't engineering unless you were writing in Java or C++<p>- Next came Python and Go<p>- Last came Javascript<p>Now at Google I met some engineer who were _superb_ C++ engineers. Like truly amazing. I also met others who were like walking Wikiepedias for the C++11/14/17 standards. And no, that's not the same thing.<p>So one example that springs to mind once was surprised to learn that I didn't know what perfect forwarding was. This is something that probably only needs concern you if you're writing a widely used C++ library using templates (especially template metaprogramming). As a user of such libraries, it's not often something you need to know.<p>My point here is that with all the knots C++ has twisted itself into over the years as a result of its origin and legacy, the trap some fall into is viewing such complexity as a virtue instead of baggage.<p>IME these aren't the people you want making design decisions on complex systems. The people you want making design decisions on complex systems are the ones who reverently follow Postel's Law.
people are definitely suckers for complexity.<p>Complexity of a system actually creates meaning for people that do understand it and can navigate it.<p>Complexity also justifies headcount and effort put towards maintaining, enhancing something.<p>Finally, people associate complexity with being smart. If you like simple things you are either a newb or you are [very] senior/old/experienced.