"Software Engineering is Dead" is obviously an overly sensational title, but let's look for the deeper truths.<p>First a little background. I have built a career developing business applications as a contractor, often in very large organizations. I am almost always very successful and I'm often regarded as a hero doing what most people here would call "just doing my job".<p>Why is it so easy to be a hero in the enterprise with software that would just seem ordinary in a forum like hn? Is it because enterprise programmers aren't as good as us? Is it because their managers are all phbs? Or because they don't know how to run projects?<p>I'd say no to all of the above. Enterprises are filled with lots of excellent people doing great work.<p>I think that the real problem is that the Systems Development Life Cycle (SDLC) that so many depend on so much <i>never really did work</i>. Why?<p>Every phase depends upon Phase I, Analysis to be rigorously done. This rarely happens for 2 reasons: users often don't know what they want and most systems analysts don't know how to extract it even if they did.<p>So almost everything done after Phase I is built upon a foundation of sand. It's either wrong or sinking fast. And what do most people do? Everything <i>except</i> fixing the problem: more resources, more project management, freezing specs (which aren't right in the first place), more rigorous deadlines, etc.<p>But rarely does anyone attack the core problem with the SDLC: defining the expected result.<p>So what should we really do? Develop something, anything, quickly, cheaply, and get it out to the right users. They will instantly give you feedback. What's right, what's wrong, what's stupid, all the cool stuff that no one thought of.<p>No one can just sit down and write a Functional Specification for a large business application. And even if they could, you don't want them spending time on it. Better to get the right people together and find out what they need. Usually, no one of them knows what the result should be, but all together, any decent developer should be able to extract enough data to write version 1.0 of <i>something</i>.<p>It's a lot easier to judge something that exists than define something that doesn't.<p>The larger the organization, the more difficult it is to change their ways.<p>Software engineering isn't dead. It's just that the process of depending upon blueprints before you get started never worked in the first place.
I came to the same conclusion in my first job out of college, working in IS/IT at a large automotive firm.<p>Now, I'm not so sure. I think there is a large craft component in the implementation of software, but there is also an engineering component in the selection of algorithms and datastructures, as well as testing and benchmarking. I think what I'm saying is that the construction of software is a craft, but the design of software (should) still have a large engineering component.
I'd prefer the software on the plane I'm flying on to be engineered. Arts and crafts are fine for your new facebook app, but please engineer the stuff that keeps me alive.
The lack of mathematical rigour is what kills it. The software industry is the one where we do not learn from past mistakes or successes, we rarely measure important things or when we do benchmark, it's to focus on stupid things like the overall performance of specific language implementations or how much RAM Firefox is using instead of figuring out how to eliminate whole classes of problems.<p>We can be artists and creative people just as mathematicians and physicists are.<p>I'm reading about John Nash and holy shit, Princeton gathered some of the smartest people together and told them not to worry about grades, just worry about research. Nash himself would just wander around the halls and just <i>think</i>. The papers written may seem formal but the way the ideas were generated was highly informal, with random meetings and discussions in hallways and common rooms and walking about lost in thought.<p>What struck me as most interesting and applicable to software development is that Von Neumann and the others at RAND were applying somewhat pure/abstract mathematics to specific problems in other fields. So why don't we take a page from them and apply some more rigour to measurement, and why don't we learn some management skills and develop useful metrics for software development, so that we can stop letting the marketing people and the snakeoil consultants from monopolizing our field and turning it into a wasteland of overly expensive unfinished projects.
Much of programming is working with complexity, not engineering.<p>If you know from beginning to end how to do something, then you can engineer. You can form processes to govern the engineering, you can compute some things to be just right, you can make some nice tradeoffs and eventually you're finished. You could build a bridge like that. You could build a GUI like that. Or the 10th website of similar functionality. Or the 100th half-assed C-library replacement because your project can't or doesn't want to depend on the standard C lib on all platforms.<p>Most importantly, basically, much of what could be engineered often gets automated. Programming always happens on the edge of unmanageable complexity and anything less than that gets quickly automated. That leaves very little to be engineered. Basically you engineer something if you insist on not being lazy and doing it by hand.<p>Throw in a "minor" scalability requirement, or a guaranteed response latency of less than X milliseconds, or some "easy" dynamic "addition", or some "basic" interoperability mechanism to another program, and suddenly your engineerable bridge must be designed to support a million tonne UFO the size of Manhattan that keeps gleaming in the temperatures of several thousand Kelvin.<p>To conclude: Software engineering does happen but only after the software has been written enough many times so that we actually have experience in doing it, and often for only a short while until the engineering part can be automated and programmers released to work on something more complex.<p>Software is so tricky that you don't want to redo it too many times unless you really, really have to.
I like the original article by DeMarco which Atwood discusses. DeMarco makes a good point that normal engineering methodology doesn't work with software. Anyone who has tried to code up something based upon an 80-page spec document written by a junior analyst who knew neither the business nor how to code might agree. Writing code is more like writing an engineering plan than it is trying to follow one. DeMarco writes about the dangers of using the wrong metrics but he doesn't offer much of a counter-proposal - something like "fail early and fail often?"
I've seen a lot of teams, and a lot of projects either fail or succeed.<p>It boils down to three things: People are the most important, over-generalizations about the development process is anathema, and always adapt.<p>The over-generalization part is especially rampant. Bloggers or authors want to make some sweeping statement about what software engineering is. Team members that worked on one successful project want to repeat that exact experience on another project. Project Managers who measured one thing on a good project become convinced that measuring that thing was part of what made the project successful.<p>Yes, there is engineering at work in software development. We load test, we plan scalability, we use set theory to model data and relationships. Debugging is about hypothesis formation and testing. There's all kinds of hard empirical science going on.<p>At the same time, the end result, the goal, is a nebulous thing that appears different ways in everybody's head. Large parts of this process give us the freedom to creatively construct things.<p>People who forget the engineering part fail projects due to bad design. People who forget the creative part fail projects due to poor user acceptance.<p>But highly-trained and motivated adaptive teams do just fine no matter what the environment.<p>The ultimate truth is: there is no ultimate truth.
For myself I agree, but I was never big into engineering approaches. Very suspicious of agile methods, too. The way I see it: for example in mathematics the first proof of a difficult theorem is often very long and complex. Over time more mathematicians might look at the proof and find ways to make it simpler. Eventually you might have a 5 line version of a proof that used to be 30 pages long.<p>I don't see how any agile method or engineering approach could help much in getting the 5 line version from the beginning. There always seems to be a kind of "crunch" step involved in solving the problem. Once the problem is solved, the solution can be cleaned up. But solving the problem at first you explore maybe hundreds of possible paths in your mind, and combine lots and lots of aspects. You are stuck in the jungle and try to find a path to the summit. Once you are on the summit, you can look down the mountain and see the easier routes you missed on your way up - because in the jungle you can't see very far, and you have to fight for every meter of progress.<p>I find the same applies to my own code - which is always crappy, anyway. But refactoring usually goes quite fast once the first working version has come into existence.<p>I'd expect having to focus on getting some agile method right would just distract me from solving the problem.