My 20-Year Experience of Software Development Methodologies

There is an insider story about how these methodologies comes about. So there are few groups of people whose sole job is to do consulting on failed&#x2F;late&#x2F;over budget projects. Mind you, they <i>don&#x27;t</i> write code but rather they observe how things are going and then prescribe process&#x2F;management improvements (McKinsey style). Once in a while, these folks bump in to terrible projects and whatever they prescribed sometime works like a charm. In that case, they take that prescription on road and advertise the hell out in conferences, magazines, blog posts. Unlike regular developers, they have all the time in the world to do these activities. They write books and give interviews and by the media power of distributing information suddenly they pop out as process gods who knows how to fix any project. Eventually the new things starts to fad, people realize what works in project X didn&#x27;t worked in Y, speaker engagements starts drying out and then these folks need new thing to repeat the cycle.<p>The obvious problem is that these folks prescribing the development process are not active developers. They are not even part of any real project over any long duration. They are in job of inventing and selling processes and handing out management advice as consultants. Whatever they prescribe, might have worked only in specific context and for specific symptoms, usually with huge dose of luck. Next time when you see new process fad, look up the history of originator of this process, what company he is part of, how much code has he written, how he makes money. You will know what I&#x27;m talking about.

Really liked this article.<p>I liken software methodologies to cooking recipes.<p>On one end of the scale you have a highly skilled and experienced chef, who can craft a great (requested) dish with a variety of ingredients and equipment. Knowing how to adapt and what to add and when. This works well, but requires a heavy cost (experience) and is, from the outside, difficult to follow and control.<p>On the other end of the scale you have fast food. A strict process and fixed ingredients that ‘just works’. Those involved in cooking have very little knowledge of why, and are almost certainly not able to adapt. This works well for scenarios where you exactly know and control both the input and output.<p>A skilled chef can ‘codify’ their recipes, but this results in a specific description of what to do, and not why. Thus marginalising the adaptability.<p>In my experience, most software engineers are trying to attain the skilled chef role. While most _managers_ want the fast-food style predictability. With the ‘process’ being the battleground.<p>The best process is where you - a complete multi-disciplinary team - make as quick iterations as possible, learning as you go. But, always knowing where you are trying to get to (and why!) and constantly adjusting accordingly.<p>Good luck out there.

Methodologies are a trap. The problem with any methodology is they tend to lead people to stop thinking about the context they are operating in.<p>&quot;The methodology says X, we should do X, why oh why aren&#x27;t we doing X?&quot; is a familiar lament in the development world. I have been there and done that, to be sure.<p>It really isn&#x27;t going to matter what the Methodology says, if something or someone in the context means it isn&#x27;t being adhered to.<p>If the person ultimately paying for the project isn&#x27;t convinced by the Methodology, or wants to deviate from it badly enough, guess what? The Methodology will be ignored.<p>You should always advocate for what the Right Thing To Do(tm) is, but you should ask yourself first: what is the most important criteria that determines what the Right Thing is?<p><i>Sometimes</i> writing the quickest, hackiest, throwaway code can be the Right Thing. It can be a horrifying truth for a software developer. Sometimes doing the ballsy re-write is the Right Thing. Sometimes letting a project fail is the Right Thing.<p>Sometimes last weeks context and decisions no longer apply.<p>More critical to project (and personal) success than any Methodology is gaining a thorough understanding of the context in which you are working in, as quickly as you can. And then doing all the Right Things that will make you effective within those constraints.<p>This means there is no Single Truth for software development. No easy answers. But there are a wide variety of principles you can draw from and apply and <i>discard</i> as needed.<p>Or as Bruce Lee put it: Be like water.

I&#x27;ve worked in a variety of environments, and the most important thing in my experience is simply to ensure that your team is creating a working, tested, and &quot;shippable&quot; update every ~2 weeks. It amazes me how many teams fail to do this, despite calling themselves &quot;agile&quot; and &quot;customer driven&quot;.<p>Use whatever methodology, technology, and processes you need to accomplish that singular task. Orient your entire organisation around doing that. Your customers don&#x27;t care how you do it. They just want frequent updates and progress. And if you&#x27;re not giving them useful updates (you&#x27;ll know, they&#x27;ll tell you) then the problem isn&#x27;t methodology, it&#x27;s probably your ideas. No methodology will fix that.

Nice post! I worked in a few big and small companies, and from I can see software companies with more then ~40-50 devs don&#x27;t scale well (I only talk about about single-ish product software companies here). The reason for this is in my opinion is absence of financial liability of devs: if you&#x27;re a dev in a company with 500 other devs - most likely you have zero financial liability to produce great code, on other hand you have all time in the world to create job security code around you. Also if there is 500 devs on one project, most likely one will see a lot of infrastructure just to get this devs busy with something. This could be seen in companies were one department creates &quot;the framework&quot;, and others departments create &quot;features&quot; based on this framework. In this case there is no &quot;market&quot; of frameworks - you basically stuck with it, and &quot;the framework&quot; has zero market competition. Compare it to normal market of middleware - if your framework is bad, I&#x27;ll just go to your competitor. Free market is much healthier situation in my opinion (could be a strawman argument though). I am not wise enough to say that creating internal competition inside one company is always good thing, though financial contracts make a healthier political discussions.

I’m surprised that Don Reinertsen’s book The Principles of Product Development Flow doesn’t come up in these discussions. Rich Hickey mentioned it once after a conference years ago and that’s the only time I’ve heard it mentioned. I’ve since found a few blog posts responding to conference talks (of which there are several on YouTube from around 2015), but not many. I think people simply don’t know about it. I wrote it down at the time and finally digested it over the past several months. It’s an incredible book that I think would resonate with many of the commenters here.<p>I think of Reinertsen’s approach as industrial agile, in the most honorable sense of industrial. There is theory and some math. There is heavy emphasis on project economics as a means of making complex trade-offs and a basis for training, deputizing, and expecting employees at all levels of an organization to make better decisions on the fly as part of doing their jobs. Lessons are drawn from many fields, such as network communications, queueing theory, and the Marines. Contrasts are drawn with manufacturing to explain exactly why some of the techniques we adopt do not apply in product development but where many techniques from kanban systems do apply.<p>The book is organized around hundreds of principles given as tools, to be considered and applied as needed to your specific situation. Annoyingly, he refuses to get into implementation. At first I thought he was saving pages, but the more I think about I realize the seeming omission was necessary genius. Digging into even one example implementation would result in the death of the message and framework as people copied that and get dogmatic about a methodology all over again.<p>The whole presentation is beautifully constructed and <i>tight</i>. It puts a perspective and language on things in a way that makes me hopeful and interested in project management, similar to how Rich Hickey’s talks raise my understanding, vocabulary, and aspirations for software engineering.<p>I would welcome anyone involved in prioritizing commercial software projects to read and discuss this book (and document their implementations).

My 30 years of Software Development Methodology:<p>* The user is more important than any of this. * The programmer is nothing. * The computer is doing all the work.<p>Thus: use what makes all the above, True.

The biggest skill when interacting with a &quot;big serious man&quot; type of client is to making him pay you for what you want him to pay you for.<p>In my circle, we have a lot of discussions on clients like that: &quot;and I told him, for what freaking thing do you need that freaking neural network when a banal statistics does the job better?&quot;. Having such guy, &quot;a Wharton grad with stellar track record at the big 4&quot; agreeing on the former requires compromising his ego, self image, exploiting him wanting to not to look incompetent in eyes of his superiors.<p>In the end, all such guys are made to pay 6 or even 7 digit sums for a fancy UI over database querying tools and a stats package, and not a &quot;big data, machine learning blah blah blah,&quot; or otherwise being given &quot;the exact thing they wished for&quot; that had no chance of even minimal functioning

The fiction I like best right now is &quot;continuous delivery&quot;: You release often, push to production several times per day, one working branch, always ready to deploy. Features being in development are behind flags. Regression tests help a lot, eg, when you make an update or fix a bug you make an automatic test that checks if the new thing works, or repeats the steps that caused the bug and makes sure it&#x27;s no longer there.

We use &#x27;waterfall&#x27; (detailed, fleshed out specs in formal language) on the hardware, firmware &amp; backend and &#x27;agile&#x27; on the frontend (web&#x2F;app). So far that served us well and it addresses the difference in developer skills <i>and</i> needs as well. With the danger of generalizing; frontend people are different from backend people who are different from hardware people (in my experience), and this gives all what they enjoy working with.

Author here: interested in any strong views on this - @ianmiell on twitter.

All these methodologies are there just to get things done. After many years involved in software development i have find out that it all comes down to leadership. Of course you need people with skills, but no good leader, no good product.

I like the car analogy, and I had additions to it with software-components in mind:<p>engine-choices representing data storage: very big engines. for big ships. very small ones, for lawnmower. self-made ones, smoking.<p>gear-choices representing libraries: single gear gear from 1 to 1000. No neutral and no reverse.<p>chassis choices (language choice): rusty old ones, hard to work with, but lasts. new shiny ones, overpriced and falls apart.<p>locks = safety: one key fits all (all doors and all cars). different key for each lock, can be bypassed by opening window. keypad with password, written on the back-side of the car.<p>interior = user interface: millions of choices, all flashy and shiny, each part looks totally different. Unusable steering.<p>break (software stability): no breaks. when used causes blue-screen<p>outside (program output): all parts have different sizes. holes covered with duck-tape and rust covered with shiny paint.<p>Enjoy the ride!

I really enjoyed reading this and I strongly agree there isn&#x27;t an easy task at all. Depends on the business, people, existing processes, it&#x27;s up to us to find what really works. I had some thoughts based on my recent experience and went into a mood to write something quite long on my LinkedIn <a href="https:&#x2F;&#x2F;www.linkedin.com&#x2F;pulse&#x2F;waterfall-scrum-kanban-scumban-whats-next-richard-shenghua-he" rel="nofollow">https:&#x2F;&#x2F;www.linkedin.com&#x2F;pulse&#x2F;waterfall-scrum-kanban-scumba...</a> if anyone interested.

I&#x27;ve not found a methodology yet that can displace smart decisions.<p>Methodologies organize projects to help further smart decisions and avoid dumb ones, but smart decisions and skill still dictate the success or failure of projects.<p>In the end, the people are still more important than the methodology, but I&#x27;ve encountered a mentality a few times where people believed a good methodology will fix all.<p>I&#x27;ve seen companies keep changing methodologies looking for success with largely the same staff, and key decision makers, but yet the results are the same. Insanity, anyone?

The style he describes as RAD is really enjoyable to work in. No constant scheduled meetings and unread documents; just use your common sense and write something.

There are no need for formal methodologies. A person with no technical expertise should not be but on a position managing technical teams. That requirement divides the industry on &quot;tech companies&quot; and everybody else.<p>The example of a tech company with gigantic dev teams and very little number of MBA doctrinaires involved in development process is Intel.<p>Companies with external development clients that practice &quot;we will agree on any methodology you want for right amount of money, but we would not let you anywhere close to our developers (who will be doing their own thing anyways)&quot; are Tatas, Wipros, Luxofts and others. Their biggest asset is their skill at managing the client.

I work for one of those fancy startups. Everything is chaos all the time, you have all these extremely smart people in the small doing actually kinda dumb things in the large. This worked better when the company was small and is causing more and more problems as we grow. Anyway, the point being that there’s lots of smart individuals and little process.<p>I mentioned this during a meeting with my team, and so a process was devised for the next project we embarked upon. The process was waterfall, except nobody referred to it that way (most of the people are like 25, and weren’t working back when it was the norm). Everything old is new again.<p>It wasn’t important anyway; we didn’t follow the process.