Software Effort Estimation Considered Harmful

There are two serious problems with this post, and it really saddens me that I see these sorts of posts so frequently here, with so many concurring voices.<p>First of all, cost absolutely 100% has to factor into prioritization decisions. That doesn't require absolute estimation, but it does demand relative estimation (which he mentions tangentially at the end of the post). If Feature A will deliver $100,000 in revenue but take 18 months and Feature B will deliver $10,000 in revenue but take 1 week, the choice is pretty obvious. What matters is never "return" but always "return on <i>investment</i>." If you don't know anything about the I side of the ROI equation, you're doomed to make bad decisions. With no estimate at all, and just a snarky "it'll take as long as it takes, shut up and let me work" response, you'll inevitably focus on the wrong things.<p>Secondly, many of us do in fact have deadlines, and they're not just total BS. If you have a customer that you're building something for, they have to be able to plan their own schedule, and just telling them "well, maybe we'll ship it to you in 10/2012 or maybe in 6/2013, you'll get it when you get it" doesn't fly. And it's totally reasonable that it doesn't fly: if they have to, say, install the software, or buy hardware to run it on, or train users, or migrate data from an old system, or roll out new products of their own that are dependent on the new system, they clearly can't plan or budget those activities if they have no clue whatsoever when they'll get the new system.<p>And if you do have a deadline, you kind of want to know, to the extent that you can, if you're going to make it or not so you can take corrective action (adding people, cutting scope, pushing the date) if need be. You can't do that if you have no estimates at all.<p>Relative estimation of tasks with empirical measurement of the team's velocity works pretty well; it doesn't work in all cases, but it's pretty much always better than absolutely nothing.<p>There's a huge, huge difference between doing relative point-based estimation and date-driven, pointy-haired-boss estimation, and it's a total disservice to the software community that so many engineers seem to not really understand that difference, and seem to think that the only two options are "unrealistic date-based estimates" and "no estimates."<p>TL;DR - Don't just rant for 3000 words about how estimation is horrible and then add in one sentence about relative estimation. You'll do the world much more good if you just help educate people how to do things the right way and spare the ranting.

He's sometimes right. There are lots of good reasons why you might need software schedule estimation. When you do, there is no point in throwing up your hands and saying, "You can't do that, everybody knows it." Instead get <a href="http://www.amazon.com/Software-Estimation-Demystifying-Practices-Microsoft/dp/0735605351/ref=sr_1_1" rel="nofollow">http://www.amazon.com/Software-Estimation-Demystifying-Pract...</a> and teach yourself how to do it.<p>Why would software estimation matter? For many companies it matters a lot if you can announce something in time for Christmas. Or you might have to hit a deadline for compliance with a legal contract. Or you may be trying to prioritize two different possible products - without a sense of the effort involved you're lacking one of the parameters you need for such an estimation.<p>That doesn't mean that you always estimate. If the value proposition is clear enough, you're going to work on it until you're done and then you will be very happy. But the real world does not usually work like that.

First, continuous deployment works fine for web projects, but it's not an option if you've got a packaged product that you have to ship at some point. Agile is not a panacea for these environments.<p>It's important to differentiate between the two kinds of software that you can write. This is post is dead-on for developing truly new software (new features, stuff that fundamentally hasn't existed before, etc). However, estimation (even the kind of project-manager-driven estimation that most engineers, myself included, generally hate) can work really well for software where you're doing basically the same thing you've already done in the past such that the risk of unknown unknowns is extremely low. I've seen it in practice, and when you have a checklist of stuff to do that you've already done with very minor changes in the past that can't be automated for whatever reason (e.g., support a new piece of hardware, except that the new piece of hardware is basically the same as the old piece of hardware with very minor well-documented changes), you can estimate with surprisingly good accuracy.

I'm fascinated to know the author's opinion about Joel Spolsky's "Evidence-Based Scheduling" approach. The author mentions that breaking the schedule into the smallest possible pieces can lead to "overfitting," but Evidence-Based Scheduling uses previous programmer estimates vs. actual time to give a probabilistic schedule that changes with new events. Instead of a rigid ship date, Joel's method can give you a probability of shipping on a given date with the current information. Is Evidence-Based Scheduling still harmful, or is it basically the equivalent of the relative points-based estimation the author brought up at the end of the article?

Software effort estimation causes three problems. First, most estimation processes model the uncertainty about the effort required for tasks but fail to model the uncertainty about the tasks themselves, leading to unreliable estimates. Second, people are overly confident of the estimates, in any case, because the estimation process looks impressive and produces impressive looking artifacts. Third, the estimation artifacts obscure the coupling between the stakeholders and the implementers by not transmitting how the one group’s decisions affect the other; the estimates, in effect, form a barrier that makes it harder for people on opposites sides of the estimates to take shared responsibility for business goals.<p>If you understand these problems and can solve them for your projects, estimation can help you to make better decisions and to allocate your resources more effectively. But, in a lot of organizations, these problems have no good solutions (for cultural reasons), and there you might be better off not sharing your estimates if you do them.

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law<p>or<p>Parkinson's Law: Work expands so as to fill the time available for its completion.<p>There is always some probability that software estimation is wrong no matter how well planned out. The greater the size/scope, amount of people and the bigger the timeline is the more chance it will be late.<p>Like Valve Time proves (<a href="https://developer.valvesoftware.com/wiki/Valve_Time" rel="nofollow">https://developer.valvesoftware.com/wiki/Valve_Time</a>), sometimes product trumps time you put in and making games is hard. Sliding deadlines are the most successful because they take into account reality of changing scope/product during development. Or short task sprints/windows like week long or month long product delivery stretches with complete task freedom in between.<p>Software estimation is hard because there are so many factors and it is a constant ship vs quality balance.

The flip side of lie-based estimation is arbitrary deadlines.<p>I'm not the first to notice that work expands to fill the time allotted. There can be real value in setting a hard deadline with almost no regard for difficulty. This eliminates the 'process' overhead and often produces amazing amounts of work in a shorter period of time than anyone would have estimated. I really like working to deadlines because my motivation is inversely correlated to the time till deadline.

I think estimation can mostly be forgotten. The planning part which the author identifies is always helpful. Sure, get the team together to plan features and get a feel for the goals. Even get biz analysts together with folks to determine req's.<p>But do we really have to tell you how long it will take? When a lot of managers just take the estimate and multiply by three can these estimates really be trusted? What's the end game? Is it any different than just building it out? If you have a hard set of features then it will get done when it gets done. Just predict what half of what year. If you're an agile enterprise, then timebox the release and build out from there. Why must we all do this absurd dance of time estimating?

A good read, and more-or-less in line with my observations from working at a couple big software companies. I think there is generally less value in estimating versus doing, especially for tasks/projects that are inevitable.

"The roughness of the fractal dimension of a problem that needs to be solved can be calculated more easily in my opinion than with classical estimation techniques."<p>We're always applying the Roman "divide and conquer" strategy without thinking about it. It wouldn't make sense to apply this, or any other strategy ignorantly! The D&#38;C strategy works because a naive solution to count the fingers in this picture without knowing the fractal dimension is: "divide and conquer". <a href="http://mark.rehorst.com/Bug_Photos/fractal_hands_c.jpg" rel="nofollow">http://mark.rehorst.com/Bug_Photos/fractal_hands_c.jpg</a> (mirror: <a href="http://i.minus.com/ibz9NsZ6ET32aV.jpg" rel="nofollow">http://i.minus.com/ibz9NsZ6ET32aV.jpg</a> )<p>I think this is also the reason why some autistic people feel uncomfortable when they don't know each detail of a not yet happened situation in advance. Because communicating the fractal dimension, or "roughness" of a problem or situation is the most time consuming and fragile phase in a project.<p>Here's an article about: "Roughness of fracture surfaces and compressive strength of hydrated cement pastes", which appears to be completely out of topic. But I believe it's nearer to the best estimation technique than other techniques. (Fig. 3) <a href="http://www.sciencedirect.com/science/article/pii/S0008884610000311" rel="nofollow">http://www.sciencedirect.com/science/article/pii/S0008884610...</a><p>While you may critique that I've not contributed to solving the problem, you may also notice that I've helped to shed some light on the roughness of the problem to be solved :) (Am lazy, it's very late and I'm just back from training to be honest=)

@akeefer nailed it. This is an essay about why project management is bad, written by someone who it seems like has never actually studied project management.<p>Good planning and estimation is the tool of the worker, not of management. It keeps pointy-haired bosses from coming in and asking "is it done yet?" When done properly, it help justify the number of resources that will be needed to deliver the project on time, and the amount of cash/resources that it will take to complete the project. Good project management results in a not-to-exceed date that is 90% confident (only 10% of projects will exceed this date), so the team has time to tackle unknowns that pop up along the way.<p>And, yes, at the end of the day, project management is a tool for accountability. Like it or not, everyone has to be accountable for their performance at the end of the day, whether as parents, life partners, or employees. Saying that you don't want to be held accountable by some stupid boss is naive and unrealistic.

&#62;In machine learning, overfitting generally results from having too much test data or focusing on irrelevant metrics.<p>Huh? Overfitting usually happens when your training set is too small. The size of the test set does not affect overfitting because the test set is, by definition, only used to evaluate the accuracy of the final learned function.<p>In addition, overfitting doesn't happen because of "focusing on irrelevant metrics". It happens because your data set is noisy, or because your learning model is too simplistic to fully model the observed phenomenon (which is known as deterministic noise).<p>If your model focuses on irrelevant metrics, that won't actually be a problem as long as your training set is large enough to reveal their irrelevance. After training, those metrics will not have much bearing on the output function.<p>This misinterpretation of overfitting really hurts the analogy.

So what do you do as a freelancer working on projects that last weeks or months? Tell your clients to go screw themselves, that you'll be ready when you're ready?

In a 2001 article, J.P. Lewis demonstrated using the Kolgomorov-Chaitin-Solomonov noncomputability theorem that there are large limits to software Estimation:<p><a href="http://scribblethink.org/Work/kcsest.pdf" rel="nofollow">http://scribblethink.org/Work/kcsest.pdf</a><p>Algorithmic complexity is not computable, then:<p>1. Program size and complexity cannot be feasibly estimated a priori. 2. Development time cannot be objectively predicted. 3. Absolute productivity cannot be objectively determined.<p>In fact, Software Estimation methods have an error margin of 100-400% (see Kemerer, C. 1987: An Empirical Validation of Software Cost Estimation Models").<p>Software Effort Estimation is harmfull because trusting in anything with a 400% margin of error is risky.

I've spent a fair amount of time criticizing the way estimates are done (for example: <a href="http://deathrayresearch.tumblr.com/post/4503505772/the-pathology-of-estimates" rel="nofollow">http://deathrayresearch.tumblr.com/post/4503505772/the-patho...</a>) but although I empathize, this post missed the flight deck completely.<p>First, as other have pointed out, sometimes estimates are needed (If the guys at YouTube finish their work for the Olympics next month, they have a problem.)<p>Second, if people only work on software where there's a clear a 50 to 1 payback (as the post suggests), there would be demand for about 10,000 programmers in the world. Competition forces most companies to fight for small, temporary advantages in their products and for incrementally lower costs with their backend systems. (And this ignores entirely the relationship between time to market and returns. Sometimes software is worth a lot if you could have it today, not so much a year from now.)<p>Third, story points? Not the panacea they're made out to be. I wrote a post on that, too, but one self-referential link is enough for one comment.<p>Finally, if you're not going to estimate a completion date, why estimate at all? The reasons he gives for estimation are actually reasons to spend time on design. Estimation done that way adds nothing.

Perfect system for me as a software developer:<p>1. When considering what is to be done, ask me to asses how hard it is in points. Also ask me to tag the task with words that I consider important like "database schema change" or "attach legacy system" or "IE7" or "find third party library". You can also ask me about additional things like, what subsystems I think I'll have to touch or source written in what languages I think I'm going to touch.<p>2. Use the prior knowledge of how long similarly hard, similarly tagged tasks took to guess how long this task will take, but do not tell me that because I'm not really the one that is interested.<p>3. If I'm given any input from people who want to get this done ask me how it changes hardness of the task and the tags.<p>4. After the task is done ask me again about same things and update your knowledge according to measured time the task was actually worked on, my initial predictions of initial hardness and tags and my final evaluation of actual hardness and tags and any other information you chose to gather.<p>Worst thing you can do to get an estimate is just ask me how long will it take. I'd rather you, not I make the WAG because then everyone is aware of the fact that it's a WAG.

This is the best article on software I've ever read.

As akeefer pointed out, this was a long rant which finally got to the point: separate effort and scheduling. Take a quick chop at estimating effort, but not in terms of duration/dates on a schedule. Relative size works fine. It's better to take a quick hit at estimation over and over again as you begin to see empirical results than it is to spend a lot of time up-front trying to do the perfect estimate.<p>Software management is not like programming. It's also not like other kinds of management. Any group of people needs some sort of management function. The trick is to put in the most lightweight structure possible. For some reason many technical people of great intelligence, when faced with planning activities, begin to construct various kinds of paper and mathematical wonderlands to live in. We naturally enjoy creating complex models.<p>Don't do that.

I'm still reading the essay -- I had to stop and comment about the following paragraph:<p>"This results in folks that think adding more developers to a project at the beginning will allow them to hit an arbitrary date by spreading out the workload. Unfortunately new software doesn’t work this way, either, as the same exponential communication overhead occurs in the beginning, too. You merely end up with a giant mass of incoherent code. And you waste a lot of money employing people who don’t add much value, sitting on the bench waiting for their part of the project to begin."<p>These must be folks who stopped reading The Mythical Man-Month after chapter 2? I would think the following chapter about The Surgical Team would counter such thinking. Unless people are only taking away the lesson about the programmer productivity gap?

Agreed! No real point throwing out random estimations instead of actually sitting down and writing the code

There is a great deal of philosophy here, but I'd prefer to see some data that backs up his point. Can you show me a project in which investing in an estimate harmed it?

Using the phrase 'considered harmful' in the title of a blog post is considered harmful

FYI: Flights are on-time if they are within 15 minutes of their scheduled arrival. Arriving 16 minutes early is not an on-time flight.

As a PM, the other reason developers (or anyone else) doesn't like making estimates:<p>1) it's hard 2) you are accountable for them<p>No "My dog ate my homework" type excuses, no leaving at 5pm for a week and telling me on the last day your work will be late.<p>When you make an estimate, you are putting your credibility on the line. No-one is 100% perfect, but you should at least give meeting the given dates a solid try. Not "Whoops, didn't make it, can I have another week please?"