We are clueless about how long things should take

Great article. When I worked at $OldJob, the leadership wanted an unsolved, research-grade problem solved in a few months. They demanded estimates, then refused to accept estimates beyond their timeline, then tried to hold people &quot;accountable&quot; for missing the estimates. Of course it was a mess, and of course we failed to hit our goals, and of course many many people burned out.<p>But I noticed that some people handled the situation fine. They stayed on management&#x27;s good side, even though they were failing to deliver along with the rest of us. I will try to distill what I observed them do:<p>1) They did not fight on the estimates. If a manager forced them into a certain timeline, they registered their disagreement and just accepted the new timeline. I think they realized that fighting would just make the manager judge them less capable. (Pick your battles, eh?).<p>2) When the schedule slipped, they would communicate it in a way that made them seem more competent instead of less. The explanation usually had three parts: unforeseen events kept us from hitting the deadline, we accomplished some great things in the meantime, and here&#x27;s why we&#x27;re in great shape to hit the next deadline! For example: &quot;When we made this schedule, we did not realize that AWS nodes were so unreliable! Despite this, our team has made incredible progress on implementing a fast method of storage and solid compression! We have reworked the schedule to reflect this new information, and we are already on track for the first milestone!&quot;<p>It&#x27;s possible that this trick just worked for the specific situation at $OldJob, but I really enjoyed learning it. They seemed to understand that certain explicit rules were not important, but that other unspoken rules needed to be followed. Are accurate estimates important? It depends on the situation! Sometimes wrong estimates can be more valuable than correct ones. Construction companies give wrong estimates all the time in order to win projects. Is staying on good terms with your boss important? Yes! Even if they are total shitbags, being adversarial won&#x27;t help, only leaving will. These people demonstrated that there are often ways to fix a bad situation by breaking some explicit rules and carefully following implicit ones, and I wish I had the acuity to see these possibilities on my own.

All the research indicates that software developers are terrible at estimating <i>time</i>, but are fairly accurate at estimating <i>relative effort</i>. The only reliable method is to break down projects into the smallest deliverable &quot;chunks&quot; that you can, and then estimate the relative effort of those &quot;chunks&quot; compared to actual, concrete tasks that the team has completed in the past. After doing this for a while, you can use these relative effort estimates to project completion dates for new projects. This is what agile &quot;story points&quot; are supposed to represent, but many teams unfortunately end up with things like &quot;2 points = 1 week&quot;, which just doesn&#x27;t work.<p>If you use this method, it&#x27;s <i>absolutely critical</i> that the team always focus on relative effort, and not on time. If you want it to continue to work, you also have to be very careful how you communicate with team members regarding their progress and estimated completion dates - don&#x27;t tell them that their component is &quot;late&quot; - this will make them think in terms of time next time you&#x27;re estimating something, and will ruin your estimates.

How it often works:<p>Manager says I want an A that does B? How long?<p>Most of the time estimate is N days, actual is N +&#x2F;- 20%.<p>Sometimes it&#x27;s N days, and you get lucky it&#x27;s N&#x2F;3. Instead of credit, you now have a reputation of padding estimates.<p>Sometimes it&#x27;s N days, and you go oops, and it&#x27;s 10*N. Now you are a slow incompetent.<p>This is why estimation is such a no-win for developers.<p>For areas that come closer like construction, they have centuries of prior experience and often the contractors are big enough to push back.<p>To use the example of this article, go to a boatyard, and say I want to 150 foot yacht with a hot tub and a 20 seat theater, and my budget is $10,000. You would get laughed out of the office.

I moved away from web development after many years of full-stack development. It almost totally destroyed the joy I had in programming.<p><pre><code> - happy Agile team? That alone will cost you 2 days of work per week due to meetings, planning etc.. - Javascript? No, it has to be Typescript for even the most futile websites nowadays, and yes, TS adds another 20% of workload - TDD, with a dedicated testing engineer?, no of course not, you do it yourself! Another 20% of added workload - A shitload of tooling from linters to bundlers and whatever else that always needs some attention - Deployement, done by a devops engineer? you must be joking right? That&#x27;s also the work of the full-stack dev.. </code></pre> Now try to estimate how long it will take to implement a simple feature request from the PO? I always did my rough estimation, times 2. And even that was often not enough because all kinds of urgent issues needed to be resolved, so you&#x27;re kind of lacking all the time which is a great recipe for burnout. I&#x27;m done with it.

I&#x27;m not sure how to avoid answering these loaded questions though. In my experience trying to &#x27;negotiate&#x27; with people like that: they will just ask you the same question in 5 different ways and it will eventually get so uncomfortable you&#x27;ll tell them anything to move on.<p>Another option might be to never do business with cash-starved companies. Founders and execs will be constantly worried about running out of money, acquiring customers, and pulling off miracles, and all that stress will trickle down on you probably for no real benefits.<p>A key question to ask an employer then is how much &#x27;run way&#x27; &#x2F; money they have left? Or whether they have a revenue source. It seems fairly probing, but realistically, not everyone wants to invest heavily in a company that might not even be around in 6 months.

This sort of thing is why engineers are paid way too much, and way too little, and why so many see the profession as a young man&#x27;s game. You&#x27;re just not the master of your own fate. Sure, the computers are predictable (for some value of &quot;predictable&quot;...) but management is not---even if they try to be.<p>I&#x27;ve been mentally moving away from being paid to write software. I can see writing it for my own use, even professional use---I just don&#x27;t want to be on the hook for ever-changing requirements, decided by people who are often kind, but not, in the end, competent. The best managers understand this and will give you leeway, but this is not a sustainable, repeatable thing. It lives and dies on one relationship.<p>I wonder sometimes: what if we just all stopped writing software one day, and started just using it? <i>Writing</i> software is a bad deal in a lot of ways---hard, socially isolating, etc---while <i>using</i> it is amazing---the computer does the work for you!<p>Don&#x27;t get me wrong, I spent an hour at work today presenting on Lisp macros and loved every minute of it. But a dev career, for many, means a capped income and a razor&#x27;s edge of apparent competence.

My understanding is that it is harder for engineers to estimate how long a job takes, than to do the job. That is to say, that the complexity of doing a time estimation task is higher than the complexity of task you are estimating. I read this in the book &quot;Making Software: What really works and Why we believe it&quot; [0]. The assumption in the article is that an engineer can produce reliable estimates for complex work, and appropriately guide their managers, and don&#x27;t think that&#x27;s true.<p>[0] <a href="https:&#x2F;&#x2F;www.amazon.co.uk&#x2F;Making-Software-Really-Works-Believe&#x2F;dp&#x2F;0596808321&#x2F;ref=sr_1_1?keywords=making+software&amp;qid=1569398896&amp;s=gateway&amp;sr=8-1" rel="nofollow">https:&#x2F;&#x2F;www.amazon.co.uk&#x2F;Making-Software-Really-Works-Believ...</a>

A few years ago I was brought in lead a team in a division of a public company. Our mandate was to replace a major component of an enterprise system that you couldn&#x27;t quite call legacy because they&#x27;d never actually gotten around to replacing the real legacy system with it.<p>My initial proposal was to build it incrementally: tackle the obviously critical functionality to get something working and ship it, then define and implement additional features on an ongoing basis. I had hoped this would fly, since the company talked a big game about being agile.<p>But no dice. We had to have a complete plan for all the features currently supported (some of them of dubious value, many of them incomplete and inconsistent in specification), and it had to come with a specific timetable. Under protest, the team and I worked hard to produce high-level estimates, and ended up basically guessing that the thing would take six months.<p>No dice, it had to be delivered in three. I made a suggestion along the lines of the strategy suggested in the article -- we could identify a subset of features that the team would be comfortable could be delivered within the deadline. It would probably be stronger, I argued, since it wasn&#x27;t clear that all the added weight added much value.<p>No dice: everything was priority one. I pointed out something like, &quot;you can&#x27;t fight the laws of physics&quot;. I apparently gave the impression that we&#x27;d get it done anyway, though my memory of the conversation was that I was sternly disagreeable.<p>Our team goes ahead and starts implementing items from a value-prioritized backlog. Fast forward three months, and we have a working system that supports the most important use-cases. We considered it past ready to ship, knowing we&#x27;d need to keep iterating. The response from management, predictably, was frustration at the missing features, despite the advance warnings.<p>Management goes into &quot;high-pressure&quot; mode, and for the next three months I do my best to keep the team insulated. After more or less six months of total development time, we finally replace the prior component. All the users agree that it has far fewer bugs. I and many of my team members grumble that it has far too many, on account of the fact that we weren&#x27;t given the chance to ship the minimum viable product when it was ready.<p>I&#x27;m not really sure what the moral of the story is.

I’ve gotten plenty of requests like this:<p>‘we just need a rough estimate, we won’t hold you to it’.<p>‘Ok based on the 2 minute conversation we just had I think about 3 months’<p>‘What! That seems way too long’<p>Other times I’ve been asked for estimates on features even though there is a hard deadline due to some external factor. I really fail to see the point of estimating anything when there is already a decided upon end date. Anyway I usually point out that they will need to just put the features in order of importance and I’ll work down the list. And they should start thinking about what can be cut. This usually leads to protests of ‘we have already cut everything we can’. But I have to laugh as we get closer to the deadline that suddenly not every feature is as important as was originally thought and magically get cut.

If managers come to developers with problems that are NP complete, no matter how well they transfer the idea&#x2F;vision, or set time expectations, the problem cannot be solved efficiently. And, unless the developers have been trained in CS&#x2F;Math they may not even understand that what they are being asked to do cannot be done.<p>For example, say a manager has an idea to find the largest group of friends in a massive social network. He wants a developer to write an app for that and has a 20K budget and 2 months. You could not write this app with 10 times that budget or time.<p>How can you determine which group of friends is the largest?<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Clique_problem" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Clique_problem</a><p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;List_of_NP-complete_problems" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;List_of_NP-complete_problems</a>

There&#x27;s no substitute for technically competent management, period. If you don&#x27;t have it, you will be limited, and its something you just have to accept -- as in, make that a high priority question to ask about when interviewing for your next job. Further, its not that software developers are bad at estimating. Its that software estimation is only useful when taken as a very rough guide for strategic planning. You need to know if something will take 2 weeks or 2 years. You don&#x27;t need to know if it will take 2 weeks or 4 weeks. If you think you do (as most shops I&#x27;ve worked do) -- you are micro managing your team. The sooner we start calling it that, the sooner we can all accepts its as much an anti-pattern in software development as it is every other facet of business.<p>In short, estimate roughly. Agree to high level deliverables. Assert that technical management needs some control over strategic business decisions, and flexibility in implementation and timelines. If you can&#x27;t get those things, temper your expectations, or look for a job with more competent leadership. The last point has become my guiding principle. Stop thinking you can &quot;fix&quot; an org from the bottom. You can only fix an org if they hire &#x2F; promote you to do so, and empower you as such. That&#x27;s what you ask for (or look for in your managers).

Project plans and estimates are usually fantasies made only to give executives a sense of control. For people who are actually trying to get something done in the world, I think the better thing to do is give them actual control via short-cycle processes and frequent delivery.<p>At my last startup, we got very good at releasing early and often. What would have been projects elsewhere got broken down into very small releasable units; our average story size was under a day. Very occasionally my cofounder would have us ballpark estimate two different batches of work using arbitrary points, just so he could get a sense of the relative cost of two paths. But we never estimated in terms of dates.<p>This worked for us because he really took advantage of the speed of iteration. Almost everything we built came with a question. In the next user test, would users understand it? Would they want to use it? Did they react as expected? When we sent some traffic to it, did people engage? Did the right people engage? Did they return to use it later, indicating value? Etc, etc.<p>The answers to those questions would drive what we did next. Because our goal wasn&#x27;t to build features, but to make things happen in the real world. And once you get used to continuous learning, it&#x27;s obvious that planning too far in advance is wasteful. All of the brilliant ideas you had a month ago weren&#x27;t based on what you learned in the last month. Eventually, sensible people learn to stop producing a lot of plans that never get fully used. And, of course, to stop asking for estimates on them.

One of the most painful types of people to work with is the CEO &#x2F; founder who got lucky with a product early and came to believe they&#x27;re a &quot;product guy&quot;.<p>As in: My early product hit whatever trend &#x2F; wave &#x2F; need therefore I must be a product guy (and not just lucky).<p>The product guy has a preternatural ability to understand what the masses want. Watching them work -- witnessing their process -- is something to behold. They will steer products in a direction regardless of cost, complexity of likely outcome.<p>The outcome, quite often, is to tank their company. Since they don&#x27;t understand why they were successful in the first place, it&#x27;s very likely that their success won&#x27;t last.<p>But if you&#x27;re along for the ride, wow, expect the following:<p>1. You&#x27;re the greatest (available) engineer we&#x27;ve ever encountered, building super-complicated XYZ is going to take this company to the next level!<p>2. This is taking much longer than expected and isn&#x27;t matching up with our expectations but I&#x27;m 100% sure of my vision because I&#x27;m a product guy.<p>3. We&#x27;re running out of money (because the market conditions that gave us early success have changed) and super-complicated XYZ isn&#x27;t going to rescue us -- because you&#x27;re a worthless piece of shit of an engineer!<p>See what happened there?<p>They&#x27;re sometimes hard to distinguish from a vanilla bullshit artist. The bullshit artist will tell you how well capitalized he is, tell you he only wants the best (meaning he thinks you&#x27;re expensive) and then try to slowly whittle your sense of self worth down until you get &quot;the offer&quot;:<p>The offer is game-changing, life-altering for you: Instead of continuing to pay you with money, they&#x27;re going to start paying you with magic pixie dust. The magic pixie dust will make you rich &quot;when everything comes together.&quot;<p>When you tell bullshit artist that you don&#x27;t work for magic pixie dust, that&#x27;s when you learn that, in fact, you&#x27;re a worthless piece of shit of an engineer.<p>I actually respect the bullshit artist more: They&#x27;re bullshitting other people but they know they&#x27;re full of shit. Product guys, depressingly, bullshit themselves.

Fantastic article. The diatribe against estimation reminds me of this old article from 1996: <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20140604112011&#x2F;http:&#x2F;&#x2F;www.thomsett.com.au&#x2F;library&#x2F;item&#x2F;estimation-games" rel="nofollow">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20140604112011&#x2F;http:&#x2F;&#x2F;www.thomse...</a><p>&gt; It is our belief that over the 30 plus years of commercial computing has developed a series of sophisticated political games that have become a replacement for estimation as a formal process.<p>Unrelated: can we not do this thing with the whole left side of the screen being one static image? It&#x27;s really distracting.

Timelines kill me. When I&#x27;m half way through a project I can tell you exactly when it&#x27;s going to be done. Or I can plan the shit out of it upfront and give you an accurate timeline but that doesn&#x27;t fly because you need the timeline upfront. I like working - can&#x27;t I just work till it&#x27;s properly done?

My usual way of doing this, is by spitballing an estimate with the team, multiplying it by 2.5 or 3, inventing some meaningless milestones and gantt-charts and presenting it to an outraged management. Then we haggle down to something realistic + a thin buffer. The management has a sense of control, we have realistic deadlines.<p>Caveat: I work in the financial industry, where the complexity of writing a compiler and whipping up a Tableau dashboard are perceived to be equal.

I’ve been participating in build pipeline work for quite a long time now, and fairly early on I noticed a pattern. If the build takes ten minutes on your local box then the tempo of code-build will tend to be 20 minutes, even when you are just fixing typos.<p>This is partly why some people are obsessed with very short builds. Some said seven minutes was ideal to avoid the developer getting preempted. Then it was three. Now some shoot for one.<p>I would bring this up and others would say they had noticed the same thing, but none of us knew why this phenomenon happens.<p>Then it hit me: it’s just Hofstadler’s Law playing out in the small. If you think you have five minutes, you will start something that you estimate will be five minutes, but it will take you ten, either because you were wrong or you get distracted.<p>There aren’t a lot of tasks that take one minute, and even if you’re wrong you only lose one minute.

&gt; If engineers stop giving estimates for their work and simply ask for deadlines then it changes the dynamic of the conversation.<p>If there&#x27;s a single line you want to take away from this wonderful essay, it would be this.<p>Parkinson&#x27;s law usually takes care of the rest once the resources have been agreed upon.

When I was a young programmer in the early nineties, an old IBM programmer told me &#x27;Don&#x27;t accept the invitation to fail.&#x27; Good advice but what if the goal posts are moved. Best then to keep positive and communicate early and often about how the changes will affect the schedule. Work together as a team and get your manager to help you and that&#x27;ll go a long way rather than keeping &#x27;obvious&#x27; things to yourself and catching management off guard when work slips.

Web designers: Thin sans-serif body font means you hate your readers&#x27; eyes.<p>I&#x27;m grateful for &quot;reader mode&quot;.<p>- - - -<p>There&#x27;s a very interesting book, &quot; Hollywood Secrets of Project Management Success&quot; that details their system from an IT point of view. Here&#x27;s a decent review: <a href="https:&#x2F;&#x2F;community.dynamics.com&#x2F;nav&#x2F;b&#x2F;navigateintosuccess&#x2F;posts&#x2F;hollywood-secrets-of-project-management-success-58-a-review-of-a-sort" rel="nofollow">https:&#x2F;&#x2F;community.dynamics.com&#x2F;nav&#x2F;b&#x2F;navigateintosuccess&#x2F;pos...</a><p>&gt; Inside this system lurks the biggest difference between the IT and Hollywood: movie industry exists for more than a hundred years already, they had enough time to develop and establish best practices and to prove them in practice to such an extent as to tell everyone: this is how you need to do movies; and everyone can trust it works, because it has worked for the whole industry for decades already. IT industry is very young, and exists for a few decades.<p>(Although, IT is technically as old as the cuneiform-inscribed clay tablet, eh?)<p>Specifically to this discussion, they (Hollywood) can set expectations <i>reliably</i> because have enough shared baseline experience of how long things take.<p>One way to interpret this is to ask yourself, of some new change in process or technology, &quot;Will this stabilize delivery times?&quot; But to even begin to answer that kind of question you have to establish reasonable ways to map between work done and results delivered, and then set up and track metrics. (Which is easier said than done.)<p>- - - -<p>One problem unique to IT is the &quot;interpretability&quot; of our end products. Anyone can watch a movie, but most software requires at least some training to understand and use.

Good article. I thought about this problem quite a bit. I&#x27;ve been on both sides of the question.<p>I thought I was brilliant when I came up with the solution of fixing the time frame and estimating the work that can be done in that time frame. Turns out I came up with sprints, 60 years after they were invented.<p>The fun solution for this would be to give &#x27;hit dice&#x27; estimates for tasks. Assign type of dice and number of them to each eastimated.<p>How long will this take?<p>About 1d20 days.<p>Nobody will be happy with this, but it is the most realistic one. Cause tasks do have that variability to them.<p>The wises thing said here is: &quot;the reality is that if you can make a probabilistically accurate estimate, then it&#x27;s likely that the task should have been automated by some other means already. &quot;<p>Is there an answer to this problem? Maybe abandon long term estimates entirely. Having really short term estimates, with frequent updates.

Agile has his concept of &quot;horizon of predictability&quot;. Basically, you are semi-accurate within the two-week range. After that, the accuracy nosedives. At 6 months, you could be off by, well, 6 months.

Great article. This is very true for game development as well.<p>But serious question - for live products especially, you do need to have some sort of schedule where you are launching new features every X weeks. So it&#x27;s important to know how long your features will take, so you can have a constant cadence of updates. Plus a lot of times you will have marketing initiatives or other things that need to be coordinated with your releases.<p>So my point is that you cannot just remove estimates. There is a need for knowing when the current sprint &#x2F; feature will be completed, and being somewhat accurate about it.<p>I do really like the point about re-framing the conversation to start by asking the manager how long they want the engineering team to spend on the new feature. That will definitely change the dynamic and hopefully should encourage a conversation about what is realistic to do in the timeframe that the manager has in mind, and how the feature needs to change in order to achieve it.<p>But after that, the engineer still needs to go through and create estimates to make sure what they just agreed on is actually possible, and then those same estimates are necessary to plan out the development to make sure you are on track. So yeah, you can never really remove estimates.<p>Am I wrong?

Estimating knowledge work is always more unwieldy, and in this context you are often working towards outcomes which ultimately have no precedent (although they may be made up of known components).<p>It is also more difficult to initially assess the skills fit of candidates for knowledge based work, especially those that require creative problem solving, and unlike other engineering disciplines past outputs are opaque and hard to rely on as simple markers of past performance.<p>For a project, in order to produce a good estimate you need to understand scope, then align it with precedent, adjust for your resources and productivity profile and then view all of that through a risk lens to set probable outcome ranges.<p>For a programme, in order to produce a good estimate you need to understand and manage the risks, constraints and dependencies across all your projects and ensure that the projected benefits (both hard and soft) are still net positive, meaning the investment makes sense.<p>From my observations at least it doesn&#x27;t look like the idea of development as &quot;investment&quot; in a product or service is very common. I&#x27;m assuming because time to market is often the ultimate driver rather than cost, in which case, congratulations, you should increase your costs on more numerous and productive resources whilst aligning your strategy and risks to iterate on smaller scopes faster so that dead ends can be quickly parked.<p>The problem isn&#x27;t so much the estimation process as it is more generally poor portfolio&#x2F;programme governance and management practices and more specifically a lack of risk management and understanding of contingency at those levels. I find IT, and Software Development more particularly, to be some of the worst offenders, but that is because the risk profiles of such projects are vastly different to the risk profiles of other types of work. The productivity of your resources is difficult to discern and a lack of precedent for similar-enough projects and knowing what their variables were, all meaning you really can&#x27;t produce a reliable estimate with incomplete information.

From my experience (3 technical co-foundings, 1 additional top-tier startup, other companies) it’s rare for management to incorporate our estimates. I’ve been coached by a VP Eng on how to blow off actual estimating, in that top-tier startup. Sometimes, it may be quasi-rational in that the apparent business constraints just don’t care about the estimates, in which case, yeah, just cut to the deadline and budget and we’ll see if we can do anything that isn’t embarrassing. Management usually dictates rather than engages, that’s how you get to be CEO in the first place.<p>“We” are usually not at all clueless. “They” tend to be.

At certain points, this article touched on the commoditization of engineers and engineering skills. Beyond extremely simple things, it&#x27;s never going to happen. Especially as you build a company and end up with a few monoliths through various acquisitions, all while concurrently hundreds of microservices. Things get too intricate and too hairy between systems. There&#x27;s no amount of handwaving that will convince me that machines will replace Software Engineers anytime soon. Solving valuable, enterprise-scale problems will never be as simple as a Wix drag-n-drop solution.

Nice article, but what&#x27;s with the web page design? On desktop, the left half of the screen is just the blog title, while the article is squeezed in the right half. Safari on Mojave if it matters.

I&#x27;m reading <i>The Moon is a Harsh Mistress</i> right now (fascinating, bizarre, and highly enjoyable thus far), and there&#x27;s a moment where the main characters discuss a timeline described as &quot;on the order of 50 years.&quot;<p>This is still a fairly common phrasing, and in general, people around me generally take it mean &quot;roughly&quot; or &quot;plus or minus a few,&quot; which is how one character interprets it. Another character then explains that its actual meaning is, paraphrased, &quot;probably not less than 5 years or more than 500 years.&quot;<p>While I had a vague awareness that &quot;on the order of&quot; was not mathematically equivalent to &quot;roughly&quot;, I don&#x27;t remember learning the concept in such a memorable way. Similarly, an order of magnitude are commonly understood as &quot;a lot more than&quot;, when they also have a precise mathematical definition.<p>I think that reason we are terrible at estimating the time something will take is that humans struggle to think in terms of timespans that are actually representative of the variance in an estimate, not to mention the fact that as additional variables are introduced, that variance will almost certainly increase (even if the midpoint decreases). I&#x27;m not sure of the causal direction here, but our misunderstanding of terms meant to succinctly express &quot;somewhere between&quot; sure doesn&#x27;t help things.<p>(side note: there&#x27;s also a discussion about designing a revolutionary organization that really clarified my rudimentary understanding of circuits. The book is truly stunning at times.)

Wow, lots of great opinions here I&#x27;ll look forward to reading and learning from. My very brief take is:<p>Estimates should be for &quot;making the best decision now&quot; and not judging performance. Those are orthogonal concerns.<p>It&#x27;s very fair for the business to need to weigh &quot;well if X takes 10 days and Y takes 20 days, and X makes more money, of course we&#x27;ll do X&quot;. That is what estimates are for.<p>However, if X ends up taking 15 days, the business (CEO, product, even CTO depending on their closeness to the solution) shouldn&#x27;t&#x2F;can&#x27;t decide if that is a performance issue--only the engineer&#x2F;the engineer&#x27;s manager&#x2F;etc. can make that call.<p>And maybe it is, maybe it isn&#x27;t.<p>Granted, pulling this off is really hard; tangentially I&#x27;m now working in the construction industry which has the very same problem: this home should take 9 months. It took 10! Who&#x27;s fault is that? Well, that&#x27;s the wrong question. The right question is how could we have known that delay sooner&#x2F;better, and mitigate it if possible this time and more importantly next time.<p>If you&#x27;re interested in working on a &quot;humane&quot; project management system (I just made that term up and it&#x27;s super early, so disclaimer&#x2F;etc), reach out &lt;&#x2F;shill&gt;.

I&#x27;ve always had strangely accurate time estimates. The basic idea is to divide up the work, and make a fair estimate each part, with special attention to unknowns (which get much more time).<p>Then double everything.<p>The individual task estimates are often out, but the overall estimate is close... as if, with a population of tasks, there&#x27;s regression to an accurately estimated mean.<p>(An alternative explanation is that I over-estimate, and Parkinsonianly, work expands to fill time.)

I think time estimation can be learned to some degree. As an anecdote a couple of years ago I had some free time and my sister was renovating her home. I volunteered to tear down all the wallpapers (basically an entire apartment). I had no idea how long it would take so I took a kitchen timer, set it to 25 minutes and started chucking away. After a couple of these 25 minute runs, I could predict fairly well how long the task would take for a certain room. I also found it interesting that it was very easy to adapt the estimate to changing conditions. For example in the kitchen the wallpaper was soaked in fat and there were 4 layers glued over one another. After working on this for about 5 minutes to see how much harder it is (answer: a lot) I was able to estimate fairly accurately how long it would take.<p>Granted this is for manual labor where you know exactly what will happen. Estimating more uncertain environments is a lot harder. Nevertheless, I have grown used to using this time-chunking approach for all sorts of new tasks. I think shorter timespans (10 minutes) work better for estimation but for actual work, 25 minute &quot;sprints&quot; are good.

There is something to be said about pushing engineers to tighten their deadlines. It&#x27;s not necessarily out of a desire to shift blame. Instead, it&#x27;s often to encourage them to work hard for the same amount, and to think up novel solutions to problems where others might not have. In other words, it&#x27;s an attempt at maximizing productivity. Start ups and some teams are running on legitimately small budgets and squeezing every moment of the process will net a large outcome if done consistently.<p>Having said this, the idea that you can ask for something in 2 weeks when your eng estimated 4 weeks is ridiculous. I would automatically add 50% to my eng&#x27;s estimates because their estimates are probably too ambitious to begin with (I fall to the same problem when estimating my own work).<p>If you consistently get undercut in your estimates, get out! I&#x27;ve seen this behavior most often in marketing agencies and game companies. I would not work for these types of people regardless of the money they are paying.

In my experience the deadlines are not a big problem; the communication about it is though. I worked with PMs that try to attack the issue by shooting into micromanagement mode (symptom: the project management system gets filled up with literally 1000s of tiny points with random deadlines in order to try to manage the fact that they don&#x27;t get how software dev works); this drives most devs mental up to a point they burn out. There are only so many &#x27; why is task xyz3835433 not done? It was set to be done yesterday?&#x27; &#x27;Yes, because it was set by <i>you</i> and task xyz3835433 cannot be done in that time&#x27; &#x27;Why did you not tell me?&#x27; &#x27;Because I do not have time to respond to 9000 points every day; they are not relevant to my work&#x27; etc. Usually these PMs come from another discipline like management of marketing or PR campaigns, were breaking deadlines is not an option. But complexity there is far easier to manage and oversee and they do not understand (even after many metaphors) why our estimates are not spot on. They will communicate with the client (internal&#x2F;external) in massive MS Project files and accompanying docs that no-one will ever read and berate the team for being so much off in the estimates.<p>Then there is the other type of PM who will continuously manage back the client expectation. To the layman, this person seems to tell the client (internal or external) bad news on a daily basis; it&#x27;ll take longer, cost more and you&#x27;re getting less.<p>It&#x27;s the latter PM + team that actually will get the flowers and the cake on launch day and a have the happy devs that did not have to sleep under their desks while the former team will be near burn out, client unhappy <i>even</i> though they probably technically did deliver more (but also too late). Usually that former team won&#x27;t get more money for it either...<p>I have seen both in startups and fortune x companies; I have seen both as contractors and as internal teams. For the anecdotal part here; the ones with a PM like the first one in a product company internal setting, those companies all failed that I have seen&#x2F;worked with. For contracting work, it can work, but it&#x27;s a stressful and panicky way of working which often results in one-off contracts.<p>You inevitably come to something like sprints and good, timely bad news talks. People who have the micromanagement type need for control are just not going to survive those as their struggle breaks down even inside a 2 week sprint period.

It strikes me that the issue is, to some degree, rooted in the transfer of the vision from one person to another. I have struggled with this as well, on the side of the &#x27;visionary&#x27; rather than the developers.<p>A quick search for &#x27;how to communicate what you want with software developers&#x27; turned up what I expect is at least modestly helpful: <a href="https:&#x2F;&#x2F;www.entrepreneur.com&#x2F;article&#x2F;224816" rel="nofollow">https:&#x2F;&#x2F;www.entrepreneur.com&#x2F;article&#x2F;224816</a> (2012) but I&#x27;d venture that the topic really is worthy of much more in depth treatment.<p>A key element missing at least from that post is: How to confirm that the developer does understand what the vision is? If that can be confirmed, they would then be in a position to modulate the vision vs. reality, and, even contribute some of their own creativity. I suspect that this modulation frequently occurs with a start from <i>misunderstanding</i>. The divergence can be dramatic.

Related article (albeit a more mathematical focused analysis of the same problem): <a href="https:&#x2F;&#x2F;erikbern.com&#x2F;2019&#x2F;04&#x2F;15&#x2F;why-software-projects-take-longer-than-you-think-a-statistical-model.html" rel="nofollow">https:&#x2F;&#x2F;erikbern.com&#x2F;2019&#x2F;04&#x2F;15&#x2F;why-software-projects-take-l...</a>

Posting one more time; David Packard&#x27;s (of HP fame) address on &quot;How to be a Manager in a Technical Company&quot; -<p><a href="https:&#x2F;&#x2F;gizmodo.com&#x2F;the-hp-way-how-bill-hewlett-and-i-built-our-company-5634378" rel="nofollow">https:&#x2F;&#x2F;gizmodo.com&#x2F;the-hp-way-how-bill-hewlett-and-i-built-...</a><p>One thing i noticed in discussions like this one is the amount of excuses made for Management&#x27;s&#x2F;Manager&#x27;s lack of Technical Proficiency. This has to stop. If you are in a Technical Domain and leading&#x2F;managing Technical People you have to know the domain decently well. There is no other alternative. Else you are reducing the effectiveness of your Engineering Team and the Company as a whole to your level of incompetency. The resulting effects may not be visible immediately but sooner or later you will drag the company down to oblivion (eg. see what happened to the same HP).

Why does everyone accept time estimates are accurate outside of software? Has no one hired a contractor to build out a bathroom, paint a few walls? Commissioned an artist for a sculpture? It&#x27;s my experience that <i>anything</i> that can run over schedule will...why do developers get so much flak for it?

How good are coracles though? Not much good as a yacht, but then you can&#x27;t carry a yacht on your back and walk it upstream can you?<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Coracle" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Coracle</a>

IMO if you&#x27;re familiar with the language and libraries it&#x27;s pretty easy to predict how long writing some feature will take (provided there aren&#x27;t bugs in anything you&#x27;re working with, you don&#x27;t get any weird emotional hangups etc. Those are fairly rare.)<p>What&#x27;s hard to predict (at least for me) is writing non trivial config files or using some new library or language or finding a new algorithm. Some of those things you might be able to estimate how long it might take to estimate the time it will take but other times you just have to say &quot;I don&#x27;t know&quot; and time box it.

This is amazing. I work in a law office and have nothing to do with tech but just change the examples and he has just described my typical day that replays over and over in matters both large and small.

Reminds me of this article [1] from Michael Wolfe. I&#x27;ve sent this to several PM&#x27;s over the years and they&#x27;ve all begrudgingly agreed with me once I applied this to the current project. The beer probably helped too :D<p>[1]: <a href="https:&#x2F;&#x2F;www.quora.com&#x2F;Engineering-Management&#x2F;Why-are-software-development-task-estimations-regularly-off-by-a-factor-of-2-3&#x2F;answer&#x2F;Michael-Wolfe" rel="nofollow">https:&#x2F;&#x2F;www.quora.com&#x2F;Engineering-Management&#x2F;Why-are-softwar...</a>

I think the commonly claimed rules about adding increment your unit of time or multiplying by x are kind of childish and a wasteful. Estimating time to complete is difficult because individual items have high variance. If you have 3 items that are likely to take 2 hours +&#x2F;- 2 hours, estimate 10 hours for the lot. Variance of the total goes down by a factor of n^(-1&#x2F;2).<p>Explain why you do this. Flag when variance is screwing you anyway.

Everyone on my team knows i can guess when they are going to finish usually to the day. They don&#x27;t know how but it&#x27;s pretty easy, measure biases

The author illustrates how easy it is to be clueless about how long things should take with his parenthetical about how trivial it is to make a Wix website. Yes, you can get the “used sailboat” class of website in Wix in an hour, but that’s probably not what you need or had in mind. The fact that the tool seems to do so much of the work for you will arguably just make the expectations even more unreasonable.

That&#x27;s because developers shouldn&#x27;t be making estimates in isolation. They only know what they know about themselves. People who watch developers across many projects can better understand the &#x27;likeness&#x27; of similar projects, similar inputs (in terms of developers), and then talk to developers about the differences of a given project to come to a better estimate.

Its honestly so so nice when you have a project manager that understands being Agile(tm) is about the schedules and deliverable reacting to shifting realities as well as the engineering team. I guess too many pms don&#x27;t understand that they shouldn&#x27;t stick their neck out with what they don&#x27;t have yet and paint themselves into a corner with promised deadlines.

&gt; I found myself absolutely astonished that tech founders could be so clueless as to assume that a simple rest api integration should take the same amount of time as a real time transactional distributed ward’s clustering implementation for peta bytes of data, or a highly available complex distributed metastore.<p>I would love to know the context behind this. Beautiful website by the way.

The article completely misses the point. No amount of reasoning will change a manager&#x27;s mind. They do not push people because they actually think it can be done faster, they push people because software developers are very weak and because they want THEIR project to be delivered as soon as possible so they can boast in front of the other managers.

Quote from the article &quot;Engineering is not getting simpler, its getting more and more complex, because we are solving harder and harder problems&quot; I totally disagree with that , for mine work environment. We are over engineering stuff. It all has to be new and shinny. Lots of sites can still be build with a small framework, html5 and jquery.

sorry for earlier post, a lot of stuff still hurts.<p>i&#x27;m in poland. my type of manager would be a person who accepts me, gives me clear target and constraints and leaves alone, person who does his job, person who doesn&#x27;t bullshit me and says the truth if we&#x27;re in shit or not and realises that we are all on the same boat and we&#x27;re also waist deep in shit so there is no point in fighting each other.<p>on 16 IT companies I worked in, there was only 2 such people. that&#x27;s 12.5%. shouting and batshit crazy behaviors are normal.<p>someone here wrote that some things are cultural. a lot of stuff is cultural in poland. its a catholic fundamentalists country. so anyone who solves problems is a problem. people here had no type of french revolution or industrial revolution. basically its feudalism in modern dress.<p>last guy i worked for was freestyling everything. he had no plan. he told me he uses his imagination and he tells people what he had imagined and they have to do it. i found out about this when i was trying to resolve communication issues that i had with him. he often imagined new stuff and forgot to tell me about it and during code reviews he blamed me for not doing what he wanted me to do. i ditched the guy because he didnt wanted to do anything with that problem. the guy sabotaged himself throught the whole time. sometimes he had some strange outbursts.<p>there is a lot of such people in poland. a lot of office workplaces are like kindergartens. there&#x27;s constant chaos. no plan. big ideas but nobody wants to wait. people who want things done asap usually are abusing others, not to mention that they are total morons cause abusing people takes time they dont have. most companies are micromanaged.<p>thing that keeps me going is knowing that, as someone said here, this stuff is cultural and that there are some better places. although after all those companies i&#x27;m kinda like a dog from the impound. i dont know how to trust people. word nation is for me the smallest joke possible to say. i&#x27;m afraid of polish speaking people.

This post is assuming that all startup founders are non-technical people...

&quot;Bro, I&#x27;ll just get some other nerd to do it. Instagram was built by 12 guys and WhatsApp was acquired for $19B with only 50 engineers. Can&#x27;t ship if you whine; so shape up and stop wasting time!&quot;<p>&#x2F;s

this is basically the whole point of correctly applied Scrum

not sure if it was mentioned, but I&#x27;ll throw out recent other issue not often talked about - external dependencies&#x2F;vendors.<p>&quot;integrate with external API&quot; - had a project where there were several external data sources (financial service providers) to import on a regular basis. One had an actual API, commercial service, good docs - took a few weeks.<p>The others are...<p>1. &quot;hey, we&#x27;ll send you a nightly file, except it&#x27;s not always nightly, because it depends on someone running the job and if they&#x27;re not here, you won&#x27;t get it&quot;.<p>2. &quot;here&#x27;s our SOAP WSDL&quot; - &quot;this doesn&#x27;t work&quot; - &quot;oh... try this other one&quot; - &quot;that doesn&#x27;t work&quot; - &quot;try this one, but just don&#x27;t use some of the endpoints cause they don&#x27;t work&quot; - &quot;OK, but this doesn&#x27;t really work either&quot;. Now... intersperse those sentence fragments with a minimum of 4 business days via email (sometimes going for a couple weeks because &#x27;vacation&#x27; time).<p>3. X was working for 7 months, then stopped. &quot;oh, we changed the file name and format of what we push to you yesterday&quot;. no warning, no documentation on what the change is. just... pissed off end users who are now saying &quot;my data is wrong!&quot;<p>Figuring out how to take data from a file or SOAP or REST endpoint and process it - that&#x27;s not terribly hard. Figuring out how to deal with more than half a dozen vendors who are not &#x27;really&#x27; in the business of providing data, but do it half-assed anyway - there&#x27;s no end to &#x27;figuring it out&#x27;, because it&#x27;s a moving&#x2F;changing target.<p>I&#x27;m not naming any negative names but I&#x27;ll mention that quovo.com was comparatively pleasant to work with - they&#x27;re an actual commercial service. however, less than a year after we coverted a system to use them, they were bought out and some of the useful functionality seems to be sunsetted already. I&#x27;m not on that project directly anymore, but talk to some colleagues still involved in it.<p>From the client&#x27;s standpoint, it&#x27;s all &quot;integrate with external data providers&quot;. &quot;You did one, the others can&#x27;t be that hard&quot;. But each provider is a completely separate island of functionality, documentation, responsiveness and professionalism.<p>For the record, no, you shouldn&#x27;t be providing me with client SSNs as their identifiers (quovo didn&#x27;t, but I&#x27;m surprised at others that do, in at least one case that&#x27;s the <i>only</i> way they provide client identifier data at all).

Hofstadter&#x27;s Law: It always takes longer than you expect, even when you take into account Hofstadter&#x27;s Law.

Doesn&#x27;t address how highly technical people dealing with new systems are also wildly innaccurate in their estimates.<p>It&#x27;s like a expert in tech, business and computer science trying to predict stock prices. I mean that&#x27;s the expert of experts failing.

most managers is drug&amp;alcohol infused money starving 10 year olds with no fucking idea what they are doing and knowing only how to abuse people like their fucking abusing power figures called parents

The Content-Type header of the article is set to the empty string. It still renders, but I&#x27;m guessing the author hand-coded a website and screwed it up somehow. He probably should have used one of those Wix-style solutions he mentions in the article.

I only hire people who can reliably estimate. How can you call yourself a pythonista if you cant reliably estimate. Incompetent estimators have no place in my organisation.