The other half of "Artists Ship"

I think this can be generalized to say that start-ups have a different cost-benefit analysis - one where losses are capped at the (relatively small) value of the company. On the other hand, large companies <i>have</i> to be risk-averse because the worst-case is several magnitudes worse. When you're working at ConEd or AIG, the "tiny probability/worst-case-loss" factors start to matter, because "worst-case" can include investors losing life savings, federal investigations, and jail time for your boss's boss.

This past summer I worked as an intern for a big company. On the first day of orientation the head of the legal compliance department said very flatly to us that any individual could do far more harm to the firm than good.<p>This mentality, reenforced by the company's bureaucratic change management system, really did not sit well with me.<p>Fortunately for my summer experience, my "buddy"/summer mentor and I found a loophole which we used to "hack" the change management system so that once we got our initial approval, we could propagate changes to prod without running through the whole process again for each change (which otherwise would have been required).

Some things that PG didn't make explicit: web applications can have an extraordinarily tight feedback loop between customer and coder. Multiple revisions per day are easy, especially if any single user's data is rather low-value. Many a startup of this kind has zoomed past its corporate competitors simply by iterating faster. If your product is embedded firmware for a home security device, this strategy is just not available.<p>But here's another twist: when you start working for a giant company that provides myriad services under the same login, suddenly everyone has to move as slowly as the slowest part of the business. It doesn't matter if you are just doing Yippee! Backgammon, because who knows -- maybe you could somehow accidentally expose the data at Yippee! Payment Solutions.<p>This is a completely rational consequence of being a giant company and delivering multiple web services under the same login. It is possible to have slightly different authentication policies for each service. But I wonder if OpenID providers have thought about this enough.

<i>"...not only wouldn't these guys have broken anything, they'd have gotten a lot more done."</i><p>I don't think the claim that they would not have broken anything can be backed by facts.<p>If you look at Microsoft, for instance, their programmers used to write very buggy code, but with the introduction of better processes, like their Secure Development Lifecycle, they saw a substantial improvement in code quality (as measured through security and reliability metrics).<p>The damage done by not following a quality assurance process and writing buggy code was so big that Microsoft's image will be affected for a long time, even though they have now improved the code quality.

I think the costs of safety checks can be more pronounced in software than in other areas of business. It's not just that good hackers are particularly irritated by them -- they're irritated for a reason.<p>A two-week release-process lag is so much worse than it might sound. If you can ship instantly, you can get feedback and make improvements rapidly, over and over. Sure, without a careful release process you might occasionally break things, but when you do you can fix them in minutes rather than weeks. When you insert release-process lag into that cycle, improvements that might have been made in a rapid series of releases over a day or two can take months. Sometimes those improvements just never get made, because no one is motivated to keep working on them for so long.<p>In software, "stable" often means "unchanging", but only rarely means "high-quality".

Some thoughts:<p>As companies grow big, they like to <i>emulate</i> larger companies, and hence the people they like to hire are from larger companies since they <i>understand scale</i> -- this comes with a lot of baggage ofcourse. This in turn, most times, brings with it a culture of people who are worried more about not screwing up, covering their own ass, and thinking about getting promoted rather than doing what's right for the company or the business and screwing up as a byproduct of making quick decisions.<p><i>Re Joel Spolsky:</i> The cost of the sale is not only the cost of the product, but what an organization would pay for a software/service, and what they would value it at. This is why most companies have a sales force, and don't advertise their prices on their website. As Joel mentions, for a lot of companies, charging less than $50k is a rounding error and not worth their time.<p>The flip side of people at big companies buying is based not entirely on the <i>performance</i> of the service or the product. As long as the service/product is reliable &#38; ok, and it doesn't screw up in any major ways, they won't get fired for making that buying decision, rather than looking at ways of maximizing/optimizing the <i>performance</i> of the product -- yet another reason committees make buying decisions.<p><i>Re SOX:</i> I like what founder's fund + facebook is doing -- letting early employees cash some of their equity out, thereby increasing the time early employees will stay with the company.<p>I also like Fred Wilson's thoughts on a secondary market for startup stock (similar to what goog does), this would in some sense, I guess let you be a private company, and not deal with the challenges associated with SOX compliance and at the same time not raise a highly dilutive Series D, in addition there is some sort of liquidation event for the investors.<p>PS: <i>(std. Buchheit comments about Limited Life Experiences + Overgeneralization = Advice apply)</i><p>PPS: Say hi to the reddit guys for me ;-)

There's an excellent prototyping policy in effect at Maxis:<p><i>In terms of time, they have a policy of permission vs. forgiveness. You need to be prepared to fail early – but that’s okay! If a prototype takes less than two days, don’t worry: just go ahead and do it. It if takes more... you should probably have permission.</i><p><a href="http://www.gamasutra.com/php-bin/news_index.php?story=11628" rel="nofollow">http://www.gamasutra.com/php-bin/news_index.php?story=11628</a>

We used to do releases whenever we wanted. After this took the site out one too many times, we moved to at most daily releases with an hour of QA.<p>This actually made things worse. Aggressive releases were often the result of a decision further up the chain that something was "critical" and needed to go out right then and there. We'd got into the pattern of expecting to be able to do this at the drop of a hat.<p>So now, everyone was rushing to get the "urgent fix" into the daily release. This is an effective way to ship buggy software. Extra emergency releases were frowned upon, and so it introduced a level of panic that "Oh shit, this better not screw everything up". Ironically, with anyone releasing whenever they wanted, there was less panic because you could always make a quick fix and put it out without too many people noticing or caring.<p>We felt as a team that we'd prefer a more relaxed schedule that also allowed testing and accountability.<p>Frequent releases were a result of bad management, and unrealistic stakeholder expectations, we as developers didn't necessarily care that our code wasn't being put out immediately. Obviously, it needs to go out at some point, and we've been equally demoralised by mammoth projects that have gone on for months without a release, but that's a different end of the spectrum. The important thing is <i>regular</i> releases, not frequent ones.<p>We've now decided to move to releases every 2 weeks with a 1 week QA period. We made this decision as a team of developers, it wasn't handed down from above, it just became a necessary check for the scale of our company.<p>I personally feel a lot more productive as a result, less panicked by the quick fix you need to drop everything to fit into the daily release, and less worried that a release will take out the site.<p>But sure, it comes at a cost. There's now a lot more going out with each release and more variables that could combine to cause issues, but these issues highlight deficiencies in our QA process that can be fixed. Also, getting our team to work according to this 2 week schedule has meant some significant costs to adopt better development processes, namely SCRUM, but I feel our team has benefited enormously from it. It all depends how well your team can adapt really.<p>So yes, there are costs associated, but it doesn't necessarily lead to demotivated developers. You need to be sure that the checks are introduced properly with their own test suite. This itself has a cost but can be extremely worthwhile. A half arsed and arbitrary change to your release schedule with no process changes can destroy you, but with a little training, the right team and the right managers, you can make things better for everyone.

Having never had a "normal" job per se (okay, I bagged groceries for a year when I was younger), I still find it hard to believe that it can take two weeks just to get something deployed. Maybe one day I'll finally <i>truly</i> believe all the stories I've heard -- two-week deployments, meetings about future meetings, etc. For now, some part of me still believes it's simply implausible and everyone is engaging in hyperbole...

I would add that in addition to having to show the cost of any new checks that one should have to show how the check will actually prevent the problem it proposes to prevent and the likely-hood of that problem occurring or re-curring. A major problem I've seen with checks is that they just become CYA material rather than anything truly beneficial or preventative and people will work around them.

One problem is that the cost of checks is only visible in the aggregate. The marginal cost of each new check seems to be pretty low.<p>I wonder if there's a way to limit the total cost without hard-and-fast silly rules.

This is what I call "playing defense" instead of "playing offense". By nature, the people at large companies are more concerned with risk aversion than innovation. This is also how managers like to "hold power" over the real innovators by controlling the change management process. What makes a manager more qualified to deploy to production than the programmer?

To "make something people want", you need to know your customer's value system: what's important to him and what's not. This can be hard for startups, especially ycombinator types which may be very low budget, because their own value system is so different. Startups start with a company worth about $1 and want to turn it into a company worth about $1,000,000. Therefore startups value the ability to change rapidly -- that's why they want good programmers.<p>Big companies on the other hand, start with a company worth $100,000,000, and want to turn it into a company worth $150,000,000. They value steady process improvement that doesn't risk their existing revenue -- that's why they want good managers.<p>It's important to know this if you're trying to sell to a big company as well as if you're trying to compete with a big company.

<i>In fact, the acquirer would have been better off; not only wouldn't these guys have broken anything, they'd have gotten a lot more done.</i><p>That's where you lost me. Sure, there are companies who add checks without thinking of the consequences, but as a company goes from a startup that is not making money to a more mature company that is making money, the cost of breaking things becomes very, very high. That is why most post-startup companies add more checks, to protect their revenue stream. And any company making decent money has a revenue stream that far outweighs the amount of money that the programmers would be willing to pay to have a faster release cycle. And the risk of the company losing their revenue stream for any significant amount of time far outweighs the risk in losing decent programmers because they aren't happy with the length of the release cycle.<p>You seem to say that these particular programmers would never have broken anything, but I find that impossible to believe unless they are working on something that is not at all complex or is inconsequential to the revenue stream. Every programmer, even the best programmer in the world writes code with bugs in it. The more complex a system is, the more likely there are to be bugs in it and as a company grows and makes money, the more complex their systems will become.<p>As an example, let's say that you were responsible for a web application that brought in a million dollars of revenue a day and you had these supposedly perfect programmers that never made mistakes who were responsible for the code. Would you let them just throw out whatever code they wanted onto the servers because you trusted them to not make mistakes? Or would you be more reasonable and put some checks in place first to make sure that the application worked correctly?<p>Obviously there needs to be a sane balance between the risk of breaking the application and the opportunity cost of not updating the application and the frustration of slowing down development is one of the costs that should be weighed, but to say that there shouldn't be any checks added as a company matures is almost laughable.

<i>Whenever someone in an organization proposes to add a new check, they should have to explain not just the benefit but the cost. No matter how bad a job they did of analyzing it, this meta-check would at least remind everyone there had to be a cost, and send them looking for it.</i><p>To me that's the most important paragraph here. That takes you from "let's make sure this never happens again" to a cost-benefit analysis. The issue is not whether or not to have checks, as much of the discussion here assumes. It's about realizing that not all checks are equivalent, and using that knowledge to get the greatest safety at the lowest cost.

<i>"... Steve Jobs's famous maxim "artists ship" works both ways. Artists aren't merely capable of shipping. They insist on it. So if you don't let people ship, you won't have any artists ..."</i><p>Steve Yegge has had the unenviable luck of working on 3 products all of which have not shipped (cited as business reasons) yet he still works for Google ~ <a href="http://blog.stackoverflow.com/2008/10/podcast-25/" rel="nofollow">http://blog.stackoverflow.com/2008/10/podcast-25/</a> So it seems there is something else going on that keeps programmers on board. Is it money, having the freedom to talking about it the process? I don't know.

Paul,<p>Surely if the costs of checks could be "discontinuous," "non-linear," "step-change," or however else you would like to term it, then perhaps the potential cost to the company of NOT having the check in place could also be the same. Larger companies potentially have more to lose by having recurring lapses in supply, or in the ongoing case of your essay, software service. Of course, software/coding may also be a special case. Additionally, it's true that your "committee" example represents a discontinuous change in bureaucracy for that firm, which would make it much more reasonable to expect an equivalent affect on cost.

There is a related concern I have about costs piling up due to corporate or managerial mandates. I have the impression that people above us only think about what will make their jobs easier -- what else do they need for us to supply them with so that they will be able to achieve predictability, manage or reduce costs, and generally manage the company better. When they come at it from this angle, it always seems reasonable that they ask us to do more -- provide more reports, track time against projects, report metrics -- when in fact most of what they ask actually interferes with us being able to get real work done. Since they/we do not have adequate productivity measurements in place already, the resulting reduction in productivity is never noticed, hence an invisible cost has been added without a conscious decision or cost/benefit analysis. Since I think most individual contributors recognize that this is taking place, it acts as a disincentive.<p>In the end I think the solution to both the article's and my concerns are the same: It is always better to ask the members of the team that experienced the problem firsthand what they think the solution would be or what they would do differently next time to avoid the problem. Especially if you phrase it as "what could you or your team have done differently to improve the outcome" and urge them to avoid proposing major process initiatives if at all possible, the improvement will be more practical and lower cost in all senses.

<i>Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times. And when I used to mow lawns, I definitely didn't prefer it when the grass was long after a week of rain.</i><p><i>Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.</i><p>Salesman like it best when they can sell more. When I used to sell used cars, I loved it best when I could keep pushing cars out of the lot.<p>Writers like it best when they can write more. They like it best when their imagination can produce thousands of stories.<p>The difference is that coders usually know what they will tackle and have a strategy so it is easier for them to get to the end point. Also they are almost sure they will hit the end point of a certain problem.<p>With other professions you cannot predict much. Your next customer is not guaranteed, your next story just does not want to show up in your brain etc....<p>That being said, Yes, a lot of programmers I know work hard.

I found this particular paragraph to be not right somehow:<p>Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times.<p>Who are the people who are "best at" fast food? I think perhaps you mean that people who build things get enjoyment out of building -- perhaps more precisely, of seeing their built item in action. I think programmers in particular are used to having short feedback loops between building and trying out, and that pattern of building up and testing as you go until a larger problem is solved is a strong motivation.<p>I guess I'm claiming that harder-working isn't really the issue. The issue seems to be who is having more fun. And in that sense, your claims still make sense. Part of the fun of building is a short feedback cycle. More checks in the process make the feedback cycle longer, therefore less fun. That explains why teachers' like their jobs. You can see it in someone's face when they learn something new.

I disagree with the claim that harm to the U.S. IPO market was "not the intention" of the "people who wrote" Sarbanes-Oxley.<p>I have two pieces of evidence.<p>First, it was well-known and easy to see that Sarbanes-Oxley would be extremely expensive. It was well-known when passed that its costs fall disproportionately on small public companies. Because IPOs by definition are small public companies, the costs fall disproportionately on them. Thus, it must have been known when enacted that the bill would harm IPOs.<p>Second, suppose in fact it is the case, as Graham seems to advocate, that Congress "inadvertently" harmed IPOs. His argument is that Congress just accidently happened to overlook the harm of the bill to IPOs. In that case, when the harm to IPOs became factually clear, the bill would have been changed or amended. The fact that the law was not amended even after the harm to IPOs became clear proves that it was Congress' intention all along to harm small public companies (the companies most dangerous to the large corporations who have the most lobbying pull).

Very nice - I think another reason why companies overpay - is because they don't negotiate, they don't check out the hot companies, they allow their employees to make decisions and most have never hired a company - so they go with the best name so they can say they hired them - it looks good on their resume - companies don't realize that most executives - director level and above are doing nothing more than building their resume - and their projects go unchecked, unaudited and bascially the company gets screwed - and frankly it is the company's own fault. Once a company loses it's founding partners, the transition team runs wild with all the changes they want to make and not looking for those things that made the company successful - spending is at all time high, brand suffers because the new team is too busy spending that they lose sight of what is important - the customer - gosh does that sound like AIG?

Another interesting read. I'd like to note that (while it wasn't the focus of the essay), many of the expensive "checks" aren't there for monetary reasons, but rather human safety and the dire consequences of a poor decision.<p>Working at Motorola on cell phone code could be a great way to give everyone in the modern world a cool new feature - or accidentally break 20+% of all phones out there. A terrible Mickey Mouse movie could destroy a brand. Take it a step further to NASA, vehicle engine design (and safety), oil pipelining and refining, etc. and the cost could be human lives.<p>That said, I'm very interested in the cost of these "checks" on those industries. Not monetary costs (although important), but rather the cost in stifled future innovation.<p>Thoughts?

I guess so many <i>checks</i> in a big company could be considered the biggest <i>mistake</i> of all?

How about a check immunity system?<p>For example, competent people who knows their own weaknesses well and often check their decisions with others should be immune to checks<p>Good programmers often let other good programmers check their codes. Good writers often let other good writers proof read. Good managers often discuss his decisions with his employees before deploy them. They should be check immune.<p>You can even pass out check-immune badges to encourage non-check-immune people to be more critical of themselves :)<p>Of course, this system itself is a check. Who decides who will get check-immunity? What's the successful-decision-ratio threshold? I think it's possible to implement a light-weight system little cost.

Yes, startups make their job better than big companies. But I think that it is a part of a more general rule: people work productively in OPEN systems, and the open systems themselves encourage working hard. Open source projects are developed more rapidly, because it is hard to imagine the system which is more open. Both users and developers control the system, and every user can become a developer. In startups we have a limited number of people who can control the process of development, but they still controll it. In other words, the system is open for founders and employees. Big companies work unefficiently because they are too closed, and a great number of checks is only a part of it.

Nice writeup on half an idea. Now, time for some release engineering. How do you give good coders free (enough) reign while keeping tabs on the stupidities of the lesser ones?<p>In a startup, it's acceptable for service to be accidentally down for short periods while the geniuses running the show fix their latest cockup. In large companies, it isn't. How do you get the best of both worlds?<p>It is not by removing all safety interlocks. Sysadmins, especially good ones, can tell you in excruciating detail why not. Most programmers are not very good sysadmins, or at least not as good as they might like to believe. Coincidence?

This would be interesting as a concept to develop past the circumstance of programmers at work.<p>I would call it part of institutionalisation. You have a need to be effective via policies, checks, procedures &#38; such. This replaces the shoot from the hip of small groups.<p>I think you can apply this to schools that need to teach approved courses with approved grading. Then it gets worse when they need examinations that are to be applied across an entire country.<p>I'm sure there are principals that can be extracted &#38; applied to lots of places. The costs can be very serious.

My experience is that extensive procedure comes because the small company start up approach doesn't scale to large companies. Every one I have worked at that has transitioned from from 10s to 1000s of people was failing because it tried to continue as if it was 10 wizards at it. The wizardy actually hindered others because it was opaque to them.<p>Interestingly the biggest company I worked for actually ran the best because perhaps its roots being 80+ years before any of the others would allow the significant difference to be tracked.

When I was an automotive engineer the industry instituted a Japanese "check" called Design Failure Mode Effects Analysis. For every component the engineers would have to predict the ways the design could fail in testing and what steps would have to be taken to fix them. Time spent doing DFMA's added significantly to the cost of producing a new part and worked counter to weight optimization. Do you think, twenty years of DFMA's later, that automotive design is more efficient?

"The purpose of the committee is presumably to ensure that the company doesn't waste money. And yet the result is that the company pays 10 times as much."<p>Well, only if they still buy every product, which would defeat the purpose of having a gatekeeper committee. If, instead, the committee approves only 1 in 25 products that would otherwise have been bought, the committee saves them money (leaving aside the cost of the time of the committee-members).

" And you know what? It would have been perfectly safe to let them. In fact, the acquirer would have been better off; not only wouldn't these guys have broken anything, they'd have gotten a lot more done."<p>My experience as a manager of a team of programmers for many years is that for sure they would love to be able to write and release w/o a QA process, they do in fact take the site down when this is allowed!

Checks have a cost, for sure. What about Eric Reis's split-testing axiom? If in a startup the developer/coder is basically the customer of the enterprise because the cost of the check is paid by the rising frustration of the coder, then should split-testing be looked upon as a check? Even if it is, the idea of an available metric to backup claims of efficacy is really compelling.

For me this is all about "Mental Poisons".<p>Mental Poisons are ideas that harm people, life and the universe. They are remarkably common.<p>Paul expresses this one very well. There's a lot more out there too.<p>Fossilised heirarchies and thought-free-rule-driven organisations are great for the student of poisons.

Thanks for this article Paul. Now i really know the why behind the what i observed working for a big company<p><a href="http://computinglife.wordpress.com/2008/11/14/hands-free-or-hands-bound-mouth-loose-3-mouth-loose/" rel="nofollow">http://computinglife.wordpress.com/2008/11/14/hands-free-or-...</a>

Two weeks seems a bit much, but once you have a significant client base of paying customers, you simply must have some form of production control and QA. Even good developers make mistakes, and letting everyone just push their changes to prod over lunch is a recipe for pissed off clients.

As a previous startup coder who was acquired by a much larger company, I can truly say you've hit the nail on the head. I never minded working 70 hour weeks before, but now I have a hard time punching the clock 35 hours a week. I hate my new higher paying job.

Oh, wow. Yes. I've often thought that a lot of this process is just to keep the lowest common denominator from messing things up. I want to spend my life surrounded with people that don't need this, because then we could do so many more worthwhile things.

It seems that checks affect journalism too. "Citizen journalism" allows people to report much more quickly than traditional media. Bloggers don't have to submit their posts to editors and fact-checkers, the way broadcast and print journalists do.

Hi Paul<p>I'm surprised you didn't tie this in to due diligence checks done by VCs, and its impact on startups, etc. The parallels are strong, and you've commented on that impact before (at FOWA in 2007, I think).<p>Regards, Terry Jones

I'd say that the reason for bureaucracy in large organizations is that central management doesn't trust middle managers. Whereas in a startup there are a total of three bosses, and all trust each other.

The cost of the checks should balance the cost of the risks. Those are not always determinate or exactly estimable, but, statistically, that would be a simple basis of a rational approach. Wouldn't it?

Small side node:<p>I think it is very cool to have the comments here instead of his website. It is nice to have them consolidated and not some comments here, some there.

The movie "Meet the Robinsons" makes a point to children that we make the most progress when we are free to fail. Thanks for making the same point for grownups.

This is exactly why we don't respond to RFPs any more.

pg worked in fast food?

I guess the best check (and benefit) for startups are commit mistakes early, commit mistakes often.

in Russian <a href="http://spring.jumpidea.com/2008/12/paul-graham-artistship.html" rel="nofollow">http://spring.jumpidea.com/2008/12/paul-graham-artistship.ht...</a>

i love the last paragraph of that post Paul. so true.

Wonderful. Well thought, researched and valuable for large and small companies alike. <a href="http://wildwildeastdailies.blogspot.com" rel="nofollow">http://wildwildeastdailies.blogspot.com</a>