It seems like a lot of people here have been in an ineffective large organisation and an effective startup. I've worked at an <i>effective</i> large organisation and an <i>ineffective</i> startup, so perhaps I can shed some light on what these kind of processes are for:<p>* You don't get to be a large organisation without accumulating generations of previous products. Well, unless you're google and can regularly fire your customers without going bust. but part from them, you start to need processes just to track what's going on. (The ineffective startup accumulated previous product attempts too, and is still paying for a lot of pointless infra because no-one still there knows which ones can be switched off).<p>* At a large scale, it starts to be pretty difficult to maneuver if each dept of 30-50 people has built or contracted all their own infra & services. Some divergence is useful for agility, but a lot of it is just waste and actually slows you down when you want two depts to work together on something - or even prevents useful collaboration. An effective organisation will make the tradeoff consciously.<p>* When you complete dozens of projects per year, it starts to be possible to invest in researching and rolling out best practices in areas where a startup just has to go with the simple/obvious answer. When my startup employer moved, it was pretty onerous, and there was a bunch of stuff we never found again. The large org had so many offices that it had a full time dept just for moving offices. (They weren't dumb, each move consolidated offices -but they kept buying other companies). When they moved us, it was like magic - we went home on a friday and went to work on a monday in the new office, and everything was set up. Well, we had to set up the lab ourselves, but
It's hard because it's meant to be hard.<p>Once you have a large org that's making money, you don't really know exactly why it makes money. Sure, there's annual filings and there's still the elevator pitch of what the org does. But the actual how of how it works is difficult to describe in causal terms: if team X or Y was not there, the business would do better/worse? How does this team interact with that? It's really a bunch of informal relationships between different people, and both the people and the relationships change over time.<p>So what you want to do when you know the whole sorta-works and you want to keep it that way is you pour glue all over it. You try to formalize processes, you give people titles and you make hierarchies. That way you attempt to stop the firm from inadvertently slipping into dysfunction.<p>The side effect is of course that everyone who works there can see ways to improve things, but they can't see a way to get those improvements implemented.
Big mistake in the article to claim that a single engineer could build Amazon, Facebook, or Google. Amazon is the most egregious.<p>> . A software developer could write a simple e-commerce website in a few days; Amazon employs 1,335,000 people [source].<p>Those 1.4 million people are not coding the e-commerce app but making e-commerce happen as package handlers, drivers, finance teams, etc. on top of that Amazon has many non e-commerce related projects: building hardware (Kindles, Alexa, etc) , AWS and support, and on and on.
Ronald Coase won the nobel prize in economics for his "transaction costs" solution to the "why do firms even exist" question. A lot of economics is about asking a question the right way.<p>He started with "if markets are so great and stuff, why do companies run like soviets & chiefdoms in practice? Why doesn't the product team just contract a UI designer, a testing team and such. Hi solution was "transaction costs." IE, it's too hard to find a designer every time you need a button.<p>Economics and social science generally is prone to a "defend my model" problem. Once you have a model, there's a tendency to see everything in terms of that model. A Coasian sees everything in transaction costs terms. Coasians ran around<p>This author seems to see everything in incentive & game theory terms. Management theory often prefers group psychology explanations like Dunbar's number.<p>IMO, you should probably keep all in mind at once, and remember that none are entirely true. Even if you see the problem as "Prisoners Dilemma," the solution may not be "breaking the prisoner's dilemma."<p>Human cooperation in groups to achieve aims is basically the secret sauce of human civilization, culture & what makes us different from other animals. If it could be predictably engineered, a lot of problems would have been solved already.<p>Most, seemingly common sense solutions fail miserably. Trying to break prisoners' dilemmas by nailing all responsibility to someone (a) may be a lot more disruptive and aggressive than you imagined and (b) will often result in elaborate CYA responsibility avoidance. See enterprise consulting for a snapshot of a mature system described this way.<p>One way or another, this problem has occured to many people over the years. It's not an easy one. Perhaps, it's <i>the</i> hard question of social science.
The article actually jumps right to the three major points that I see, rephrased to: (1) misaligned priorities, (2) policy flux, and (3) attrition.<p>The most damning trend I have witnessed first hand is that directors have extreme agency and are most of the time not incentivized to help other directors, especially if they’re under a different VP or SVP. The high-level leadership are too detached from the details (as they should be — but this is the thing that is most different in large orgs — it becomes more and more difficult to know everything, eventually becomes impossible, and then even knowing “enough” becomes impossible) to make centralized decisions so you’re left with one organization at a standstill with another, which causes attrition in the lower ranks of people who just want to get stuff done. That attrition results directly in knowledge loss and overall reduced productivity, which hampers most teams from breaking out of their set of problems. And it all just gets worse with time.<p>Large companies have way more guardrails and standards than smaller companies and they also change at a greater rate. Since these policies so rarely help improve or ship features, it’s just an ever-increasing source of productivity loss from the perspective of someone who wants to get stuff done. When internal politics are complained about, this is typically the root cause.<p>Hiring and retaining talent becomes harder as all companies grow and previous top talents becomes disenfranchised over time. Average talent hits an inflection point and then begins a descent. This just adds fuel to the fire, but I believe things would still become extremely difficult without this trait as well.<p>All of this becomes only more difficult to deal with over time as the company increases in size and age.<p>Nepotism, cronyism, etc.. all factor in as well, but those issues also affect small organizations and their impact there is even higher.
See the work on Scaling Laws that <a href="https://en.wikipedia.org/wiki/Geoffrey_West" rel="nofollow">https://en.wikipedia.org/wiki/Geoffrey_West</a> has been working on at the <a href="https://en.wikipedia.org/wiki/Santa_Fe_Institute" rel="nofollow">https://en.wikipedia.org/wiki/Santa_Fe_Institute</a>, there are loads of interviews on youtube where he summaries his work.
> The Prisoner's dilemma is the reason.<p>It's not that simple.<p>A bigger org is slower often because it does bigger things. No, you can't just buy a tractor even though it'll be cheaper and faster for your project than to do all the work to do internal billing to rent the tractor the company already owns.<p>Because if everyone did that then we'd have a thousand tractors and there's nowhere to put them, and we don't have time to sell them, nor do we know how to sell them.<p>And your button on your website is not a one-line change because you're contracting with the US government and the contract has ADA clauses, and it needs 34 language translations.<p>And no you can't make that change because last time someone improvised in this space we got dragged in front of congress and were made to promise to not do that again. So you need to follow procedures, and talk to product councel for exemptions.<p>And you can't change that API because 6000 developers rely on that, and they need to launch before cyber monday.<p>Sure, in that last case that's prisoners dilemma misalignment, but to be clear it's <i>you</i> who are misaligned with the goals of the business, so don't do it.<p>It's not reasonable to suggest a solution of "why don't everyone just stop and do what I want". If you want to tell 10k people to stop what they're doing to work on your thing, then it'd better be right. Which means you'd better know what those 10k people were already doing. And you don't.<p>And you can't just do it alone because it's 100 human-years worth of effort even if nobody's in your way, and you need to launch within 2 years or it's pointless to launch at all.<p>That's what hard about making large changes in large orgs.<p>Actually, correction: it's not even that simple.
Large organizations are hard for startup people but easy for people who want a straightforward job. In a sense sitting in meetings and making decks is an easy way to make a decent amount of money.<p>If you are self-driven and seeking to improve things and do cool work these jobs are psychologically quite difficult.<p>I was always frustrated, first as an enterprise guy, then as a consultant.<p>Eventually I started a company and quit my job, and then it clicked.
I've said no several times when asked if I'd be interested in management - I like writing code and solving problems, management is only vaguely necessary in some bureaucratic unfortunate way. I would say that that approach to writing software defined my 20s. But in the past few years I've become fascinated by the problem of organization and it's made me appreciate good management so much more - getting more than two people to act in concert is _hard_. Getting 10 or 30 or 1000 of them is so much harder. If I think about it too much I start wondering how anything ever gets done at all, anywhere
I’m a small company guy - worked in startups for 30 years - and every time I’ve had to work with senior managers who come in after cutting their teeth in large companies I end up heading for the door.<p>The goals and motivations of large companies are really different to small companies. Large companies are all about risk management - at heart they are about revenue defence - while effective small companies should be all about attack. Being disruptive in a small company is the reason they exist; disruption in a large company is confronting and to be resisted. Going from one to the other is likely to be super stressful.<p>Unfortunately, as I’ve experienced, you do get these same problems in small companies run by big company people who think they know better. In my experience they just add weight. They don’t understand YAGNI.<p>IMO the discipline and skills needed to run a startup, especially pre revenue, are very different from those needed for an established company. It’s not that one is less disciplined than the other. But the size of a small company gives them benefits and room to move that a large company can’t touch. Small companies are already risky so a bit of extra risk doesn’t count for much. It’s to be embraced.
I think the article has some good points about interpersonal comms and alignment problems and their solutions but doesn't actually mention some of the structural / political issues associated with really large organisations. For example, I have worked at or seen organisations that are actually loose groupings of 'cottage industries' (often called divisions or similar) that have separately evolved local solutions to problems (technical /commercial, etc) with a small centralised head-office. Moving between divisions is hard because they are running separate systems. Aligning their processes can improve efficiencies but can also lead to loss of local knowledge or inappropriate one-size-fits-all solutions.
I’ve given the opposite side of this question a lot of thought: given how hard it is to run a large organization, how do they stay in business?<p>The only answer I’ve come up with is that economies of scale are much more important than most people think. And the economies of scale of a Walmart/GM/P&G/AT&T are so big that they override all the petty stupidity in those firms.
I tried to get people to record meetings and link them to features being discussed, so that working on a feature you can review everything that was said. IMHO it would reduce the amount of unnecessary conversation by 50% and save everyone a lot of time and trouble. The initiative failed, because not everyone was eager to give consent. Its their right, but I think if you dont want something to be recorded, it probably shouldnt be said at all in a work meeting.
I think the author actually nails the primary reason in an aside:<p>"As though that were not bad enough, in some organizations there's a separate compounding problem: limited resources. For an organization, attention resources are {time, money, staffing}. In that case you get to layer on the negotiation for resources among competing teams within an organization."<p>While I think the other points are valid, this process of resource allocation alone probably explains most of the problem. As an organization grows, its resources grow proportionally, but the communication required to allocate those resources grows non-linearly. The overhead needed to get $50K from each of two people is more than twice that needed to get $100K from one person. Add in the other issues raised, and this core problem is exacerbated even further.
It is as simple as having too many middle managers. It doesn't matter how large or small the companies are.<p>Each one need to justify the existence of their department. And to get promoted, they usually need to hire more middle managers.<p>These departments will inevitably have confusing directives and even non-sensical overlapping products.<p>Eventually the built-up inertia becomes too large.
The triad of solutions to the prisoners dilemma gives some really fast heuristics. e.g. if time-boxed, then reduce consensus; if hero req'd, give up credit; if consensus req'd, give up time commitments, etc.<p>The best easy book I can recommend about large organizations is Smith & DeMesquita's "Dictator's Handbook" because it provides a practical applied model for determining incentives between groups. Reality is, the people you are dealing with are running their own mental version of the model, where the coalition that keeps them employed and prevailing is made up of essentials, influentials, and interchangeables. They intuit a salience matrix of stakeholders (decision power, vs. how much stakeholder cares), and while everything is dynamic, it is usually within these parameters. It's a tool that can provide fast insight into group/project/org behavior pretty well.
"Everything" really means "changes that I think would be better for me/the company/the world", because if you are happy with what the company is doing, you don't need to change anything.<p>In a truly large organization, it takes institutional knowledge to identify who will have to make a change, who maintains it long term, and who will be impacted.<p>Then you need to convince each group that the change you want is desirable. Sometimes this is as easy as convincing the person in charge of the group. A large one-time change is often easier than a small change to daily procedures.<p>As your knowledge of how the organization operates grows, so will your ability to find the necessary people and phrase things in terms of their interests. In a thousand-plus person corporation, I would expect this to take at least a year or two.
The increase in people increases the relationships and amount of communication geometrically.<p>The increase in levels is like a game of telephone and reduces the fidelity of the message/strategy/goal from executive down to line level team members.
For anyone interested, I wrote a fairly detailed case study of some of the difficulties faced when trying to reinvent the technology stack at one of the world's largest car rental companies:<p>"Why are large companies so difficult to rescue (regarding bad internal technology)"<p><a href="http://www.smashcompany.com/business/why-are-large-companies-so-difficult-to-rescue-regarding-bad-internal-technology" rel="nofollow">http://www.smashcompany.com/business/why-are-large-companies...</a>
My personal experience with large organizations (over 100,000 people) is that everything is so hard because 10-20% of the people do most of the work and the others are not just minor contributors, but many times just roadblocks.<p>For example 1 hour ago I was discussing with my business counterpart about a project we are working on. We just got approval to hire 4 more people (that means we have to hire them, it is not optional) and half of them are diversity targets (meaning: skills are irelevant, we will get them anyway) and the other 2 will be hired and given to us without knowing what are the skills we need. Based on such organization dynamics, we plan to be sufficient without these 4 people and if any of them has any useful skills, it will be a nice surprise. At the same time, we need to give them some work to do, so we need to invent some roles that should be as harmless as possible.<p>This is just a project, in my department of ~100 people about half are just mouth breathing, but it makes it a lot more complicated for the ~ 20-30 that do work to just do their work: some of these people have invented roles to rubber stamp steps in the projects, so they are adding 10-20 steps and a few days of work just to justify their existence. With the department half the size, everything would be much simpler, faster and even more productive.
Group think is well known, but it's also a force that works against individual think. Similar versions of the same symptom emerges as group hearing, group speach, and group action.<p>Thinking alone is easy. Thinking with someone is hard. Now keep adding people. You stop thinking.<p>Same with hearing. Listening to one person is easy. How about two people. If they don't share their time, you're listening to two people talk at once. Same with actions.<p>With more people, thoughts become more ideological and less mobile. Same with speech, same with what they listen to, and same with their actions. And each individual is not only physically a fraction of the group, but so are their senses. And to act humane or to simply perform on an individual level is directly hindered by this group dynamic. Individuals then become lethargic, and begin to experience greater stress and pain. You could also think of this as the Group Flu. I know I've experienced it. It's the feeling you get when you know you need to talk to someone higher up just to be heard. It's the feeling you get when you're talking to a customer service rep and you know you're not talking to a person.<p>Now take a large organization and we have a <i>group of groups</i>.<p>We already know one group cannot do it all, and one group is too inefficient. But if we're just creating more groups, the problem persists.<p>So what you need is to undo the group dynamic. But then again, with a big successful organization, maybe change is what you don't want? Ideology is what you do want, and suppressing the individuals leads to your ongoing success.
> The Prisoner's dilemma is the reason. There might be a way of doing complex multi-participant tasks that is better than what's being done currently, but the incentives for each participant are not aligned. Even when that lack of alignment produces a suboptimal outcome, each participant in the process lacks skin in the game. Each participant in the process creating the suboptimal outcome is not accountable for the consequence of their collective action.<p>The article goes on to describe solutions without actually explaining how the Prisoner's Dilemma operates in large companies.<p>This makes the article less valuable than it could be because it interferes with the reader developing a mental model of the problem.<p>For a take on this that does present a very clear (and possibly actionable) mental model, see "The Gervais Principle":<p><a href="https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/" rel="nofollow">https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...</a><p>The idea is based on the axiom that "organizations don’t suffer pathologies; they are intrinsically pathological constructs."
The biggest component of the answer is simple: risk aversion. The status quo has been working so far, and making changings introduces risk that things might no longer work. An organization will probably exhibit risk aversion comparable to the most risk averse decision maker in the a particular decision chain. This imparts the status quo a type of inertia.
I have definitely experienced effective small org -> ineffective large org.<p>Lots of make work, "lawyers don't get paid to simplify". Growing from small to large company you can end up with a lot of bureaucrats that tell you things like "this policy is what big cos do", but is really entrenching their position and keeping them employed. Creates adversarial relationship between devs and IT/ops/sec.<p>They don't have it easy, because one leak or bad PR due to data and they're shot into space.<p>Best to move to small effective teams, move fast, and show value. Often by currying favour of C/SVP you can get barriers removed fast and get things done.<p>Ecosystem is large, so find your niche.<p>Seen entire teams destroyed by well meaning PMs and other idealists implementing a "system" leading to dev/project slowdown and even abandon. Adding a couple check boxes to a JIRA process seems simple, but every choice is multiplied by the number of developers you have. You can see this in places that implement scrum poorly.<p>I'm reminded of this slide:
<a href="https://speakerdeck.com/holman/how-github-uses-github-to-build-github?slide=56" rel="nofollow">https://speakerdeck.com/holman/how-github-uses-github-to-bui...</a><p>Overly complex security policies, JIRA check pointing, and constant meetings can grind you up. A lot of it is mistrust of employees, or employees also don't have enough context to do what they need to do.<p>It is also entirely possible that I am a particular type, attracted to small/medium fast movers, and then I get tired of battling security/ops/managers for access to systems/data I need to do work, or the new sets of flaming hoops to jump through to do completely mundane tasks.<p>You can always roll the dice! Bonne chance!
I think a large part of the problem is over-specialization.<p>Once you have groups with a leader that are in charge of [something] narrow, they start trying to make [something] more important, more complex, etc, than it should be. For various reasons...exposure for career progression, empire building, etc.<p>Then, when you need to get something from an idea into production, you're dealing with some large number of these groups. All of which have some form of demand management (jira tickets or similar), some form of requirements you have to meet, different ways of interaction, and so on.<p>And, some of those groups have to work with other groups to do "task a" that you asked for. So you get this complicated dependency graph of tasks that's not even visible to you.
The larger an organization gets, the more typical a chunk of society it is, so its effectiveness gets closer to the societal average, and further from whatever can make a small organization special. I think this explanation covers all other explanations I've seen.
Couple of big things missing/brushed aside, really.<p>1) Uncertainty. On larger things, success is often uncertain (e.g., on new products, etc.), so the final consequences of the decisions are not known in advance<p>2) Generally, running processes well across an organization is not something that comes naturally. It needs investment in skills. For example, most meetings should have an agenda and should end with next steps as well as agreed and assigned actions. Tracking and planning similarly need effort and skill.<p>It might very well be that on top of it the incentives are wrong for people to take risks in (1) or developed skills for (2) or even to cooperate in the first place, but most people are not naturally good at administration to start with.
How much space do I have?<p>I think the skin-in-the-game is very significant as the OP says. In a smaller organisation, you might have a person who does several things and one of those is HR. The existence and success of the company depends on hiring well so they do their best. Also, the more efficient they can be (sometimes the worse they can be) the quicker they can get back onto other things.<p>In a large corporate, you now have an HR team. There is usually much less feedback on performance and it is easier to blame the org or the market for poor hires.<p>Another issue is that most employees like most people are only average. A few are terrible and a few are excellent. We can be picky in a small company and find that great person with alignment to the vision etc. What about in your team of 10? You aren't going to find 10 great people so you get a whole range and now some are doing much less than 10th of the work of the team so the excellent people are now picking up the slack.<p>Maintaining multiple communication paths is hard and risky. I worked with a Bank where each requirement spec was signed off by about 15 people (compliance, legal, marketing etc.) how complicated is it to make 1 change? Holidays, other priorities, change of staff etc. can make the smallest thing take really long.<p>Ego: Some people simply cannot compromise. If you are the CEO, you might get away with it but what if someone else is throwing their toys out of the pram because e.g. the legal team need your product to look terrible for compliance reasons? Things you should just accept take weeks or months to argue out.<p>Specialism: The head of e.g. marketing should have the final say on all things marketing ideally. But in many large companies, they might have a disagreement with people who might or might not know more about something but vocalise it anyway. This is magnified if you don't really trust your department heads are good at their job.<p>Oh, so many more things.<p>Is there an example out there of a large organisation that does work well or are there just too many variables to make large organisations work period?
I would say, the bystander effect. The more people you have, the more you have the tendency to think "oh, someone else will pick it up".<p>I am guilty of that myself. In a small team where I know no one is coming to help me, I step up and become 10x productive.
The first half of the article is ok but the part about large organisations is completely misguided.<p>I think the author is conflating prisoner's dilemma with the communication challenges in a large organisation, which arises naturally from their sheer size (as other's pointed out, these companies work in a bazillion products in parallel, not in a single one as the author claims).<p>As an actual employee of one such company, one of the ways they reduce the impact of this is with Principal/Staff level engineers. This is similar to the concept of hero mentioned there. These high rank ICs oversee projects that span teams or orgs, and align them to achieve global positive outcomes.
I don't know if this topic has a name (study of work social structures) but it fascinates me. I see so much counter intuitive events, misery, and low efficacy that I cannot stop thinking about what are the major forces at play
Systems theory tells us that in large organizations nobody actually does what they say they do. For example, one person building a boat is a boat-builder. But nobody at General Dynamics Electric Boat can properly be said to be building boats (or ships).<p>Thus, in a large organization, you can only improve the ability of individuals to do what they actually do, which may or may not improve the ability to the organization to accomplish its nominal purpose.
I find this amazingly insightful.<p>One thing I've been thinking lately is, what if everyone had as part of their "goals"/"performance review" structure -- how have you helped another team (in a non-close branch of reporting structure) do their job/accomplish their goals better?<p>It's probably naively optimistic to think there's a bureaucratic solution to bureaucracy though, which is basically what that suggestion is.
Not a lot of evidence presented to support the idea in the headline. My personal experience is that large organizations can be very, very effective. Bureaucracy works. That's why it exists. In my experience it is the medium-sized organizations that are aggravating: too large for their original ad hoc processes to be effective, too small to develop better ones.
Does prisoner's dilemma really apply in your typical command and control scenario? When enough of person's future wages, wellbeing and so is being held hostage by single party, the choice is obvious.<p>The article doesn't dive much into complexity of problem. Prisoner's dilemma doesn't have much place in traditional organisations. It is more subtle than that.
This line of thinking will eventually lead to Deming's 14 points. It's a failure in management. Bad to OK managers are managers that became managers - they never viewed it as a craft to improve. Great managers build muscles that allow them to see and manipulate the system.<p>Read Working Backwards and it's clear Bezos got it.
This piece is based on learnings from working in a government beauracracy but has lessons for any large organisation. <a href="https://profserious.substack.com/p/10-things-i-learnt-in-government" rel="nofollow">https://profserious.substack.com/p/10-things-i-learnt-in-gov...</a>
Compliance and risk - small organizations don’t even know all the laws/regulations they’re breaking, or in many cases the risks they are taking on. And if they screw something up, it could put them out of business, but it won’t cost 20x what was invested (which could be the case in a large organization)
This is not really an appropriate invocation of the prisoner's dilemma, because the essential feature of that thought experiment is that the participants cannot communicate, and that it is a single round. In a large organization, everyone i a repeat player and communicate to at least a limited extent.
Because adding more engineers to a project increases communication overhead.<p>Obligatory Tweet: <a href="https://twitter.com/RichRogers_/status/1159872097205805056" rel="nofollow">https://twitter.com/RichRogers_/status/1159872097205805056</a>
Large organizations are usually internally run like communist states with planned economies and share the same pathologies. They are basically dictatorships, have 5 year plans, leadership that’s detached from the reality of the base, a lot of effort is put into propaganda, there are many people whose only job is to check on other people, the fruit of successes goes to only a few, and so on. Same structure, same outcome.
Once an organization gets large enough, the different divisions start competing with each other, essentially making several "smaller companies" within the large organization.
Based on my observations, suboptimal incentive policy leads to entropy barriers … overcoming entropy barriers requires time and effort if at all possible …
Good read, the citations are also good. The long story of why we can't have nice things, and why shower thoughts on reforming the system are doomed.
Also relevant: <a href="https://codahale.com/work-is-work/" rel="nofollow">https://codahale.com/work-is-work/</a>
The author of this piece frames this in terms of game theory. I honestly don't follow the line of argument.<p>To me, it's simpler. The people running the organization put the wrong incentives in place. No need to complicate with individuals playing one-shot games and then finding a way to a coordination equilibrium as a solution. In large universities at least, the problem is that most of the people running the outfit start as faculty or low-level administrators, and they move into high-level administration positions without any understanding of how to do their jobs well, or for that matter, what it even means to do their jobs well. Most do not care. As long as the higher paycheck keeps getting deposited in their account every pay cycle, they feel they are doing their jobs.
Didn't read the article but the answer is because of collaboration. As soon as a task requires other teams involved, it becomes really slow and hard to coordinate and understand how something actually works.<p>You gain true expertise only by understanding everything yourself and then you are also extreamly fast at accomplishing your tasks.