Not everyone finds BigCorp problems to be the interesting problems. Not everyone wants to live in the Bay Area. There's a lot of jobs in the world that aren't at FAANG, and it's okay if the right job for someone is at a place whose entire dev shop is smaller than the team at Google that maintains Blaze. If you find yourself in a job like that, it is absolutely the right call to invest in the best commodity tools you can (where best is a local, not a global, maxima) and focus on the things you can do that add value to your org.
I have to strongly disagree. When doing business you are solving a business problems. Facebook, Google, Microsoft as business has grown so much they have problems of scale no open tools are able to handle. On the other hand, any kind of smaller company doesn't have those business problems, but a cost of having and maintaining in-house solutions of that scale will just effectively topple and sink them.<p>This is why I'm always frowning during job interviews with engineers telling stories about they migrations from monolith to microservices and putting it all into k8s, while their headcount in the firs tens and their DAU in about 10000 users.<p>Disclaimer: I worked in Yandex, virtually Russian Google. Loads and operations scale there are comparable with second rank international tech giants. Yandex has and have lots of internal tools to handle that load and scale, but I've almost never really <i>needed</i> them in smaller companies. Maybe "nice to have" couple of times.
At one of the places I worked, a relatively large university-hospital, there was an entire Java-UI framework build based on C#s XAML. But it could also experimentally be compiled to html/css/js.<p>Writing any type of UI in Swing is just painful - but this system made it so much nicer. It also had a WYSIWYG editor that you could interop with actually writing the code. Plus you could write either the 'markup language' or the "Java" flavour directly.<p>Making complex responsive UIs was a breeze. It's a shame it did not get open sourced - but it was tied into a great amount of internal tooling. (The entire codebase was 28+ million lines of code when I left).<p>It's one of the few non-open source libraries I've worked with that really impressed me.
Doesn't it take months to train new developers in these mystical internal tools? I remember hearing specifically that Google expects 5+ months of training until a developer is productive. How many companies can afford to spend ~50k+/hire on on-boarding ((100k/yr * 5 months) + benefits)? What is the opportunity cost of holding so many people in training mode for this time period?<p>Different companies have different problems. Big companies need big expensive tools to help with their problems. Little companies usually get by with smaller more portable tools. Let's not use this to demean people at small companies with some false sense of superiority please...
I'd miss the engineering machine shop. Those facilities only get developed by accident, for instance if a company gets rid of its professional machinist but keeps the equipment, or something like that. The machines and collections of tooling take a long time to curate. Any modern manager would refuse to allow that much stuff to be purchased if they were asked, and believe in the magic of outsourcing everything.
This whole premise falls completely flat if you've only ever worked at small and/or young companies:<p>> but which didn't exist on the outside world.<p>> You're not allowed to include things you or your friends built<p>If you work for a service provider, you probably have zero to not much internal tooling, because you're usually solving the customer's problems, not your company's. When I did, we open sourced everything cool we built - sometimes tried and true defaults, or sensible Vagrantboxes or something like that. (Today that would be Dockerfiles, I guess) - and I consider myself lucky to have worked at a smaller company that was close to the FOSS scene, and not something like Accenture. Companies like ThoughtWorks might be an outlier here.<p>You're unlikely to build something so immensely cool in your first 1-2 years in a startup that it will change your live (as a developer) completely. (This is not about your company's product, but internal developer tooling, so if your startup does developer tools as a product, you're the 1% where this would apply).<p>And finally, if your startup grows.. you might probably still not be the next Google or Facebook where the internal team providing nice things to your fleet of developers is bigger than some companies. Internal tooling will often be seen more of a cost factor than a tool for productivity enhancement. And often it's actually the case. In my current company we just have so many software development teams creating parts of our products.. so many languages (with good reason) - the only internal things I could think about that would even make sense for all of them would be some code review tool, or anything targeting source control or maybe CI. Anything else would have to be really workflow-centric and language-agnostic. Anything regarding the mentioned borg/k8s/BigTable would be completely useless, for example.<p>That said, at my last company, I think we created some pretty cool things that made a lot of things a lot easier for the team. But we were two small teams, so point to "we built it ourselves, for ourselves" applies. Also it probably wasn't so revolutionary in the greater sense, but everything made sense for that company and I'm often still sad if I have to solve some similar problem these days and don't have my old setup (more than a specific tool).
This really resonates with me. I have seen the “buy before you build” attitude turn into “buy because we can’t build”. I know correlation is not causation and there are always plenty of other things going on... but it sure seems like the pride in our work dropped off and we were on the wrong side of every tipping point.
Couldn’t it be argued that all software is built with more “glue” than non-glue? That’s just the nature of the beast. If a user wants to, say, view a record from a remote database in her web browser, how many separate layers of code does that involve? (Several.) How many of them aren’t “glue”? (I dunno, like two?)<p>Perhaps it’s true that most glue is poorly written, but doesn’t that imply that writing good glue is, in fact, difficult, and therefore it’s as hard — or maybe harder - than building “real” things?
I don't know. Having an "80% solution" but not needing to support or fix it is not such a bad price to pay. Also this seems to be a bit skewed towards large networked solutions.
Wanted to see what Facebook Scuba was, and found this comment thread that really highlights the point of this blog. <a href="https://news.ycombinator.com/item?id=13463016" rel="nofollow">https://news.ycombinator.com/item?id=13463016</a>
One problem that I've seen in one such large company, and heard of from some people in others (FAANGs), is that there's no incentive to maintain these internal tools. Sure, they often start out as sort of amazing and there's a ship email and for 1-3 years a strong team, but you are unlikely to be promoted a lot polishing UI, fixing bugs and adding features or even support for new backend systems... so, eventually there's a skeleton crew maintaining an old tool, everybody complains how it's slow and sucks and behind the times, and then a brand new replacement is built that is going to fix everything. There's a ship email, people gradually switch to it, rinse, repeat.
ime, for many companies, standard, off-the-shelf tooling that's a mix of OSS and paid services works great. And one benefit is that when you leave, you can bring whatever tools you loved into the next company.<p>The reason Facebook and Google built their own is that they're at such a huge scale both in terms of usage and number of engineers, that none of the off-the-shelf things work for them. For most companies that's not the case.
Nothing wrong with writing glue, if that’s what the business / company / market needs.<p>Having said that, I do miss the data that’s available at FAANG companies (at least ones I’ve been at). I could dig up all sorts of system and business data, and really get good insight into both what my application and the users are doing. Tends to lead to a lot better and more clear decisions and thus helps to build the right thing.
I don't agree without reservations but like how against the grain this currently is. Outsourcing everything does seem like a sign of immediate hyperoptimization and missing building a more lasting engineering value. Do as little as possible to extract <i>some</i> margin. There is business wisdom in this but indeed such companies would seem more brittle. Sometimes it's like there's an ecosystem of plumbing companies selling to other plumbing companies without much visible external cash intake.<p>That being said, I think most of the things she mentions are not customer facing but still pretty differentiating to the businesses.<p>Another perspective is that the first crop of truly successful Internet companies were in the position to solve the big backend problems with running big Internet companies in-house. Nowadays getting to the scale while building some fundamental in-house stuff makes less economic sense (you'd lose to someone who doesn't). It's probably better to have some unique technology but more closely related to the specific business.
For 99% of engineering organizations, internal engineering tooling is not your source of competitive advantage. Your advantage in the market comes from understanding the needs of your target user better than anyone else, and building a product that meets those needs.<p>For every piece of infrastructure or developer tooling you need, you have to make a buy vs. build decision based on cost/benefit analysis. Your cost to "buy" needs to include the upfront cost of developer time to configure the software for your needs, and the ongoing support cost to make sure it continues to work. When you include this, even open source software isn't "free", although it may be cheap. Your cost to "build" includes the upfront engineering time to design & code, ongoing support time to keep it working, <i>and</i> opportunity costs incurred because until the system is actually ready, no one can use it, and they have to either meet their needs in some other way or simply block until the tool is complete.<p>Very large engineering organizations like Facebook, Netflix, etc. can amortize the cost of tool and infrastructure development over thousands of developers and an enormous infrastructure budget. If a tool would make most of Facebook's developer's 3% more efficient, than it's worth staffing a team to build that tool. If better caching infrastructure could reduce Netflix's network usage by 3%, then it's worth staffing a team to build that infrastructure.<p>If you're a small- or medium-sized business, then that math doesn't pencil out. A 3% (or even 10%) developer efficiency improvement doesn't even pay for the opportunity cost of one developer. A 10% decrease in a single dimension of your AWS bill probably doesn't pay for the developer time either.<p>If you're at a mega-corp with a huge engineering staff: Yes, absolutely invest in tools that make your engineers more productive, especially if you've identified truly unique needs that can't be met by off-the-shelf software.<p>If you're at a smaller business: Focus your developer energy on your user-facing problems, and almost always prefer to "buy" instead of "build" (where "buy" includes using open source software).
<i>If you can't come up with anything, it's possible you're just beyond repair and too snarky to think reasonably</i><p>Or perhaps that we've mostly worked in crap dev jobs/companies? There are easily enough of them around to encompass a career.
This article makes little sense. "Engineering" by definition is some sort of "glue" work that reuses existing libraries/wheels and builds solutions out of them to address specific problems. I as an engineer am proud of this glue work that I do, just like the carpenters, plumbers, cathedral builders, doctors etc. If you really want to be <i>unique</i> and invent something, you should be in the academia and are probably in the wrong place to begin with.<p>I have a friend who worked in the TensorFlow team and still quit to pursue a PhD. In his words "all engineering jobs are boring" from his perspective. I don't think being in a BigCorp or not makes much difference at all.
I've recently branched out on my own. Never had great tooling, was in government or a F50 where chains of approvals for the slightest changes were cursed by all, including those doing the approving. But, there are some organisational things I miss:<p>Having someone/a team to ensure network uptime is fantastic and that have the scale and experience to anticipate and mitigate potential trouble before it comes up. Likewise all infra in general. Having a legal department to ask "Can I do this?" where often the answer may be "We don't know" but that's still an answer. A security force, property, accounting, providing peace of mind. Someone making sure the projector's setup, tiny things that add up quickly.<p>There are downsides to this, such 'organisation' often grinds things to a glacial pace, hence why I'm breaking out by myself now, but... not tooling, I can take care of that myself and I'm incredibly thankful for the SaaS ecosystem that would have made this impossible even a decade ago, or at least very inconvenient with 3rd party consultants providing such services for high fees and little more. There are a lot of things an organisation provides that we can take for granted, and having that experience provides a great bonus in expectations of robustness not only in technology but in all processes of a business. Tooling is, in the end, in the mind.
I'm always so disappointed that we're all so closed when it comes to tooling. It's fortunate that some tools get open-sourced, but it's also astounding the amount of time and money that is wasted building the same tools over and over at each company.<p>Sure, sometimes paying for a hosted solution for your CI, deployments, container platform, data pipeline, etc. isn't the most ideal route, depending on your business, but I'd much rather focus on building actual product over reinventing the deployment system wheel for the Nth time.<p>> <i>People from Google probably miss Borg (and lament the whole Kubernetes thing) or BigTable.</i><p>I've never worked at Google, and thus have no experience with Borg, but Kubernetes is <i>exactly</i> what I want to see companies open-sourcing, or, if they must, selling. I don't see what there is to lament. Being able to start a new company and not have to reinvent and rebuild your own deployment/container platform is a <i>huge</i> gift.<p>For the most part I find this blog to be insightful and fun to read, but this post IMO completely misses the mark. Given that I think the author works a lot on the operations side of the house, I guess I'm not surprised. Internal platform tools are there to serve the needs of the product teams; they don't exist just to make someone feel good about their ability to build something instead of buying instead, which she unfairly looks down on so disdainfully, as if it's not <i>real</i> engineering. Part of doing engineering is putting your ego aside when it's more cost-effective and robust to use someone else's solution instead of rolling your own.
The only thing I'd miss from a larger company is the travel budget. The 'devil may care' attitude to drop a couple grand on a short window flight... compared to travel with the small company where that meeting could break the company. Double goes for the in-house travel agent who sorted everything while in route - Jane... you were missed.
I prefer Kubernetes and Prometheus to Borg and Borgmon. Better interoperability, portability, and documentation. Does any other Xoogler have thoughts on this? The author didn't mention it, but I also prefer Capn to Proto3.
Every time I make a neat tool I open source it using the unlicense so I can use it elsewhere with no attribution. I'm happy to work at a company where this is possible. This doesn't apply to business critical code a competitor might want, just libraries and tools for making generic 'stuff'. Typically, I make the tools at home first and decide I want to use them at work, but I eventually end up patching them at work for company purposes.
I solved this problem 10 years ago by building my own open-source stack and only working with that stack: <a href="https://github.com/tinspin" rel="nofollow">https://github.com/tinspin</a><p>You should build your own tools, that you bring with you, that is what all artisans have done for all time.<p>No reason to change that just because you can use the newest buss-word every week.
The only thing I miss are teammates and some of the office moments we had together, but those luckily survive employers and we get to meet in other contexts.<p>I never been attached to work tools, or technology fads, things are ephemeral, best practices of today are the legacy codebase that no one wants to work on tomorrow.
This article is dangerous in romanticizing the “not invented here” culture at many big tech companies and seems rooted more in the 90s than present day.<p>The world of open source tooling and easily re-usable SAAS offerings means everyone has access to the best tools, whether you’re a small startup or a big company.<p>Anyone who longs for internal, corporate tooling baffles me when they can use things that actually have polish, user experience and likely better implementations under the hood.<p>Companies should spend their time/energy building things unique to their problem domain, not weak also-ran corporate tooling.
If you’re not building complicated new systems and just gluing things together, you’re not doing engineering? 1) definitely gate keeping 2) good engineering is about what not to build as much as it is what to build.
This topic is one of the primary reasons I enjoy working at MS these days.<p>The tech stack is full of OSS or industry standard tools, not a bunch of internal NIH things.<p>My current stack is:
React, Node, Kubernetes, Linux (for dev and deploy), Git for sc, GitHub, vs code.<p>I can’t even think of some internal tool that I depend on at the moment.