Around 2010 I was working at Urban Airship (now just Airship) offering an API for mobile app developers to send push notifications. We put a ton of effort into making sure we didn’t drop pushes and our API was HA. At one point very early on I was restoring pushes out of error emails in mutt (not great, I know, I know).<p>The reason? Not some unrealistic view of ourselves as Google or Amazon, but because we knew we served pushes for a popular pill reminder app (among other reasons).<p>So just because you don’t have Google or Amazons scale, the data you store and process may be as precious to your users as Amazon’s shopping carts are to them. Over 10 years later there is a lot more medication in my life, and I’m really proud of the sometimes-ridiculous lengths we went to to get the pushes delivered.
What I’ve seen more often is software engineers preparing for their next job (or the job they want next) by pushing to build a project with a tool chain that they want to have on their resume. It’s not so much that they want to use X because Google does and it solves their problem, but that they want to look like they know how to use X because then maybe Google will be interested in hiring them—and if not then at least they know that they’re as smart as all those stuck up Googlers anyway.
The paradoxical thing about this for me is that both the most junior and senior engineers fall victim to this line of thinking.<p>I have done so myself, on both ends of the experience/skill spectrum. As a junior it was because the tech was exciting, and the cool kids were doing it so obviously it was the Right Way to do things. As a senior because I'd been a part of the scramble to scale short-sighted systems in a startup that had found product/market fit and didn't want that pain again.<p>Turns out you can build the most scalable thing and people still won't flock to your product. And you wasted all that iteration and learning time.
There's a little too many of his examples I've seen in the real world.<p>We where trying to bid on hosting an application that required Cassandra and Kubernetes. The application was built that way, that was how it currently ran. The customer wanted to move to Azure and try to save a bit on hosting. We reached the conclusion that they could use CosmosDB with the Cassandra API enabled and given that it was just a single container it could actually just run in Azure Websites, no need to Kubernetes. Then we started to dig into what the application did, and how much traffic it received. It was just a few hundred http requests per day, and a stable 2GB of data. This should just have been a tiny Java application and SQLite or MariaDB. We did not win the bid, neither the customer nor the developers seemed please with our conclusions.<p>Kafka is another fan favorite with developers. It's not that Kafka isn't great, it is, but if you are only pushing a 2500 messages per day, it's a bit heavy on infrastructure, maybe just stuff those messages in your database.<p>We've build infrastructure for large national projects (in a small country) and you can easily service an entire population on a single MariaDB on a modern server and few VM for the application.
There's also "You Are Not Microsoft", i.e. you do not have millions of users who would be impacted if this API changed. Sure, if you're at a scale where a few thousand people would be impacted, or at the severity where whoever is affected would be <i>really</i> affected (e.g. medical software), do not break compatibility. But if your product is used by a dozen people and it's not keeping them alive, it's maybe okay if it breaks. Sure, you alienate those dozen people, but you can't keep a company running on a dozen users.<p>And besides, if you have a dozen users, you can literally afford to spend time fixing their specific setups. That's the other flip side. People assume that because Microsoft/Google/etc. provide impersonal, poor customer service, they should provide equally impersonal, poor customer service. Bad customer service is a trade-off you make to have scale. If you don't have scale, you can avoid that trade-off.
I don't think the reason I'm about to discuss is behind all of these stories, but I think it is a persistent force in our industry.<p>I am somewhat convinced that a not-insignificant portion of engineers is under-employed in an intellectual sense. When $BIGCO announces the release of a new tool that supposedly helps with their planet-scale needs, the engineer just lights up at the thought of needing to dig into a new challenge. (The impulse to grow is well and good!) It also helps that places like HN buzz with new tools, it is good resume material, and a vague feeling of not being left behind if said thing takes off.<p>The real fix is to get away from domains/companies that bore you in this way, but it is not an easy realization. There are a ton of interesting problem domains out there! I did this and am much happier with my career.
OK. But... some of us do need to use Cassandra? Some of us do need to use Kafka? Like, more than 5 companies definitely do work that requires that level of scale...<p>Of course bad engineers make bad engineering decisions, and most engineers are bad, but that isn't really interesting to point out imo.<p>> How much data do you have exactly?<p>Petabytes or exabytes at every company I've worked at, none of which are names mentioned in this article.<p>> But have you done the math?<p>Yeah.<p>I mean, I've read this before, so I'm gonna stop here. I get it. Engineers are bad at their jobs so they default to picking the technologies they've heard of instead of the technologies that make sense. But like... working with massive amounts of data is not that rare anymore. It wasn't in 2017, it definitely isn't in 2022.
This is such a great thing to keep in mind. As a junior dev trying to build independent projects for the first time, one of the biggest issues is figuring out what the appropriate level of complexity should be. Building a shed in the backyard doesn't require advanced skyscraper architectural design software. You just want the shed to robust, not flimsy, and for that much simpler tools (a level, a square, a plumb line) work fine. Otherwise you end up never building anything because all your time goes into trying to configure the advanced tools you read about, and it turns into another dead-end mess.<p>It doesn't hurt to know that such advanced tools exist, but I'd think that part of getting an introductory job at any decent firm would involve being trained on how their advanced tools and systems are used in practice.
Some developers underestimate the cost of building slow webapps.<p>For example Magento. Magento is extremely slow. I know companies who pay hundreds of dollars for hosting per month to make it faster.<p>Working with Magento always means thinking about scalability and complex architecture.<p>But in reality webservers are extremely fast, even cheap ones. If Magento was built as a good app you would only start to worry about scalability at the time you had a million dollar webshop.
I interpret the problem a little differently. I don't think the reason people want to copy Google is out of a misguided delusion that their app is going to have the same scale. At least not most of the time. The reason is that software is so bad. Plain and simple. Google is one company whose products seem to be reliable. Most software is a soul-sucking exercise in futility to use them. They error out and do unexpected things randomly. In contrast, Google does a great job in providing a quality experience. People naturally want to know how they do that so well. And to do so at such high scale just adds another layer of impressiveness.
This <i>thinking you're Google</i> is the opposite of <i>technical debt</i>, avoiding doing something and then paying a recurring price for the expediency. This is paying a price up front by doing something which is then never used.
This is a very good piece except for this part..<p><i>Did you know you can buy a terabyte of RAM for around $10,000? Even if you had a billion users, this would give you 1kB of RAM per user to work with</i><p>Not sure if the author is being facile..
People dont do this because they misunderstand they aren’t google.
But because it’s cool and will look good on a resume.
Esp when, you know, apply to Google.
Discussion from 2019<p><a href="https://news.ycombinator.com/item?id=19576092" rel="nofollow">https://news.ycombinator.com/item?id=19576092</a>
The author is 100% correct here, but it's not going to change anything, and I'll tell you why.<p>Companies LOVE requiring all of these sexy over-engineered tools in the requirements for the jobs they post.<p>I've definitely used these overkill solutions in my own work, sometimes because they were the right tool for the job, but many times simply because I knew that one day in the future, I'd be applying for another job, and that job would likely require that I have experience in these technologies. It sucks, but it's the way the world works for now.<p>This has a couple of downstream effects too. If I'm trying to hire, I need to include at least <i>some</i> of those sexy tech stacks in my posting if I want to get good applicants. Why? for the same reason I need to use those things myself even when they're not the best tool for the job.<p>The folks applying to my role know that they're not going to work for me for the rest of their lives and want to been able to check the right boxes when it comes time apply for their next job. Then of course when they get hired in we have to use at least <i>some</i> of these technologies even when we don't totally need them, because after all, it said right on the job posting that we use those technologies.<p>I saw the inverse of this too. There was a team managing a series of small-ish but incredibly important databases. They had these thing running on Microsoft sql server and were doing a beautiful job. There was exactly zero reason to migrate them to something else. But....when the folks in this department went to apply for other jobs inside the company or out, they were greeted with: "You're not using Snowflake??!!", "you're not doing Cloud?!!", "you don't even have an EMR cluster??!!".<p>Not surprisingly, after a while this group had a hard time hiring because folks saw it as a career dead end.<p>TL;DR
people do what they're incentivized to do. Stop incentivizing dumb behaviors.
I didn't know that there were video lectures on youtube by Richard Hamming: <a href="https://www.youtube.com/playlist?list=PL2FF649D0C4407B30" rel="nofollow">https://www.youtube.com/playlist?list=PL2FF649D0C4407B30</a>
From the article - (and I find this attitude rampant!)<p>Software engineers go crazy for the most ridiculous things. We like to think that we’re hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person’s Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.
> MapReduce/Hadoop is a soft target at this point because even the cargo culters have realized that the planes ain’t en route.<p>Side question: what is the state of Hadoop? I remember it had a huge ecosystem, but a lot of it seemed half-baked and is dying, so I'm curious how popular it still is, what other tools people are using for massive data transforms, and which parts of the ecosystem (Pig, Hive, etc.) ended up being popular vs. dead ends.
I like UNPHAT idea, as long as it is executed in an agile way. That is, at a given moment, perform multiple stages that can be done safely without feeling blocked by other stages, then iterate, where in each iteration, one tries to do all possible stages better. For example, after reading papers, don't consider previous stages "DONE", and revisit understanding the problem based on the new understanding, and so on.
I wonder if the title and content ("you can't work the same way big companies work") were inspired by this earlier article from 2014:<p>"No, You Can’t Manufacture That Like Apple Does"
<a href="https://beneinstein.medium.com/no-you-cant-manufacture-that-like-apple-does-93bea02a3bbf" rel="nofollow">https://beneinstein.medium.com/no-you-cant-manufacture-that-...</a>
You may not have google’s scale problems, but even if you’re much smaller than then, if you have a 24/7 web presence you do still have the same browser compatibility, security, accessibility, latency, analytics, monitoring, dependency management, compliance, etc. etc. problems that Google has.<p>Not every piece of technical complexity is designed to solve a scaling problem.
> ... one student’s company had chosen to architect their system around Kafka. This was surprising because, as far as I could tell, their business processed just a few dozen very high value transactions per day.<p>I found this immensely funny as a business case that could (and should) be done with a basic SQL-style database like SQLite.
I can one can even say for those in these larger companies that you are also not the part of the elite core products. You are not on Google search, or AWS cloud, and so on. You are on that crappy Analytics dashboard that’s slow and confusing.
The reason a lot of people chose something out of Google book is not because they get confused about not being Google (which is actually a pretty insulting and also idiotic assumption), but because they wanna benefit from Google putting a stamp of approval on certain tech and pouring resources into it. Tech changes fast, disappears quickly, there is a lot of it out there, most of it is readily available, and so we make use whatever signals we can gather to navigate this space.<p>Which doesn't mean that you can close your eyes and pick whatever tech for whichever use case (duh?) but Amazon or Google doing it differently in the past or a company not being as big as them is certainly not a great argument for not using parts of their stack.
When the people up the dominance hierarchy are impressed with fancy tech, plebs use fancy tech.<p>Maybe call it what it is - a bunch of clueless idiots with money. Fix that, the rest will fix itself.
Off topic! This author combined the folksy 'aint' with the fancy 'en route'. Was it jolting for you too? Not only for the weird combo but for it's awkward meter. :) -- I apologize in advance for this comment.