I feel this entire post.<p>My career record (US Navy) for cost savings was something over $50MM. Every time I did something I had to do PowerPoints, present to Flag Officers, etc. Hell, once I almost got hugely punished because I didn't let my boss take the credit (he had no desire to because he had zero idea what it was I even did).<p>Note that some of that was as a Lean Six-Sigma Black Belt doing Enterprise projects (I hate every single bit of terminology in that entire godforsaken sentence), where it was literally my JOB to save the DOD as much money as possible with the least disruption possible. Those were the absolute worst years of my career. That period of my career was my reward for just going rogue and fixing things that saved millions.<p>I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.
This reminds me of some of Dan Luu's stories, <a href="https://danluu.com/nothing-works/" rel="nofollow noreferrer">https://danluu.com/nothing-works/</a><p><i>Likewise with chip software tooling; despite it being standard to outsource tooling to large EDA vendors, we got a lot of mileage out using our own custom tools, generally created or maintained by one person, e.g., while I was there, most simulator cycles were run on a custom simulator that was maintained by one person, which saved millions a year in simulator costs (standard pricing for a simulator at the time was a few thousand dollars per license per year and we had a farm of about a thousand simulation machines). You might think that, if a single person can create or maintain a tool that's worth millions of dollars a year to the company, our competitors would do the same thing, just like you might think that if you can ship faster and at a lower cost by hiring a person who knows how to crack a wafer open, our competitors would do that, but they mostly didn't.</i>
The whole post is gold.<p>- managers asked how it was possible we saved that much without help from them<p>- asked to prepare slides<p>- asked many times on how it happened<p>- had to roll it out slowly to make it look like they did it over time incrementally vs one small toggle<p>- asked for a raise due to impact and did not happen.<p>Sir, for your sake, apply to a FAANG or something, you'll be at least taken care of better.<p>Also please implement Twitter card metadata in your blog so it looks better on twitter :)
You didn't accidentally save half a million, you deliberately and intentionally saved them half a million, but now you regret it. That's not the same thing.<p>Large organisations are so woefully inefficient that I'm surprised they're able to compete at all, but they have a ton of money and economy of scale and all that, and along the way there's more than enough money to waste millions on stupid nonsense and inefficiency and nobody really cares.
Once upon a time I uncovered a bug that recovered $4MM/year in revenue. It was swept under the rug to protect the team and executives that let the blunder continue for as long as it did. I didn't get a raise, but I made some allies and got to coast for a while.
This kind of thing has been going on forever. Once upon a time, I walked into a room full of VT100 terminals, and bored CS students waiting for their compiles to finish. (It was a weekend near the end of term) I took a look at the system, and realized they had pushed all the compiles into a batch queue, but that queue defaulted to BELOW interactive priority, so any keystroke anywhere had higher priority... so all the people checking their position of the compile job in the queue, slowed it down even more.<p>Over the next 15 minutes, I kept bumping the priority of the top job in the queue up, an hour later everyone had their work done, and went home. I had the room to myself. Rogue sysadmin for the win. ;-)
I saved my company $1 million a year a couple of months ago by noticing that there was an S3 bucket that kept growing and costing 80k a month.<p>I poked around and realized that there was a system that we weren't using anymore that was copying files to the bucket I reached out to the stakeholders and they turned it off and we deleted the files.<p>The higher-ups didn't seem to really care, My boss's boss told me to reach out to another team that should've caught this and that was about it.
"For example, you've run 234,745 INSERTs into [table name deleted]_HOURLY and ALL of them were under 1000 rows."<p>That's an actual quote from our org's snowflake support slack, about pretty much the same problem: lighting up the cluster with a bunch of tiny transactions spread out in time. The product is not made for common use cases like trickle loading.<p>I'm an actual experienced data warehouse admin, so I'm watching them rediscover my job description one cost overrun at a time. They unironically say things like "It's self-managing, but you need to monitor costs and rewrite loads and queries to run more efficiently". I'm afraid to ask them what they think I do all day.
Long time ago in a galaxy far far away, I replaced the need for N * (Oracle license + Sun server) with a simple perl script (< 100 lines) and one Sun server for a total cost saving in $300M+ / year. While doing it, I also invented map-reduce (it was before Google time).<p>The problem was to calculate bunch of stats from web servers logs (e.g. 10 most popular pages). The original solution was loading it all into Oracle database running on multiple servers since logs were huge. And then running bunch of SQL queries. Rinse and repeat daily.
The author's gallery of poignant HN commentary on their writing is <i>chef's kiss</i>: <a href="https://ludic.mataroa.blog/compliments/" rel="nofollow noreferrer">https://ludic.mataroa.blog/compliments/</a>
This is such a good article. My skin was crawling.<p>What really scared me is that I couldn't identify any of the issues raised in my own organization, even though we often run into similar, smaller-magnitude problems caused by a blindness to obvious mistakes. It makes me fear I, too, am blind to massive bleeding wounds. Here's hoping they actually don't exist.
This reminds me of the company I worked at that was paying for dozens of unused Redshift clusters at any one time, peaking over a hundred. There were common data science (read: ETL configuration) tasks that required spinning up a Redshift cluster, and the "data engineers" (read: fresh grads with a few weeks of training) sometimes forgot to shut them down afterwards, or got diverted to different work temporarily and couldn't find the original cluster when they returned to complete the task.<p>The first proposed solution was to ask the forgetful data engineers to manually tag all the resources they created, so un-tagged resources could be deleted promptly. When we asked how the tagging scheme would allow us to distinguish between clusters that had been forgotten and clusters that were being actively used, we were told that it was not necessarily a complete solution, but it was "a step in the right direction." Ultimately the problem was declared solved, but I wonder if it really was.<p>The company was swimming in VC money, but even so, that level of waste (which was consistent across all of our astronomical AWS spend) was impossible to ignore. Checking back now, all of the engineering jobs they're hiring for are overseas, so they must have ultimately nuked the entire department.
><i>I am asked to write some PowerPoints, which include phrases like "a careful statistical analysis of user usage patterns indicated an opportunity to more effectively allocate resources", implying that nothing was wrong, we just needed to collect more data before deciding not to let the expensive machines idle all day.</i><p>Yep.<p>If you don't make them realize it was a Hard Problem™ that was only solved by their smart hiring, funding, and task-deciding, you might shatter their whole world view.
I have to play devils advocate here because for every one of these cases, there is probably a dozen of similar stories where the ambitious new guy actually did nuke the system with a risky friday release and then logged off for the weekend xD
At $large-scale company this happens pretty regularly if you just look for expensive things and ask:<p>* "Can I just remove or slowly deprecate this?" I.e this job generates data no one uses, or alerts people just bin-bucket or has been already replaced by faster/ better.
* "Can I cache this?"
* "Can I run this at a lower-priority/off-peak/less available?"
* "Can I reduce the frequency or move processing to deltas?"<p>This is just the easy stuff you can usually do in a few lines of change without even getting into basic optimization rewrites. Most internal things at $large-co are built because someone thought they might be useful, they might get promoted and using that hypothesis and moved on, but few things are actually continuously validated as still generating value > their cost.
I love this for purely self-appeasing reasons.<p><pre><code> this machine kills imposter syndrome
</code></pre>
or at least it helps. Having a background in solid CS theory from High School, and having a degree in Art, I find it very hard to apply for engineering roles, and my mixed bag of experience often lands me in Support Engineer / application admin / integration roles, fuming tremendously when the people with SwEng / Developer titles fumble on with implementing some JavaScript change for a feature I need in Service Now for my application's customers.<p>It is incredibly reassuring that it is not simply my organization that is hamstrung by the pretense of complexity, when really someone just made it complicated to make it seem important.
And this is the difference between competence and incompetence in tech. We can argue about whether 10x developers exist, but one thing I hope we can agree on is this: Certain problems can't be solved no matter how many incompetent people we throw at the problem.
In my company we worked with a platform launched on local machines that had an admin console where you could execute java code. Pair that with almost everyone not turning on the firewall and all engineers connected to the same wifi network, anyone could do whatever they wanted.<p>I showed a demo how easy it is to read private ssh keys to the head of infrastructure, and after some months people could connect to network only using custom credentials (ldap) which was good, but also asked us to install "spyware" that among other things checked the firewall. I never installed the "spyware" but nobody pushed me. I didn't think I somehow prevented a disaster or did some heroic deed because everyone in the company was professional and nobody would exploit this. But of course I didn't tell about this to anyone except the infra because such information should not be disclosed until is fixed. And once is fixed why disclose it?<p>I really miss the Mac checkbox to enable the firewall. On linux I use nftables which is really powerful, but with so many possibilities it is easy to miss something during configuration.<p>I observed a lot of senior engineers don't have sufficient network knowledge. A lot of people on linux don't use the firewall which is really bad if you work on shared wifi.<p>Also when running docker images, if you map a port when using docker run (ex. docker run -p 80:80), docker will automatically add firewall rules and bypass the enabled firewall, exposing that port publicly.
Tried to save my company 100k per month by cutting down idle compute, clearing out petabytes of unused buckets, and using on-demand compute in testing envs. This resuted in me getting pummelled by directors asking questions, staff engineers saying random what-ifs, until I eventually gave up.<p>I used to think startups would be one of the few places that actually gave a shit about being lean and efficient - but turns out that's only true if they're bootstrapped.
I did something similar.<p>Success and super efficiency was rewarded with additional work, including hinting I should help other departments (apparently a joke).<p>I received no extra remuneration despite asking, yet my company continues to hire new staff weekly.<p>I've learnt my lesson, just like this author.
"My team has spun this as a huge cost saving, when really we just applied a fire extinguisher to the pile of money that we had set alight."<p>I love this so much
This is the first time that I saw this blog and I've now read a bunch of those posts.
There is so much gold there, it's incredible.
So much is going wrong in all those companies, that it is just frustrating to be working in those.<p>A colleague of mine and me got the opportunity a while back to basically work as wanted to in one side project.
So we just worked truly agile, without any of the "Scrum" bs around it.
Everyone involved was blown away how successful it was in terms of implementation.
Two companies tried it before in the span of a year, and they didn't get it working at all.
We worked just some evenings and weekends outside of our full-time job over the span of six weeks, and we got 90% done, with everything essential included to be able to start working with it.
We then implemented everything else the customer wanted within the next weeks by following the same principles.
There were no problems at all (apart from the customer basically almost running out of money because they spent most of on the previous companies which couldn't deliver).<p>At my day job, the managers usually argue against anything we want to do, question every decision from our side, want to have regular big meetings with 16+ people to talk about everything at length until no one wants to do it anymore, and then micro-manage everyone to death.
We tried to at least keep these guys out of the daily by pointing to the Scrum guide where it says that only people working on tickets should participate, so one of these guys just created a bs task for himself, then talked every day about it for a lengthy amount of time without providing any real value at all.<p>I never got those silly shows before, which play in offices where everyone just acts silly and they never really work.
But now I know why they are so popular. They just provide comic relief for people who really have to work in those environments and it is really as bad as in the television shows, if not worse.
It costs a million dollars to run a service that essentially “uploads a 2KB CSV to a database” on Snowflake?<p>And you can cut this in half by changing some defaults in instance lifetime?<p>I’m starting to understand why Snowflake’s market cap is something like $50B. This sounds like a nice money-printing business if you can convince enterprises to use it.
I know of an intern (with a master mentor) who did a one char change and saved our company many millions - the change was to update the threshold of data size of when to not compress small messages - this threshold was updated after many many years - old threshold was based on old compute and network costs.
> I'm not sure what the original estimate was, but I think it was intended to cost something like 200K for a year of operations, but we were now close to a million dollars.<p>> ...<p>> I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.<p>Wow, so they're still over budget by 300k dollars. This is a funny story, but the company sounds incompetent.
Great post on why LLMs will have Management Consultants looking for their future. Average skill is highly profitable. One skilled senior and 20 juniors at an affordable rate is always most profitable.<p>Developers - Always do this as bonus paid by contingency of saved money and a signed off scope if it works out, especially if you’re d looking into it on your own time.<p>Specifically if you like you can get signatures from everyone on the hierarchy on how much money or time this will save, cost or make them.<p>Do it enough times and the right kind of CEp/President will tell you to stop bringing the business case to them for approval and just do them if they make sense and you have your backup.<p>You will enter a side door, bypassing most politics and c-levels (and maybe triggering some new), reserved for people who say let me see what’s possible instead of coming back with reasons why it can’t be done.<p>From there, paint a picture of what if you built a team of only doers, minus talkers across the organization.<p>:)<p>Tech should never report into Finance, the group that can’t even tame spreadsheets.<p>Yes, business owners pay.
I'm sad to read that.<p>I can only testify that this is not my experience at all, so it might be that you're in the wrong place.<p>In every company I've joined, I've found issues, often large ones, managed to get assigned to them, and had fun fixing them (or at least the satisfaction of seeing them fixed). Then moved on to the next issue.<p>I haven't always received much recognition from the powers that be (although me and a few colleagues have once been wined and dined at one of the most expensive restaurants in the city for having saved the company 1M$+ with a few days of work each, that was nice), but I've gained respect from fellow engineers and from myself, which was nice, plus lots of things to tell during job interviews.<p>So, may I suggest getting out while you can and finding a better place to work?
I saved my company $50,000/year. A previous consultant (that they were still paying monthly) had written an application that was the backbone of the entire ordering system. If it went down, no work could be done and they needed to call this guy and pay him $150/hour.<p>It was a .net app that I reverse-engineered that was basically calling 10 or so encrypted stored procedures in SQL server. Because the older versions of SQL don't actually use encryption, but a type of encoding, I was able to decode them all and replace the entire app with a script that I wrote.<p>The whole process with testing took about a month (they thought it would take me a year).<p>The consultant that made it wasn't very happy when he was told they would no longer be needing his services.
The amount of time and money wasted on incompetent people who shouldn't be running or working on things is outstanding.<p>Fire the majority of people leaving the A players, unmanaged, and the world will begin to work near-flawlessly in a few weeks to months.<p>But this line of thinking hurts feelings which are (presumably) paramount to actually solving the problems people set out to solve.
Huh, I wonder which of the "big-4" advised them on their practices?<p>Am willing to bet it was 'KTMJ'... (Batman reference, but... strangely similar to one of the "big-4")
I had a friend in a similar situation, he explained the issue and how he would fix it.<p>I was baffled and urged him to be careful and assume he was wrong that it had to be wrong.<p>Anyways, he is still fixing it, getting some people to validate it.<p>I would think he will save at least hundreds of thousands a year.<p>But seriously, why a company that spends millions of dollars in an area would not hire an expert to try to save some money in it is beyond me.
This post hit right something I dissected in my last role.<p>In short, I was revamping/enhancing a property insurance policy pricing system as the lead from the business side (i.e. portfolio manager/actuary).<p>However, much of the development team was truly incapable of writing software in any useful way, so I inevitably dipped into C# and SQL to help diagnose issues as we (as a team) worked on the enhancements.<p>My business coworker and I found something that remains funny to this day: code littered with references or attempts to use the .NET TPL library to make the platform do more things "in parallel," but without actual knowledge of using such a library in practice.<p>Nothing worked in parallel! There were blockers all over the place! As someone else said, TPL might well have been a //TPL TO DO.<p>We suspected that TPL was there as one of those "look what I can do" intrusions that received praise from more senior devs/mgmt. even if it never actually did what it was supposed to do.
At a small company, I found a bug in the sales system that was giving away $10k per month by miscalculating customer's tax. CEO of the company essentially pretended that he couldn't hear anything when he was told.
The unsaid part is that this kind of situation is everywhere. It only really hits home when (IMO) when you’ve had a couple of decades in the industry. We have totally squandered the abundance of resources in many sites/situations/use-cases. Whether it be IOPS or threads or memory or network bandwidth and latency. Yes premature optimisation is the root of all evil, but the other side of that coin is pretty ugly looking too. Not even because of monetary waste in many cases, but because of unnecessary complexity and fragility. For what? A slightly higher level of abstraction? A bit more interoperability? A marginal gain in some other metric?
I could fully relate with this post. I personally share a similar mindset: someone who can take criticism with pleasure, who has sorted out thoughts about how the industry operates, who has not subscribed to the belief that people in leadership positions are world apart from us, and who can openly express that wrong is wrong.<p>I think many of the people who disagree have either been serving in non-engineering roles since past couple of years, or have bought in the pitch of rosy world created by managers and so-called leaders.
TLDR: They changed the idle time settings for running queries in their Snowflake database. Originally, they were set to idle for 10 minutes after every query, but most queries only took about 2 seconds to run.
<i>what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems</i><p>Well duh...of course you'd want that. But it is not so easy is it? I'd also instantly fire half my staff and hire hard working geniuses instead. But that just isn't an option, unfortunately.
Like others said, I absolutely feel this post. I don’t think I can say any other post I’ve ever read feels this relatable. Every one of his posts is both accurate and a depressing assessment of the overwhelming incompetence in this industry.<p>> I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened.<p>I know he doesn’t work the same place I did but damn, it sure feels like it.
Sounds like something out of a Gene Kim book.<p>Definitely a pattern through the places I've worked. The ones that are smart, have good intentions and go a little bit rogue tend to make the biggest impact.<p>Executives are too concerned with risk, and honestly, it playing out this way suits them better: if someone goes rogue and it works, they can claim the win, if it doesn't, they can blame and reprimand.
Apart from the apparent topic: If the architecture of the software is a mess like he describes I would bail or ask to make significant changes (and threaten to leave if I wasn't allowed to). I would definitely not just try to optimize using settings.
Line manager here.<p>When I find someone very talented, I go out of my way to promote them because good engineers who want to stick around and not jump when the times are good is hard.<p>The sad reality is that I’m limited to how much I can bump their pay, give them time
off, etc.
Indeed, under a pay-as-you-go model, if there's a lack of precise control over the warehouse, such as a 10-minute suspension, it could lead to significant waste. This is because most queries might only take a few seconds, and the rest of the time is wasted. If you find Snowflake expensive, consider Databend. It's an open-source, cost-efficient alternative to Snowflake, and it maintains a consistent product experience with Snowflake.<p>Open-source: <a href="https://github.com/datafuselabs/databend">https://github.com/datafuselabs/databend</a>
Databend vs. Snowflake: <a href="https://github.com/datafuselabs/databend/issues/13059">https://github.com/datafuselabs/databend/issues/13059</a>
Cloud: <a href="https://app.databend.com/" rel="nofollow noreferrer">https://app.databend.com/</a>
I think the most charitable interpretation of this phenomenon is that <i>bureaucracy is hell</i> and the entire organization tends to get put on a leash pegged to the level of incompetence of whomever is in charge of #thing.<p>(Peter Principle writ large.)
> I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K.<p>Only one thing left to do. Leave 'sleep(600)' calls all over the place and resign.
It's good to know there are worse places than mine. some things are reminiscent of what I've seen, but nothing so extreme as even half of what's described.
Whenever this happens, I really have to wonder about all of the people I call "good" on my team. Like surely someone gave a shit enough to know this is how it works, right? ...right?
Repost of <a href="https://news.ycombinator.com/item?id=38064086">https://news.ycombinator.com/item?id=38064086</a>, should probably be merged
I'm on the tail end of my career (hopefully), so I can share some unsolicited advice for younger workers.<p>1. Remember who you work for. Yourself. Not your boss, not your grandpa who always wanted you to become a writer. You only have so many breaths before you get your ticket punched, so make them count.<p>2. Don't seek work as something you should love. Those should be your loved ones, and hobbies etc that stimulate you. Sure there's a small minority who get to do what they love, but over time love can turn to disgust.<p>3. When starting a job, figure out the organization's incentives. Some orgs want change, some will fight any change. Figure out what the org wants and your life will be much easier. My job is like the Maytag repairman, kind of waiting for stuff to break. I refer to it as being a digital janitor.<p>4. Nothing you build digitally will last. Hell, between linkrot, bitrot, and the heat death of the universe, nothing lasts. So don't expect what you've built to last, or to hold value. It'll all be re-factored/re-engineered/re-architected and become obsolete.<p>5. Large organizations are toxic and often sociopathic. So are small orgs. If you can find a way to start your own business, try to avoid becoming that way. Good luck, most businesses either fail or become that way. But at least you have more choice in the matter.<p>6. Find a mentor (sometimes referred to as a rabbi) who can provide guidance to you when times get tough. Don't choose a rabbi from your reporting chain. You will find yourself adopting a worldview that might not be aligned to your own interests (in other words, the bastards will manipulate you). This rabbi can be a friend, or just someone you vibe with. Be careful in choosing your rabbi, and take care of this relationship.<p>7. Illegitmi Non Carborundum
> I saved my company half a million dollars in about five minutes. This is more money than I've made for my employers over the course of my entire career because this industry is a sham. I clicked about five buttons.<p>I'm sorry, but this needs a privilege/gratitude check. You are guaranteed your salary, and you're welcome to take on the same level of risk your company is by starting your own. If you think it's so easy go ahead.
one of my employees saved an unnamed tech company 250M in regulatory fines. They were flagged as top performer and subsequently laid off in the first wave of the tech layoffs … probably because stock compensation clawback made it very juicy.
> I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K.<p>Pain I feel very acutely. As an employee, I saved a FAANG company billions of dollars in potential GDPR fines after they carelessly declared 'mission accomplished', weeks before the deadline, and never even got a 'thank you', much less a raise.
Reason number 543895 why I like small company life:<p>When you find shit like this, people will go "my bad" at the worst, and you all are happy the company has more money. Small teams and small firms have their downside, but honesty and transparency, especially when it comes to cost savings... tend not to be one of them.
if you want to learn more about consulting firms watch John Oliver
<a href="https://www.theguardian.com/tv-and-radio/2023/oct/23/john-oliver-mckinsey-consulting" rel="nofollow noreferrer">https://www.theguardian.com/tv-and-radio/2023/oct/23/john-ol...</a>
> <i>The entire thing is stitched together by spreadsheets that are parsed by Python, dropped into S3, parsed by Lambdas into more S3, the S3 files are picked up by MongoDB, then MongoDB records are passed by another Lambda into S3, the S3 files are pulled into Snowflake via Snowpipe, the new Snowflake data is pivoted by a Javascript stored procedure into a relational format... and that's how you edit someone's database access. That whole process is to upload like a 2KB CSV to a database that has people's database roles in it.</i><p>Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.
"this whole department, like many departments, is some sort of weird political PsyOp to get executives promoted. It's cosplaying as a real business and the board thinks the costume is convincing."<p>Came for the engineering, stayed for the blisteringly on-point observations of corporate life