DevOps is broken

I’ve long held this opinion but I consistently get drowned out.<p>DevOps has different meaning depending on who you’re talking to, even some definitions that <i>appear similar</i> are different in nuanced but important ways.<p>All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be.<p>There is also a lot of rewriting of history that gets thrown in, similar to how when people talk about cloud then the <i>only</i> alternative is to start making CPUs by hand and begin building your own nuclear reactors. It’s the <i>idea</i> of what came before, not the reality, that people seem to be defensive of.<p>It’s honestly exhausting to discuss.<p>So instead I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.<p>Roles so simple that you already have a clue what they do, which is sort of the point of job titles.

I have bewilderingly tried to discern why software development continues to grow more and more complex.<p>It wasn’t always like this. There was a time when we talked about languages and OSes and libraries as if they made a difference on how much you could get done with as little people and cognitive load as possible (the claims were very much overrated, but the point was we acted like it mattered).<p>And then it started ballooning. It seems to me that much of that coincides with a lot of new money being dumped into the economy, and software moving past the necessary-evil-so-how-do-I-drive-down-my-costs and on to the gold rush of you must have a web app for your service that rivals a video game in complexity and visual finesse. It seems to me that as long as their is so much free money sloshing around venture capital funding, that it was inevitable that the process of making software would complexify to soak up the extra cash. After all, you can only add so many levels and varieties of managers to add value. After that comes the variegation in roles of software development.<p>That’s my working theory anyway.

This echoes the philosophy of Google Site Reliability Engineering, which (this is key) is an engineering discipline.<p>The job of DevOps is not to close tickets. That&#x27;d be like driving a car by shouting directions at someone lying on the floorboards holding a wrench to the steering pinion.<p>The job of DevOps is to build a steering wheel (and ideally, teach SWEs how to drive... at least enough that they understand what a &quot;road&quot; is and why it&#x27;s a pleasant experience for everyone if you stay on it. If the road doesn&#x27;t go where they need to be, <i>then</i> it&#x27;s time to file a ticket, but that ticket had better be &quot;Build a new road,&quot; not &quot;Offroad this one car to the cabin in the woods and call it a job well done&quot;).<p>The raw hardware of an enterprise deployment is so flexible it solves <i>nobody&#x27;s</i> problem. DevOps is in the business of writing the operating system for a mega-computer physically represented by hundreds to possibly millions of heterogeneous computers. It&#x27;s a process of continuous growth to make that work.

&gt; If the “DevOps” team ships a Postgres RDS instance it will run fine forever, that is until an application starts using it. All of a sudden a cascade of N+1s hit, the CPU spikes, and queries grind to a halt. Who is woken up? And why does this always happen at 2 AM? In this scenario, there is nothing for operations personnel to do, yet here they are.<p>This is definitely a symptom of a broken model and not what I would call devops. IMO the most important tenant of devops is &quot;if you build it, you run it,&quot; meaning the appdev team that decided to use Postgres RDS is the one getting woken up at 2am.<p>It&#x27;s also, in my experience, one of the best ways to reduce masturbatory engineering decisions and get people to focus on picking boring technology that works. Coding up a serverless application in Rust that&#x27;s using a CockroachDB backend at a Python&#x2F;MySQL shop would get a lot of engineers excited, but those people would be less excited knowing they&#x27;re going to be the ones paged at 2am when this new and exciting architecture falls over in an unfamiliar way (as opposed to Python&#x2F;MySQL, where a wealth of operational knowledge at the org has already been built up).<p>Similarly, it naturally reduces architecturally complexity. Younger senior engineers love drawing boxes of queues, multiple microservices, event buses, etc to show off their skill in creating the ultimate engineering fantasy, but once you throw enough late night operational incidents at a senior engineer, suddenly the preferred architecture becomes &quot;an executable running on a box that I can SSH into when things go wrong.&quot;

&gt; The problem is most engineers don’t want to do operations work.<p>There&#x27;s your problem. You have people who build stuff without caring where and how it runs. Recipe for disaster.

Hard disagree but Mandy Rice-Davies applies.<p>Some interpretation of DevOps may be BS, e.g.<p>&gt;&gt; Need a database? File a ticket with DevOps.<p>That does seem like a bit short of the mark. E.g. that should be automated provisioning in 2018, never mind in 2022.<p>Counter points as to why DevOps is not BS:<p>Today there&#x27;s just an acceptance that you version your code in VCS, it wasn&#x27;t always that way. &quot;Hey, this doesn&#x27;t look right, i know you said you based your change on gui-app.latest-final2.zip but was that Steve or Laura&#x27;s version of latest-final2?&quot;. If you didn&#x27;t work through this period you&#x27;ll struggle to believe how common it was for shipping products not to be fully up to date in VCS or not to have VCS at all. DevOps changed this.<p>Continuous integration? No there were people hired as &quot;merge masters&quot; or &quot;build managers&quot;, i promise i am not making this up. DevOps changed this, the idea that you wouldn&#x27;t do at least CI or perhaps CD is unexpected today.<p>Deployment automation? Sure, you email it to the ops team, send a few more emails with attachments late on Friday with ammendments and hope that they deploy the right one. Automated deployment as far as the developer was concerned. The ops person on the other end? Sure they had a batman&#x27;s belt full of hand crafted tools and scripts but it was definitely pets not the cattle DevOps has made us strive for.<p>Testing automation? The testers sit on level 3 not next to dev on level 4, there&#x27;s about 4 banks of desks over by the cupboards, that&#x27;s all the testers. They have lotus notes databases with checkboxes to confirm when they test something. If they find an issue a regression report will arrive with the dev team in under a week.<p>I could go on an on. Platform teams are great when used correctly. You can say something similar about DevOps.

Over the last ten years, the market has tried to kill off the hardware, systems, network and security people, and mostly succeeded.<p>As a result, it&#x27;s relatively easy to find someone who advertises as &quot;full stack devops&quot; who has never actually operated any infrastructure more complex than a LAMP webserver cluster. And it&#x27;s hard to find a senior sysadmin who has enough years of experience to understand and troubleshoot all of your infrastructure from layer 0 up. People with that experience have moved into management or consulting or retirement, and there are no jobs for new folks.

I had a very weird sensation while reading this where I was shaking my head disapprovingly at this article. Specifically when they were talking about the commoditization of DevOps.<p>Speaking personally, I used to work on a Platform Engineering team for a major multinational that rhymes with ay-do-bay. And their requirements for the workloads that we needed to be built and hosted where very unique because you have teams running completely different languages and toolchains, esp. from acquisitions. I can argue there that the template K8s setups wouldn&#x27;t work.<p>In the author&#x27;s case, the only reason why they feel the way they do is because we finally have some sense of standardization on what Infra looks like for most companies. It&#x27;s no longer a question for most folks if they should adopt Docker, we have accepted images into our lives. Same for K8s (after a certain point of scale). So uh... yea. Catchy blog title to sell some thin layer on top of K8s in which it doesn&#x27;t solve the root issue that they are talking about.

I&#x27;m not sure if having two teams is always going to be better than having one DevOps team, but my experience in having two teams is that it&#x27;s rare to have the incentives aligned. The author of the post pointed out that dev teams will cut corners and throw broken applications over the wall to ops to deal with.<p>When ops gets woken up at 2am because someone in dev cut corners, what happens? Does dev feel the pain? Almost never.<p>The same thing happens when outside contractors develop code. They often provide a buggy and undocumented mess, and then no longer work on the project. They never feel the pain, so they&#x27;re not incentivized to provide good code.<p>Until we find ways to align incentives, we&#x27;re going to keep getting crap whenever more than one team is involved.

I disagree. DevOps was intended to break the organizational mindset of having this group of people here doing this set of activities and this other group of people over there doing that set of activities when in reality both groups of people are needed to work together and deploy software. It was about breaking down those organizational silos. Anybody creating a dedicated &quot;DevOps&quot; team was way off the mark, unless that team was a coaching team whose job it was to help other teams become self-sufficient. Unfortunately the fact that an engineering team had responsibility for their software from implementation, to deployment, to operations was a news flash to large organizations having traditional IT staff. The DevOps mindset has led to much better operational excellence.<p>The DevOps mindset has also led to the Internal Developer Platform this article discusses. Honestly, I don&#x27;t see how traditional IT organizations are going to easily arrive at such a platform without having adopted the DevOps mindset first.<p>So DevOps isn&#x27;t bullshit, but it&#x27;s not the end goal either. It&#x27;s a necessary step needed on your journey for getting somewhere better.

When I ran a fairly small team of engineers, I created what I called an &quot;Infrastructure Engineer.&quot; I staffed it with a fairly junior, but still brilliant, engineer.<p>He rapidly became the most popular member of my team.<p>His job was to commoditize configuration management, and strip away as much of the overhead from the coders, as possible. He didn&#x27;t do release management, and we didn&#x27;t really work automatic testing into our release workflow. This was because Japan did not trust auto-testing, so each engineer did their own unit and harness testing. Japan also wanted each engineer to make their own &quot;official&quot; release, as opposed to having a CD system spit it out.<p>There were reasons. I didn&#x27;t necessarily find them that compelling, but they were the boss, so I gave them what they wanted.<p>Japan liked him, as it gave them one single person to talk to, and he also helped them to streamline their own infrastructure. In fact, he is the only employee that I ever had (including myself), that traveled to Japan before being there a year.<p>For myself, I find using things like Fastlane, JIRA, and Jenkins, aren&#x27;t actually helpful, for a one-man shop. I tend to do a lot of stuff by hand.

One thing I miss:<p><i>Most developers have no fucking clue how Linux, networking, storage or whatever works under the hood</i>. They know how to develop whatever stack you&#x27;re at, but stuff like latency, packet loss, redundancy factors, backup policies, monitoring or other classic ops topics are completely beyond the comprehension of 99% of developers.<p>&quot;DevOps&quot; usually means some C-level execs say &quot;fire the expensive neckbeards that have the time to properly understand a system&quot; followed by them saying one of<p>- &quot;oh fuck, someone managed to compromise a service and because no one knows what the fuck firewalls are &#x2F; Kubernetes doesn&#x27;t come with ones OOTB the hacker got complete control of everything&quot;<p>- &quot;oh fuck, production is down because someone fat-fingered in Elastic Beanstalk which recreated the environment, dropped the RDS database and there were no backups&quot; (I&#x27;ve been personally bitten in the arse by their definition of &quot;recreate&quot; - all I wanted it to do was to replace the damn EC2 instance)<p>- &quot;oh fuck, we&#x27;re seeing insane AWS bills because someone DDoS&#x27;d us and no one created a cost limit or a sensible limit for autoscale or a simple dumb monitor that alerts someone&quot;<p>- &quot;oh fuck, we&#x27;re seeing an insane AWS bill because someone got his AWS access credentials stolen and some shithead spun up a ton of g3.xlarge instances to mine shitcoins&quot;<p>Other C-level execs see constant issues between &quot;ops and dev teams&quot; because their team leads are friends of the silo model and decide that instead of getting rid of the dumbass managers they&#x27;re getting rid of the ops team because &quot;cloud&quot;, with the exact same result.

I&#x27;m not sure I agree with author. Sounds like they have worked at shitty places that are using the word &quot;DevOps&quot; but really they are doing things the old way.<p>My company practices &quot;DevOps&quot; and it feels great:<p><pre><code> - Infra team build self-service tools (build, deploy, scale, observe) - Infra team write good documentation for these tools - Dev use tools to build, deploy, and monitor their apps - Dev not use use words like: AWS, k8, Envoy. These are abstracted by tools. - if problem with app (very common) Dev fix it using the tools - if problem with tool (very rare) Infra fix it </code></pre> We have no build engineers, release engineers, etc. However we do have a rotation (similar to on-call) whereby Dev is responsible for releasing code that week.<p>Sure there are sometimes problems, frustrations, etc. No system is perfect. But you are getting paid lots of $$$ so shush with your whining &amp; instead help improve the system.<p>For context my company is mid-size ~150 engineers

DevOps has suffered the same fate as Agile. In particular there is one thing that both had in common that got lost somewhere in most implementations: cross-functional teams [0].<p>Agile teams were supposed to be composed of not just devs doing everything (quickly, because agile is about raw speed, right?) but people competent in the various aspects of the system development, potentially deployment, customer needs, etc. working together as multidisciplinary teams to meet a particular objective. The purpose of this was to break the silo that is common because it feels natural to many managers (the same sort, I presume, who don&#x27;t like when their mashed potatoes touch their fried chicken on the plate).<p>Silos impede communication and promote the &quot;throw it over the wall&quot; approach to product&#x2F;system development. A system engineer (or team of) made the design after sales (and possibly only sales) talked to the users. Throw that design over the wall and let the devs build it. Devs throw it over the wall to test, maybe there&#x27;s a volley. Eventually it&#x27;s tossed to ops. A goat is sacrificed and maybe it works.<p>Multidisciplinary teams are able to communicate across those boundaries because instead of the role-silo the roles are all in the same team, working (more clearly) towards one common objective. But then businesses managed to fuck it up. They got rid of test, devs do <i>all</i> testing now. Devs do all the database logic. There are no UI&#x2F;UX experts anymore, it&#x27;s all full-stack, and on and on.<p>DevOps was supposed to be the same. It was supposed to take that Agile cross-functional team and add in the sysadmin&#x2F;operators (among other things). The critical problem being solved was the two (really more, but at the limit) silos of dev and ops failing to communicate. A Friday release with Ops spending the weekend rolling things back because Dev wasn&#x27;t there and didn&#x27;t even know the system wouldn&#x27;t work as intended. Maybe their test environment was too different from the operational environment, the reasons matter a bit but there are too many to enumerate.<p>Instead, businesses did what they do best, they fucked it up. Again. They said, &quot;Take the sysadmins and teach them to code. If they can&#x27;t, dump them in the woods. The devs will takeover the world.&quot; Then they started making &quot;devops&quot; job listings and full-stack grew to encompass a new set of skills.<p>[0] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Cross-functional_team" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Cross-functional_team</a>

Funny, I thought DevOps was introduced as a way of having the specialists be part of the same team, so that the dev side would keep ops in mind and the ops side would be getting dev help in automating. Creating a separate DevOps team seems like a manager reading an article and implementing without understanding.<p>The problem in a lot of orgs is having various priesthoods that have their own goals that aren&#x27;t aligned with other teams&#x2F;the business. For example:<p>- hardware purchase<p>- software purchase<p>- DBA<p>- ops&#x2F;infra<p>- networking (firewalls esp)<p>- security<p>- HR&#x2F;recruiting<p>- business analysis<p>- front end&#x2F;design&#x2F;ux&#x2F;dev<p>- back end dev<p>- technical documentation<p>All of these are roles and they can have specialists, but you probably want them aligned and rewarded for keeping the business running, not serving specific measurement goals like uptime, to the detriment of selling product. These priesthoods often have their own religious virtues that they espouse, like 3rd normal form or low TCO, that they pursue in absence of directions and understanding of their role in the business.<p>It&#x27;s easy to see the problem but wicked hard to prevent it or fix it. I have a small organization and it&#x27;s difficult to get people really on board with a vision.

I was semi salty reading this....but they are spot on in a lot of ways. There are times where my code looks like a bad 4 year old wrote it, but 10 minutes later I&#x27;m in a conversation where I have to explain basic security concepts, like not leaving everything wide open to the internet, to some random Sr Software Dev.<p><pre><code> For every operations person without software development skills, there are FORTY engineers without cloud operations skills. If you are going to build an internal platform, you’ll need experts with overlapping experience in both fields working together. </code></pre> I guess I just need to find a role with more inter-team collaboration; Being able to mostly self teach is great, until you have no one to learn from anymore.

The problem is that devs simply can&#x27;t accept that ops exist.<p>We spoiled devs way too much.<p>Software are still being thrown over the wall and ops takes all the blame.<p>The problem is that many devs can get away with spawning servers and doing the easy parts of ops, when it requires rules and discipline, then we all know what happens, over engineering and security holes.

&quot;From full stack engineering to DevOps practitioner, our industry loves to pretend everyone can do everything.&quot;<p>To me, this gets to the heart of the matter. As example, I present the required skillset for a front-end engineer:<p><a href="https:&#x2F;&#x2F;frontendmasters.com&#x2F;guides&#x2F;front-end-handbook&#x2F;2018&#x2F;" rel="nofollow">https:&#x2F;&#x2F;frontendmasters.com&#x2F;guides&#x2F;front-end-handbook&#x2F;2018&#x2F;</a><p>See the index on the left. Do you need to know everything about all of these things for every single project? No. But as you grow into a senior, you&#x27;ll be touching almost everything in that list.<p>It&#x27;s an absolute explosion in complexity. Web development once was barely considered engineering, now it&#x27;s one of the most complicated roles in the industry. Consider also that almost everything on that list is constantly evolving, this list being 4 years old.<p>Has all this added cognitive load and complexity resulted in massive productivity wins and dramatically better outcomes (UX, quality)? I&#x27;d say no, or at least it&#x27;s questionable.<p>My point being is that this is already too much. I work in teams with a distribution like this: senior (20%), medior (50%), junior (30%). So the vast majority of them are median. And the median programmer is severely lacking against our ever growing demands. It&#x27;s crude, but the typical programmer really sucks at programming.<p>So if next you&#x27;re going to add even more to this pile with all sorts of devops and funky cloud tooling, the issue becomes clear: we&#x27;re over-asking.<p>We over-value flexibility and scaling but ignore its dramatic costs.<p>&quot;Devops&quot; is just a made up word.

I hate devops. We used to have a dedicated systems team. Now programmers are expected to both write code and manage their own cloud infrastructure. These are two entirely different skill sets.

&gt; The problem is most engineers don’t want to do operations work.<p>Nah, the problem is that most (software) engineers don&#x27;t have the skills to do operations work.<p>Let&#x27;s not fool ourselves: software development is 95% development (writing code&#x2F;docs&#x2F;test etc) and 5% system administration (getting your local mysql or whatever up and running etc) whereas operations is usually 95% screaming at the machines (various linux tasks, writing glue scripts, infrastructure, networking etc) and 5% development (the aforementioned glue scripts, writing internal docs).<p>The fields do overlap, but very little.<p>They require two different skill sets.<p>They require two different mindsets, too: a software engineer is usually optimistic (works on my machine, will work in prod too) whereas a sysadmin&#x2F;operations person is usually pessimistic (what&#x27;s our disaster recovery strategy?).

Managing complexity should be job 1 for technical leadership. We&#x27;ve failed utterly when it comes to DevOps. The model should be Heroku-like deploys for 99% of apps&#x2F;companies. Instead we have an army of DevOps experts for k8s, helm, terraform, etc. Changing an environment variable now requires a DevOps person and&#x2F;or a PR in your GitOps setup. The feedback loop for troubleshooting issues in these environments is too long.<p>This is me right now: <a href="https:&#x2F;&#x2F;i.kym-cdn.com&#x2F;entries&#x2F;icons&#x2F;original&#x2F;000&#x2F;019&#x2F;304&#x2F;old.jpg" rel="nofollow">https:&#x2F;&#x2F;i.kym-cdn.com&#x2F;entries&#x2F;icons&#x2F;original&#x2F;000&#x2F;019&#x2F;304&#x2F;old...</a>

At this one place of employment, I had been struggling to divest myself of system administration duties for some time. I can <i>do</i> them, but I don&#x27;t particularly <i>like</i> them. I have a few things I&#x27;m good at in that area (troubleshooting, being paranoid), but I really despise middle of the night calls, early morning patching, and so on. I had finally gotten to a good place.<p>And then someone newly promoted decided to rewrite everyone&#x27;s job titles and I was suddenly DevOps, sucked back in.

It takes really good engineering leadership to build a platform team. With the wherewithal to follow through on migrations and marketing of the platform. Also, sourcing the right people internally and elevating them. These engineers become true full stack (the whole OSI model) engineers and their work most definitely shouldn’t be thankless or taken for granted by product leaders. I’ve seen efforts like this fall apart in very ugly ways due to a lack of leadership.

&gt; When was the last time you saw a product manager high-five the ops person and say, ‘fast fucking autoscaler!’?<p>That&#x27;s me. I did that.

&quot;DevOps Is Bullshit&quot;<p>Wait until you meet DevSecOps...

Couldn&#x27;t agree more top-level.<p>The other thing that this misses is that the deployments to the cloud are only half the battle. CI&#x2F;CD pipelines, Dev environments, and other SDLC phases like planning and testing are just as important as terraform. Making the infra as code better and more reusable is a great step in the right direction but only part of the puzzle to making software development workflows better in the big picture.

PaaS is grate, sure it doesn&#x27;t fully eliminated oprations but it makes it much easier for you to do what the article described.<p>Like most things it comes at a cost (for a good PaaS mainly the running cost).<p>But I increasingly come to believe that for small startups which fundamentally don&#x27;t have the resources to do what the article describes or anything close to it using PaaS and keeping operational complexity as low as possible is the way to go.

DevOps is an education and communication position. You won&#x27;t get the devs to do ops, since they are good at development and don&#x27;t know much about ops and the same applies to ops in reverse. You need people (ideally, on each product team) who understand enough of both sides to efficiently communicate problems and facilitate solutions. Before you make your devs write provisioning scripts for databases (which they can do, but probably won&#x27;t get the trade-offs of different parameters), you might want to explain the difference between running database migration scripts on application server startup on a single-instance dev laptop and on a distributed master-master replicated database with multiple application server instances. You increase awareness, you provide platforms, you communicate. That&#x27;s what DevOps and especially DevOps teams do. They are glue between people of different competence areas. The alternative is Peter&#x27;s principle applied to both sides.

&quot;devops&quot; is just when developers do operations, and most developers generally don&#x27;t want to because they are more interesting in writing software. If your job is just to manage infrastructure and you use terraform, you&#x27;re still just a sysadmin. It&#x27;s a &quot;boring&quot; job but someone has to do it.

I knew DevOps jumped the shark when I started seeing &quot;DevSecOps&quot; and other such nonsense. But I don&#x27;t think it was always so.<p>Several years ago, I did &quot;DevOps&quot; work for a year or two. My gig essentially consisted of automating the repetitive Ops tasks and giving engineers self-service tools to get their stuff into prod efficiently.<p>At the time, things like Puppet &amp; Chef were commonly-used tools, Vagrant was a widely-used development tool and Ansible was the new kid on the block. I don&#x27;t remember there being much YAML yet, and I&#x27;d only heard a few things about Docker. I wrote several custom tools in Python, and we used Jenkins as our CI&#x2F;CD server.<p>&quot;DevOps&quot; in those days was a cultural thing. The DevOps engineers owned the infrastructure and the software engineers owned the software. It was the DevOps engineers&#x27; responsibility to give the software engineers the tools needed to deploy, and also to ensure that the SWEs weren&#x27;t causing outages (where we were the first line of defense).<p>It started to get bollocksed up when hiring managers started defining DevOps roles in terms of tools used, and the tools themselves supplanted communication &amp; culture as the definition of DevOps.<p>With YAML, k8s and the rest of the nonsense, I don&#x27;t think DevOps is very possible these days, because when your config is tens to hundreds of thousands of lines of YAML, the tooling doesn&#x27;t even allow for a culture of self-service &amp; communication. SWEs more or less have no choice but to chuck stuff over the wall, and DevOps or DevSecOps engineers (or whatever buzzword nonsense the industry has adopted) have to perform augury to construct the configuration. Because of cloud vendor lock-in or just sheer complexity, whatever runs in prod is quite different than the dev environment, and everything just limps along.<p>EDIT: Now I read things about &quot;MLOps&quot; and so on. When will the madness stop?! Apparently we keep allowing recruiters and dimwit HR bureaucrats to define software processes and culture.

&gt;platform engineering and enabling developer self-service.<p>in any sufficiently large company platforms arent engineered, they are acquired or purchased&#x2F;licensed based on their viability as an &quot;enterprise grade&quot; asset that drives business success and reduces cost through identifiable if not meaningless KPI and record keeping.<p>In any sufficiently large company self-service is supplanted by rigid controls, authorizations, approvals, and annual reviews. this is done in the service of jira and the need to make-pretend work by an ever growing cavalcade of pseudoworkers who recognize self-service as the killing stroke of their career.<p>it can then be said, grudgingly and with scorn, that devops seems designed by its very definition to operate as an antipattern to some of the worlds largest, most successful corporations.

DevOps is bullshit in the same way that Agile is bullshit. So, it&#x27;s actually not at all, but just has become so fashionable that the term is sufficiently abused to become almost useless.

Very shortsighted article ignoring a lot of responsibilities developers don&#x27;t want to assume and want somebody else to be on the hook for costs, secruity, breaches, on-call rotations, SOC-2 compliance - you name it. With freedom comes responsibility!

Well, to be fair, most of the engineering is bullshit.

Part of why I am fed up with this industry is that I no longer know what my role is. It seems that I have to:<p>* Develop new features<p>* Fix bugs<p>* Sit in standup, &quot;alignment&quot; meetings, and any meeting created by a Zoom-promiscuous engineer (let&#x27;s hop on a call!)<p>* Reply to dozens of Slack messages per day<p>* Write documentation<p>* Develop and maintain the CI&#x2F;CD pipeline<p>* Be an expert on Observability (aka Debugging in Production)<p>* Have Galaxy Brain level knowledge on the entire cloud setup in all environments<p>And now I am thinking, if being an engineer involves all that, why not just find a gig with pure DevOps and not worry about product development?

Call the person or the team or the practice whatever you&#x27;d like: DevOps is the merger of development and <i>business operations</i>. It is not developer + sysadmin. Companies with no &quot;devops&quot; (people&#x2F;team&#x2F;practice) build big sticky balls of mud. If they&#x27;re lucky, they hit product-market-fit before the mud dries - if not, well, this is why most software companies die and why most large organizations cannot make progress. DevOps simply plots the trajectory of our drying ball of mud, and attempts to keep it moist for as long as possible.<p>I don&#x27;t care if I&#x27;m by myself, if I have a team, if I have a devops title - all I want is to prevent the existential death-by-garbage-fire that eventually consumes all technology. If you&#x27;re a developer and you&#x27;re <i>not aware</i> that you&#x27;re marching slowly towards a complexity-cliff, that&#x27;s fine - we just have different jobs. If you&#x27;re a developer and you&#x27;re aware commits aren&#x27;t by default equivalent to progress, congrats, you&#x27;re an SRE&#x2F;DevOps&#x2F;Senior Engineer&#x2F;Platform&#x2F;whatever.

A lot of this rings very true but I think it&#x27;s missing the actual nobel intent of DevOps that is still true even if it&#x27;s been hopelessly corrupted. He touches on a lot of the counter examples that lead to the rise of DevOps. Things like ops&#x2F;admin teams making themselves a gatekeeper or bottleneck to delivery. That&#x27;s a classic problem in traditional IT departments. Dev is incentived to ship features, Ops are incentived to avoid downtime. The two goals conflict. DevOps says we all work together to ship features that don&#x27;t cause downtime. That&#x27;s a philosophical aim and not a new discipline. The maturity and sophistication of the tools and practitioners is orthogonal. If you hire a team and set them apart from Dev and give them an incentive to avoid downtime and call them DevOps then you&#x27;ve accomplished nothing. If you hire SREs or platform engineers or whatever else you call them and align their incentives to dev and dev to them then congratulations.

DevOps is outdated; in fact, most stuff nowadays in engineering is starting to get outdated due to how fast technology is evolving

There must be some a rite of passage or initiation ritual that I haven&#x27;t been good enough for yet where you need to write an opinion piece about the term DevOps, because otherwise I&#x27;m not sure why in 2022 we&#x27;re still having to read these articles. Whoever wrote the first one has trolled an entire generation of engineers.

How fucking annoying that stupid Newsletter Button is on mobile!

DevOps is still a great idea, just like Lean and Agile are a great idea. But a great idea isn&#x27;t enough. A million people have &quot;great ideas&quot; for businesses all the time. How many of those succeed? You can&#x27;t just have a great idea, you have to execute really well on the great idea.<p>Most people don&#x27;t know what DevOps is. Of the very very few that do know what it is, they are powerless to get other people onboard with the idea, because people are lazy and don&#x27;t want to learn things or change how they work. Even if DevOps is great, if the executives in your company don&#x27;t <i>force it down everyone&#x27;s throat</i> with business policies, training, etc, nobody will ever actually <i>do</i> it, because nobody really <i>wants</i> to.<p>Platform engineering is not gonna solve what DevOps is failing to solve. It&#x27;s just another silo. &quot;Let&#x27;s empower developers&quot; sounds great. But developers still lack most of the knowledge to maintain a complex system. Give them a million super-powered tools and they will still screw things up. Most developers I meet today don&#x27;t even understand the concept of DNS. That&#x27;s not exaggeration - they literally don&#x27;t understand how hostnames work, record types, zone delegation, authoritative records, ttls, much less propagation or transfers. And you want to, what, give them a fancy tool to change the system that they don&#x27;t understand? You still need operations, in any business, not just tech. Somebody has to be paid to care about the boring shit that keeps the business working. There is no way to automate away that responsibility, in any complex system in the world. A platform eng team is just adding another team on top of the Ops team you will always need.<p>What would <i>actually</i> solve a lot of this - and nobody is going to like this - is boring-ass business management best practice from the 50s. W.E. Deming. Lean. Six Sigma. The stupid shit that MBAs nerd out over? That stuff works. High-performing businesses that don&#x27;t just pay it lip service, but <i>actually</i> do PDSA, <i>actually</i> train their workforce, <i>actually</i> continuously improve their process, and make better business outcomes. But who among the tech nerds wants to listen to that? They just want to play with their toys and have no responsibility. &quot;Build me a platform so I don&#x27;t have to use my brain.&quot;<p>People have been studying businesses for the better part of a century. There is no easy way out of the morass. No single team or tool or paradigm will make things better. Until you consider <i>everything</i>, holistically, and put into place a barrage of different solutions, and actually teach people to do their jobs better, the actual outcomes of the work won&#x27;t improve.

So a symbol i.e. &quot;DevOps&quot; gets corrupted. What do you do? Make a new symbol, like the author is suggesting? Or try to reclaim the term? This happens time and again and I find the question fascinating.<p>FWIW, &quot;Platform Engineering&quot; used to be something different than what the author is suggesting.

&gt; The problem is most engineers don’t want to do operations work.<p>Glad to finally see someone else saying this.

&gt; Congrats, that’s not DevOps. I’d wager most of what they are doing is using Terraform and YAML to do menial tasks for the engineering team. &gt; Need a database? File a ticket with DevOps.<p>Shouldn&#x27;t the &quot;DevOps&quot; team build APIs and consoles so eng teams can provision their resources without knowing TFE&#x2F;YAML from a mile away? Really, this is year 2202. Please, learn from AWS and Netflix. Do software development. Get rid of the god damn ticketing process. Friends don&#x27;t let friends touch TFE&#x2F;YAML or whatever configuration files.

This is an old, dead horse larger shops solved:<p>- Systems get messy unless you have configuration management that converges on top of ephemeral instances<p>- Build packages that work the same in all environments if possible<p>- Dev-prod parity: if that means running Docker locally or on some throw-away dev servers, do it<p>- 12factor principles<p>- Every business department has an API and shares the same or similar messaging and storage tech<p>- Change control: have high-available (HA) to where you&#x27;re never upgrading individual machines and always backup&#x2F;restore&#x2F;replacing with fresh nodes<p>- Repeatable, precise, cached, incremental, distributed builds. Even if it means throwing out timestamps in the binaries<p>- Cache build artifacts forever (almost)<p>- Cryptographically sign commits and build artifacts (don&#x27;t sign hashes because those are weaker)<p>- Canary, sharded deployments<p>- Rarely&#x2F;never allow changes directly to boxes<p>- Require PRs made through configuration management<p>- Require PRs have to have another developer code review them<p>- Automate the heck out of linting and testing before it&#x27;s generally allowed to drop in prod<p>- Store secrets either in conf mgmt or in a separate system like Vault<p>- Have SREs who know how things will end up in prod to troubleshoot the complexity and provide CI&#x2F;CD and {I,P,Db,S}aaS<p>- Reduce the number of duplicated systems to a minimum but not to where it is too awkward or difficult to maintain<p>- Remember that prod, corp, and endpoints (phones and laptops) aren&#x27;t the same but try to share bits as much as possible while isolating differences<p>- Corp tends to run more heterogeneous than prod. It&#x27;s okay but don&#x27;t let it get out-of-hand without having recommended (as opposed to mandated) standards except for security and third-party component review<p>- Eliminate technical debt: don&#x27;t allow layers of crap or be precious about gross code just because it works<p>- Consider the risks and impact before making changes<p>- Have a backup&#x2F;DR&#x2F;BCP plan

I like doing &quot;devops&quot;&#x2F;operations work. Getting a service out the door with well thought out infrastructure, plus observability&#x2F;alerting&#x2F;runbooks to be handed off to an Ops&#x2F;SOC team is a great feeling. Then you work on the next service.<p>If you don&#x27;t have some sort of Devops team (yes, the term is too broad) to push back on infrastructure complexity and ensure that logging&#x2F;metrics&#x2F;alerting&#x2F;documentation is done, developers just won&#x27;t do it.

I think DevOps team maintains infra and monitor ops so that other teams can &quot;plug-in&quot; their applications. For example DevOps team sets up Terraform but dev teams plug-in their own repos and configurations to use that private enterprise Terraform &quot;cluster&quot;. Another example: DevOps team sets up Airflow clusters but Data engineer teams use them.<p>At least this is how the role works in my company.

&gt; In production, we are running in containers, but developing in the container is too slow, so the team leans towards asdf and a README full of stuff to copy, paste, and pray. During a sprint, an engineer adds convert (ImageMagick) to the mix to support manipulating images and forgets to update the Dockerfile, and then production goes down.<p>Wow, it&#x27;s like you can prove anything with a contrived example!

Well sure I would like to do DevOps as well as a software engineer but I&#x27;m getting drowned in agile ceremonies already, meanwhile the devops guys has like 10 hours a week more time as he doesn&#x27;t have all the agile re-&#x2F;pre-&#x2F;postrefinement&#x2F;retro&#x2F;intro&#x2F;outro&#x2F;daily&#x2F;weekly etc. They do like bidaily devops meet or 1on1 calls and care for their infra.

Co-opted, branded and misinterpreted by the enterprise - &quot;#DevOps&quot; is not DevOps as it was intended. Pretty spot on.

&gt; The problem is most engineers don’t want to do operations work.<p>I was convinced this was exactly what devops wanted to address and solve.

DevOps to me is:<p>* Infra as code<p>* Infra code sits in a repo<p>* Infra code is managed by the (senior) devs<p>* No separate ops team, or ops role, is needed: it&#x27;s a responsibility of the devs now.<p>* Monitoring production: also by the devs -- if an issue arises &quot;we&quot; take turns in solving them as devs.<p>DevOps is no longer managing your own hardware, but using hardware-behind-an-API in the cloud.

DevOps (the term), like Agile, made sense when it was introduced but lost its meaning as the industry evolved.

How many companies are there where a Dev team writes code and &quot;throws it over the wall&quot; for Ops to deploy and maintain? I hear people talking about this as an anti-pattern all the time, but I haven&#x27;t been in enough workplaces to see what this looks like when it happens recently.

The name meant something a while ago: Something along the lines of:<p>&quot;Turning the carnival bumping cars into a ferry&#x27;s wheel or maybe the teacups&quot;<p>Today&#x27;s Dev-Ops is a Teacher&#x27;s Lounge Room with a big suggestion box by the door.<p>One more shortcoming in the long list of modern needs.

I don&#x27;t understand why we can&#x27;t take more ownership of our work. Even if you are not immediately and directly compensated for every inch that you go above and beyond, you are still in a position to make yourself irreplaceable in terms the business cannot ignore.<p>Think about the long play. You join a startup with a broken, hot-potato-style &quot;devops&quot; process. Instead of saying &quot;not my problem&quot; all day, you can take some ownership of the items slightly outside your space and try to arrive at a better solution. If you do this often enough and with enough persistence, the end customer will eventually see benefit. At some point, management will likely notice the correlation as well. Even if they don&#x27;t, you have gained far more experience than you would have otherwise and can maybe go start your own damn company, realizing up to 100% of the value of your labor.<p>This is why I try hard, even if someone doesn&#x27;t make me.

I feel like I need some definitions to understand this article and discussion. What is dev? What is ops? What is devops? What is system administration? What is sysadmin?<p>I&#x27;m 47 and what is described seems like the old problems that I thought devops solved.

I think of DevOps as &quot;deploying and maintaining infrastructure on which applications run&quot;, vs SW Engs as &quot;building applications&quot;. Not sure if that&#x27;s the &quot;accepted&quot; definition or not, but works for us. YMMV.

see: &quot;The Big DevOps Misunderstanding&quot;<p>[01] <a href="https:&#x2F;&#x2F;linkedrecords.com&#x2F;the-big-devops-misunderstanding-8435a910a5fd" rel="nofollow">https:&#x2F;&#x2F;linkedrecords.com&#x2F;the-big-devops-misunderstanding-84...</a>

Certain articles need a list of &quot;required reading&quot; before reading the article.<p>You should at least read the &quot;dev ops handbook&quot; before reading this article.<p>The article should then talk through its points using the book as a reference.

The article was interesting. It seems like it misses that the complaints of why DevOps is dead is a culture problem, which I&#x27;ve found are hard to solve by throwing new tech at the problem.

I identify as a sysadmin.

&quot;The growing zeitgeist is that “platform engineering is the future.”[1][2] And given that I co-founded a product in the space, I sure hope so! &quot;<p>oh, now i understand this blog post..

I&#x27;ve worked with a &quot;devops&quot; team that spend hundreds of thousands of dollars on puppet consultants and never got it working. Another team spent it&#x27;s time badmouthing various technology choices, but then didn&#x27;t realize that our customer&#x27;s onsite deployment dictated the technology choices and chose the wrong tech.<p>The real reason devops exists (what used to be called Application Engineering) is because development is pretty clueless about the customer&#x27;s runtime environment. They don&#x27;t really understand the tuning necessary to create a production system.<p>Example: very few developers understand how big your DB connection pool should be, because they tend to only test one connection scenarios. That&#x27;s assuming they&#x27;re actually know enough to use a connection pool. And they almost never handle failover scenarios.<p>DevOps should be the purview of the senior engineers. If you&#x27;re building without an understanding of your deployment environment then you&#x27;re screwed. And more importantly, you&#x27;re not taking advantage of platform features.<p>As an example, deploying to SSDs (which everyone should be doing) means your database performance has just gotten an order of magnitude better for free. You need to retest and get rid of a lot of those performance-related changes.<p>Let&#x27;s put it this way: one monolithic Java&#x2F;spring application I worked on was basically a SOAP server with a UI that took up gigs of RAM. It was that big because the scaffolding required to handle all that was, well, huge. But really, it was essentially a web server that served pages to connected clients. All that other shit was overhead...so on AWS it got transformed into a few lambda functions and REST apis. Without an understanding of the deployment environment (and the possibilities associated with that) it would never have happened.<p>TL;DR: senior engineers should be the devops people, because how and where you deploy software should determine how you should make software.

This is one of those things that just drives people out of tech and leaves the ones still in it super frustrated. So many buzzwords and PR non-sense from non-tech mgmt.

&quot;DevOps is broken&quot;<p>Said the 4th article on the subject this month.

&gt; The problem is most engineers don’t want to do operations work.<p>My experience tell that most engineers don&#x27;t want to do any work and it&#x27;s ok.

I&#x27;ve started calling devops teams that are hostile or incompetent &quot;DevObs&quot;. Developer obstructions.

Why was the title changed from the original?

OP that newsletter button on my mobile browser is covering up words in the article. What the fuck.

DevOps is like everything else. It is what you make out of it

&gt; You’ve got a DevOps team.<p>Yeap. Largest red flag there is.

I have a hard time trusting someone who calls something &quot;production-grade.&quot;

Platform Engineering is the future.

TL;DR; s&#x2F;DevOps&#x2F;Platform&#x2F;

Re-posting a comment I made in a different thread, because I think the friction in devops is primarily a socio-cultural issue whose origins lie outside of tech orgs.<p>===<p>A colleague of mine once shared his lay sociological theory about dev vs ops, and if taken for what it is -- an essentialization -- it&#x27;s an interesting perspective. The idea is that ops people have inherited a blue-collar culture, whereas devs have inherited an office-worker&#x2F;academic culture.<p>Ops people conceive of their work as fundamentally operational: progress is measured in terms of actions taken, and while automation is greatly valued, there is nothing inherently &quot;messy&quot; with one-off fixes; the objective is to get things working now. The pathological case for this mindset is that of constantly being on the back-foot, responding to incidents with one-off fixes without recognizing that many of them share a common cause that could be addressed.<p>Dev people conceive of their work as fundamentally intellectual: progress is attained when the problem is correctly conceived, at which point the solution follows naturally. While writing code is greatly valued, most effort should be spent understanding the problem; the objective is to solve it correctly, once and for all. The pathological case for this mindset is that of over-engineering by an ivory-tower idealist, disconnected from the messiness of real-world praxis.<p>In nearly all orgs I&#x27;ve seen, the proportion of first-generation college grads is greater in ops teams than in dev teams. So too is the proportion of people who come from blue-collar families, or are mechanically inclined (ex: look around and see who tinkers with cars). Likewise, the proportion of people holding graduate degrees is greater in the dev crowd (ex: look around and see who&#x27;s into math).<p>It should hopefully be clear that neither is superior to the other. The point is rather that the divide between dev and ops is partly sociological, which means it is largely based on values. Ops will tend to over-value &quot;honest hard work&quot; and dev will tend to over-value &quot;clear articulate thought&quot;. There is also some latent, historical tension between these sociological groups, which has a funny way of masquerading as a technical problem. It is helpful to view arguments about &quot;just ship it&quot; vs &quot;design it the right way&quot; through this lens.<p>Far from being a cute &quot;just so&quot; story, it&#x27;s been my experience that this dynamic is very important for two reasons: (1) it&#x27;s harder than you might expect to foster the sense of common destiny required for real &quot;devops&quot;-style collaboration, and (2) each side of the dev-ops divide has a lot to gain from learning when the other side&#x27;s mindset is helpful, and how to cultivate it in themselves.