TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Software Developers Should Have Sysadmin Experience

577 pointsby beekumsover 8 years ago

70 comments

cs02rm0over 8 years ago
This might be controversial, but I don&#x27;t think you get to be a half decent developer without being a reasonable sysadmin.<p>Maybe my experience is unusual, but I&#x27;ve never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue.<p>And I&#x27;ve never thrown code over a wall without having tested it in a representative environment.<p>The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800&#x2F;day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone.<p>When a sysadmin sees an issue in &quot;their&quot; environment that they understand there&#x27;s a tendency for some of them to just see that issue as the only thing the developer has had to deal with that month. In all likelihood, in a productive company, it&#x27;s the most trivial issue the developer has had to resolve that day.<p>Often this stuff goes more smoothly where the developers (I mean, it&#x27;s not as though if you&#x27;re going to drop one of the two groups of people it&#x27;s going to be them going) manage production and there aren&#x27;t people with separate job titles and the resulting friction between them.<p>Sorry. There must be great sysadmins out there struggling with terrible developers, I&#x27;m sure of it. I just haven&#x27;t seen it.
评论 #13355145 未加载
评论 #13355210 未加载
评论 #13356715 未加载
评论 #13355038 未加载
评论 #13356191 未加载
评论 #13356134 未加载
评论 #13355812 未加载
评论 #13355929 未加载
评论 #13356352 未加载
评论 #13355794 未加载
评论 #13356099 未加载
评论 #13357224 未加载
评论 #13355684 未加载
评论 #13356082 未加载
评论 #13355873 未加载
评论 #13356305 未加载
评论 #13356873 未加载
eviln1over 8 years ago
Hi, I&#x27;m a Sysadmin, and I&#x27;ve been a grumpy one through a larger part of my 15 years experience. My main issue was that Developers were acting like Users: they don&#x27;t care about what you have to deal with, they want things to &#x27;just work&#x27;. In return, I&#x27;ve treated them like children, in some instances yelled at them when they did dumb stuff. I&#x27;ve tried to educate them when possible, and was angry when the education didn&#x27;t stick. At the time i was the &#x27;King of the Hill&#x27; type of sysadmin - natural leader of a very small and tight team, kind of irreplacable, and with enough years in the company behind me to consider myself as a demi-god.<p>When I switched companies, I came across better developers. Some had decent sysadmin skills, but the main difference was that they actually took interest in how things worked past the &#x27;git push&#x27;, and when I asked &#x2F; required them to make some changes that would make my life easier, they listened, discussed and adopted when appropriate. With those same guys, I took interest in what they were doing, what their actual job was and came up with ideas that would make things easyier and run smoothly on both ends. After a while I figured out that they weren&#x27;t actually better developers - they were better people. (Also, I figured out that being grumpy was not the best approach and that patience, kindness and gratitude could get people to do more than snark, humiliation and flame-throwers.)<p>I guess my point is: you don&#x27;t really NEED to have sysadmin skills to be a decent developer; what you really need is to care about what sysadmins do - be curious, talk with them and trust them when they say that your brilliant idea won&#x27;t work in production.
评论 #13356230 未加载
评论 #13360261 未加载
评论 #13356206 未加载
评论 #13356041 未加载
评论 #13356240 未加载
评论 #13359071 未加载
评论 #13356183 未加载
评论 #13357510 未加载
bandramiover 8 years ago
As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management.<p>Why do things like Docker exist? Because developers got tired of sysadmins saying &quot;sorry, you can&#x27;t upgrade Ruby in the middle of this project&quot;. Why does virtualenv exist? A similar reason.<p>Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us on the sysadmin side of the aisle capitulating and saying that developers can&#x27;t be stopped from having the newest version of things, and I think that&#x27;s a bad idea.<p>15 years ago, when a project would kick-off, as a sysadmin I&#x27;d be invited in and the developers and I would hash out what versions of each language and library involved the project would use. This worked well with Perl; once the stacks started gravitating to Ruby and Python it was a dismal failure.<p>Why? Because those two ecosystems release like hummingbirds off of their ritalin. Take the release history for pip[1] (and I&#x27;m not calling pip out as particularly bad; I&#x27;m calling pip out as particularly average, which is the problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Furthermore, there were more regression fixes from previous bumps than feature additions. You&#x27;ll also notice that none of these releases are tagged &quot;-rc1&quot;, etc., though the fact that regressions were fixed in a new bump the next day means they <i>were</i> release candidates rather than releases. Ruby is just as bad; the famous (and I&#x27;ve experienced this) example is that an in-depth tutorial can be obsoleted in the two weeks it takes you to work through it.<p>Devs are chasing a moving target, and devs who haven&#x27;t been sysadmins may have trouble seeing why that&#x27;s a bad idea.<p>[1]: <a href="https:&#x2F;&#x2F;pip.pypa.io&#x2F;en&#x2F;stable&#x2F;news&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pip.pypa.io&#x2F;en&#x2F;stable&#x2F;news&#x2F;</a>
评论 #13356672 未加载
评论 #13356402 未加载
评论 #13356903 未加载
评论 #13356643 未加载
评论 #13356358 未加载
评论 #13356412 未加载
评论 #13357771 未加载
评论 #13356337 未加载
评论 #13356582 未加载
评论 #13356557 未加载
评论 #13356312 未加载
nokyaover 8 years ago
I&#x27;d say there is too much effort in reasoning on the wrong problem. What worries me the most is the &#x27;why&#x27;: why do (too) many software developers don&#x27;t know about sysadmin?<p>I have been involved as a consultant in large software projects in the last two years and a vast majority of money lost in delays and bugs was caused by devs not understanding: 1) the difference between virtual memory and physical memory 2) the difference between costs of data storage per storage medium 3) the concepts of network round-trips 4) and hardware bandwidths 5) how to install and configure a web server on a workstation 6) how DNS works 7) how AD authentication works 8) what ORM frameworks do 9) how to write a raw database query (not necessarily sql) 10) the difference between navigating through database records on a database server vs. an application server vs. a client, 11) HOW TO INSTALL THEIR OWN WORKSTATION AND TROUBLESHOOT IT!!! N) etc. and those are just the topics that I can immediately remember.<p>As I see it, it&#x27;s not about &quot;they should&quot;. For me it&#x27;s about understanding how many devs deal with such a level of ignorance on the systems they interact with, on a daily basis. This situation hurt my feelings everytime it happened and I struggled to accept it. I am not a sysadmin nor a developer but my daily work is insanely improved by my (even basic) understanding of how my workstation works and how to manage it.
评论 #13354886 未加载
评论 #13354963 未加载
评论 #13355069 未加载
评论 #13355129 未加载
评论 #13358215 未加载
pjmorrisover 8 years ago
Should sysadmins have software development experience (e.g. DevOps)? For what values should &#x27;X have Y experience?&#x27; Should we go as far as...<p>&quot;A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.&quot; — Robert Heinlein, Time Enough for Love
评论 #13354790 未加载
评论 #13354780 未加载
评论 #13354782 未加载
评论 #13355205 未加载
评论 #13356357 未加载
评论 #13355706 未加载
评论 #13356602 未加载
评论 #13354983 未加载
评论 #13355328 未加载
评论 #13357322 未加载
评论 #13354770 未加载
blowskiover 8 years ago
Sysadmin knowledge definitely helps, but so does an MBA, knowledge of writing, public speaking, design, user experience, networking. Oh, and the domain of the problem itself.<p>The skills I require of my developers depend on the rest of the team and the project.
评论 #13354726 未加载
kabdibover 8 years ago
I used to work on a product where the three major teams were Client, Server and Ops. Tens of millions of customers used our stuff.<p>Client and Server folks were on separate floors of the same building. We didn&#x27;t interact much at all, except by trading bugs back and forth. The management chains met at a VP. Three or four times a year we tried to coordinate a release, and it always took at least a month. Getting a feature out the door might take a year, with all the paperwork and pipelining of release schedules.<p>Ops was in another building. The only things that both Client and Server teams were sure of was (a) they did a bunch of customization of our stuff so that it would work, and (b) they hated us.<p>Support was in another state. We were not <i>allowed</i> to talk to customers. Maybe once a year Support would fly in to talk to the teams about pain points. I think we did an okay job addressing these, but it took a <i>long</i> time, and customers suffered a lot.<p>I won&#x27;t talk about the disaster that ensued when Scrum was thrust upon us, or the splinter projects that spun off to try to fix things (but wound up being lots worse).<p>You <i>have</i> to be close to the customer or you won&#x27;t know if you&#x27;re succeeding, or even on the same page. You <i>have</i> to know what your software is doing in production or you&#x27;re just sitting in an ivory tower pontificating about angels-on-the-head-of-a-pin nonsense. You <i>have</i> to spend time in the trenches measuring and fixing stuff or you&#x27;re hatching an unmanageable disaster. The good news is that most of this is actually kind of fun. The bad news is that, when managed badly, this can turn into a horrible and soulless grind of pager duty and making legacy code even more legacy to fix wee-hours downtime.<p>I still get a rush when I get feedback from a real, live customer, and I think that isolating your teams from customers is one of the worst things you can do to a team and to a product. Getting teams to work on ops and support aren&#x27;t bad ways to improve this.
curun1rover 8 years ago
We took it one step further on my team. The best way to ensure that the pager(duty) doesn&#x27;t go off at 2am in the morning is to put developers in the first on-call group. Not only did it work, but the developers got a much more nuanced view of how systems operate. When we first started, I&#x27;d see developers using ping to determine whether a nameservice entry was correct. After a few months of handling almost all of our own ops, developers knew how each piece of the puzzle worked and how it all fit together rather than the hazy, largely abstracted view they had before.<p>But we learned that we needed it to go the opposite direction too. We had ops people who, when given a corporate-wide mandate to apply a security patch or some such task, would log into every machine and apply the patch, despite the fact that we&#x27;d been practicing immutable infrastructure with zero-downtime deployments. We hadn&#x27;t given them the necessary exposure to the dev side to understand that you had to apply those fixes to a base image and trigger a redeploy. There was a lot of finger pointing a couple of weeks later after it was discovered that the fixes were overwritten by an application deploy.
评论 #13355341 未加载
gedrapover 8 years ago
That&#x27;s what I did at my tiny engineering team (3 folks) and results have been great.<p>If you have a small organization (say &lt;10 engineers), it&#x27;s crucial that every developer writing server side applications can also do at least some sysadmin work. As the article says, it leads to deeper understanding and it really helps when thinking about scalability, fixing certain kinds of issues, etc. It also often shortens the feedback cycle and requires less throwing over the wall. As a bonus, you increase the bus factor.<p>Automation is the key though. If everyone connects to the boxes and does random things manually over ssh, nothing good will come out of it.<p>Still, you need to have a person or two who are responsible for the vision of the architecture&#x2F;systems and who make sure that things don&#x27;t go off the rails.
kqr2over 8 years ago
Somewhat related, I feel that mechanical engineers who design cars should have some experience servicing vehicles.<p>Sometimes engineers will not leave enough space, use weird fasteners, etc. that make a simple job much more complex.
评论 #13354638 未加载
评论 #13354601 未加载
评论 #13355012 未加载
评论 #13355437 未加载
xenadu02over 8 years ago
I don&#x27;t think &quot;should have&quot; is the right statement; more like having at least one person on the team with sysadmin experience is extremely helpful. I know it has been for me.<p>Of course in HS&#x2F;college I ran a website that was a frequent target of hate in the late 90s&#x2F;early 2000s and it taught me about XSS and CSRF before people invented fancy terms to describe them. It also taught me about HTML&#x2F;JS escaping, DoS&#x2F;DDoS, SQL injection, and how all your defenses are useless if someone social-engineers their way into root and nukes everything. I have the assumption that everything is compromised and user input is toxic waste burned into my subconscious.
PaulRobinsonover 8 years ago
I worked for an ISP in the late 90s&#x2F;early 2000s for about a year before my final year at Uni that when I started had 2,000 customers and by the time I left we&#x27;d got to 750k customers.<p>It was the best training I could have for writing software for the Internet. Just a couple of months ago we noticed one of our core apps was not scaling no matter how many docker containers we had spun up in Mesos. Spent a week breaking it apart using that experience and being able to make changes directly to the code base and being able to talk to devops in a language they understood, and in under 5 days we managed to identify and fix 7-8 different issues.<p>I would value a dev with sysadmin experience _far_ higher than one without in a tech business with a headcount under 500: it&#x27;s going to lead to fewer problems and issues in the short and medium term.
just2nover 8 years ago
Anecdotal as it is, I started my career working as a sysadmin while studying CS. When I was younger I was quite interested in security (e.g. I followed defcon and CVE lists and read a lot of manuals and source code). I built server software, broke applications, and did a lot of reverse engineering. In that time I was forced effectively to learn about how the OS worked and how to utilize it, both Linux and Windows, from C APIs to scripting, package management, and everything else needed to effectively work in those environments, primarily around reverse engineering and vulnerability research.<p>That experience naturally lead to me becoming a sysadmin during my study at university. It was a fairly straightforward application of what I already had learned with a much larger scale of management. The primary thing I gained out of it was a drive to automate everything. When I started that job most of the sysadmin work was manual, but a few of us spent a huge amount of time focusing heavily on automation and when I left most of the work was automated and we were just doing meaningful firefighting and supporting development.<p>As an engineer, the main benefits have been understanding how my software is going to run in an actual software&#x2F;hardware stack, easily jumping into a production environment and debugging complicated issues, being able to quickly have my OS do what I want, and that drive to automate everything. A lot of that informs how I build software and in general it feels like it makes me a lot more productive.
评论 #13359737 未加载
tyingqover 8 years ago
I would change it from &quot;should have sysadmin experience&quot; to &quot;should have operating system knowledge, or ask the right questions&quot;.<p>The example in the article isn&#x27;t what I tend to see in real life. What I do see are things like:<p>- Not knowing about various limits (number of open sockets, or listen queue depth for example), how to know you&#x27;ve hit one, how to deal with it.<p>- Not handling various error situations correctly (can&#x27;t open file due to permissions for example)<p>- Security issues. ( socket listening on 0.0.0.0 when localhost would work, for example)<p>- Making assumptions about things like &quot;current working directory&quot; or &quot;certain environment variables will be already set for me&quot;<p>For many of these, including a sysadmin or system knowledgeable architect in the right discussions would suffice.
arca_voragoover 8 years ago
This is a short little article that just barely touches on a much deeper, often hidden issue; The state of system administration in business is abysmal.<p>It&#x27;s not the developers fault though, at least not as much as devops types would like you to believe. I think the author has a good point, in that its good to get Devs thinking about real world environments on deploy, but the real world is much more complex than concurrency of servers.<p>All that being said though, very rarely have I as a sysadmin of 10+ years seen problems so easily attributable to devs. Of course I haven&#x27;t lived in hn&#x2F;sv startup land either, so take that into account, but failures in systems I have seen have almost always been a failure of management, up to and beyond C level.<p>I could go into detail, but I&#x27;ll save it for another time. Suffice it to say, what businesses need to be doing is getting better CTOs and CIOs who can bridge the gaping chasm between sysadmins and managment.<p>Devs, you keep being Devs, and let the sysadmins be sysadmins. Cross train and communicate when you can, but don&#x27;t fool yourselves, it is management that bears the responsibility and burden of you both. Management just doesn&#x27;t like to admit that to themselves or anyone else, so don&#x27;t play into this Devs vs sysadmins dialectic too much, lest ye find yourself the scapegoat next go round.
评论 #13354840 未加载
gbuk2013over 8 years ago
Software developers should also have customer-facing support experience (supporting other people&#x27;s code) so that they realise, the hard way, the importance of having extensive logging and debugging capabilities accessible to those unfamiliar with the codebase.
jedbergover 8 years ago
Having someone who is cross trained is always better than not, but they will be harder to find and more expensive.<p>A frontend UI person who can write their own backends is great. A backend developer who knows javascript and can build their own frontend is great.<p>A coder who manages their own cluster is great too, as is a sysadmin who who can write code.<p>And a developer who can do customer service, and therefore can fix a problem for every customer at once through modification of the application, is better than just someone who can answer the phone and make the customer happy.<p>All this is to say that someone cross trained will always add more value and will also cost a commensurate amount.
评论 #13354849 未加载
rsinglaover 8 years ago
While I agree with the general sentiment, the examples feel fairly contrived.<p>What I think is more practical is understanding when certain problems fall outside of your own scope or capabilities and when to engage someone else for their advice.
icameronover 8 years ago
Sysadmin or devops is responsible for uptime directly, being ultimately responsible for the services to remain performant. It&#x27;s a mindset and temperament that may help developers see solutions or foresee pitfalls while designing and implementing code. Zero-downtime attitude. Making and testing deployments, upgrades and migrations to be least impactful as a core function is probably the biggest effect of putting a developer in a sysadmins shoes.
fauigerzigerkover 8 years ago
What he&#x27;s talking about is systems architecture not sysadmin experiece.<p>Of course developers have to write code for a logical model of a machine that isn&#x27;t necessarily the same as the laptop they&#x27;re working on. But I don&#x27;t think anyone would ever dispute that.<p>Storing pictures on a local disk when the software is supposed to run on a cluster has absolutely nothing to do with sysadmin experience.
评论 #13358261 未加载
pmontraover 8 years ago
The post has a valid point even if the example is more about system architecture than system administration.<p>In my experience a developer doing one year in an Operations department gains some insight in what is going to be important in the longest phase of the software life cycle, that is when it&#x27;s exposed to customers and the company has to react quickly to problems. Proper logging, anything that can help pinpointing the cause of problems and even hot patch them at 3 AM.<p>The usual lean startup might have little thoughts for that (is it even going to have customers?) but established businesses do.<p>If it&#x27;s 6 months to develop and it runs for 10 years (with maintenance and new features), which phase is more important to a company, development or production? I&#x27;m a developer which did a couple of years of operations and I&#x27;ve little doubts about it. It&#x27;s where the company makes the money from.
bthornburyover 8 years ago
With the rise of devops (docker, puppet, ansible, etc...) , and cloud VMs, how often are sysadmins different from software devs?
评论 #13355678 未加载
评论 #13355578 未加载
评论 #13354590 未加载
评论 #13354704 未加载
评论 #13355617 未加载
mybridover 8 years ago
UC Berkeley&#x27;s approach to teaching computer science I believe is the correct approach.<p>1. In CS150 one has to build a hardware computer. 2. In another class one has to build an OS. 3. In another class one has to build a Database. 4. In another class one has to build a Network.<p>Education is key. Understanding the TLB inside the CPU gives one an appreciation for context switching between the only 2 users the CPU has hardwired into it: kernel and user.<p>Being a sysadmin is insufficient experience as education. Every developer should know how to build a hardware computer from the ground up.<p>Computer science at Berkeley doesn&#x27;t have a &quot;Java&quot; class. The language one programs in changes depending on the Professor and topic. That way one doesn&#x27;t get too attached to a language.<p>Most developers do not understand the hardware implementation of a thread, or now-a-days a docker. As a result both get used wildly incorrectly.<p>Threads are useful blocking on IO to keep the CPU from starving. However, switching threads simply to swap CPU bound jobs is inefficient. Just adding a thread doesn&#x27;t guarantee equal service to users just like adding processes does not.<p>Developers are using dockers today who have no clue why they are doing so...it&#x27;s just that everyone is doing it and they do not want to appear stupid.<p>Experience as a sysadmin is no substitution for education.
Steeeveover 8 years ago
The unspoken elephant in this thread is that sysadmins and developers have broken relationships surrounded by organizational dysfunction.<p>Personally, I feel that system administration experience of some sort is essential to development skills, and development experience of some sort is essential to system administration.<p>Communication and leadership skills are essential to both. It _used_ to be OK if you were the jerk at the office if you were good enough at your job (sysadmin or developer). That&#x27;s no longer the case because we&#x27;ve progressed well past the point where a single developer or sysadmin can be enough of an asset to look past their ability to work well with others.<p>Operations needs stability. Developers need to hit moving targets. Sysadmins need to keep things manageable, compatible, and secure. These aren&#x27;t incompatible goals, and usually when you get the techies talking they are sympathetic towards each other&#x27;s challenges and helpful to one another. With as many project managers, engineering managers, manager managers, directors, and executives in the mix I wonder if the problems in all these environments isn&#x27;t more of a historic issue with leadership failings.
jcadamover 8 years ago
You do not know pain until you&#x27;ve had to deal with government sysadmins as a contracted developer. And the govt has a strong preference for bringing sysadmins into the civil service rather than contracting that work out (like they used to) these days.<p>Which means the ones with actual, marketable skills aren&#x27;t working for the government.<p>Spending <i>months</i> waiting for a sysadmin to perform a task you <i>know</i> takes about 3 minutes is great fun.
vinceguidryover 8 years ago
I think the problem here is that decent admin jobs are getting harder to come by these days. Used to be any reasonable organization needed an admin just to keep systems straight, now they all rely on less-skilled help desk types and make it up with outsourcing companies, which in turn are shedding personnel and relying on big vendor contracts rather than individual skills.<p>I predict sysadmin culture and skills will be the next Fortran.
AliAdamsover 8 years ago
The underpinning logic seems false.<p>To extend the analogy of the house on a slope, if you don&#x27;t tell the developer you need a house designed to be built on a slope, that is a problem with the spec provided to the developer.<p>Although it can at times be more efficient for a dev to have sysadmin experience, making blanket statements <i>requiring</i> all developers to come with such a background is just sensationalism.
评论 #13355937 未加载
b212over 8 years ago
I&#x27;m a front-end developer with very little knowledge of sysadmin stuff, but I can&#x27;t agree more, the best developers I&#x27;ve ever seen had massive sysadmin experience.<p>Can you point some good resources about sysadmin I could use to improve my knowledge on the subject? Note I&#x27;m a noob when it comes to all this server-side magic.
评论 #13356025 未加载
ohstopituover 8 years ago
I&#x27;ve had to work at start ups where I was one of the initial developers (and the most experienced in the team) so I&#x27;ve been forced to learn a lot about Sysadmin and design (and a lot of surrounding dev tools for instance) but all in all - knowing more about sysadmin has made my dev experience change - I think a lot more about how it&#x27;s deployed, how it works in prod, how to get data to improve my code and so on.<p>I do feel all devs at one point should have some sysadmin experience - if not professional, then informally.<p>That said, sysadmin is strongly divided into 2 types of tools - configuration and code.<p>I love tools that can be coded (as an example - Gulp), over configuration (like Grunt) - as they generally have a lot less undocumented &quot;gotchas&quot; and getting started is generally a lot more simple. (I&#x27;m looking at you Webpack!!!)
kweinberover 8 years ago
A great reason to have cross-experience at the management level is to understand the different motivations of the two roles: * devs are incented (paid) to introduce changes to the system...mostly in the form of new features. * sysadmins are incented to stabilize the system and make it scale.<p>In organizations with uncoordinated or siloed management, this leads to infighting... sysadmins try to stop &quot;troublesome&quot; devs from making changes at all and actively slow down changes in the name of stability and developers at the other extreme try to get changes in without caring about stability and blame sysadmins for that.<p>Great management with experience in both will find ways to incent the teams to introduce regular change while maintaining stability.
raartsover 8 years ago
Wherever I went, I have seen the war between developers and sysadmins being fought. In enterprise the sysadmins have the upper hand, in frontend, mobile, SMB the developers.<p>Developers don&#x27;t have the burden of maintaining and upgrading multiple applications on the same server, with conflicting dependencies. 24&#x2F;7. Sysadmins don&#x27;t feel the pressure by end-users to deliver new features yesterday.<p>At the moment developers are winning because Docker. But when they grab that responsibility they will be the ones called when their 50000 containers are being hacked because vulnerabilities. And they will be under fire for the system being down and that costing the company a lot of money, so everybody breathing down their necks.
krosaenover 8 years ago
I think it&#x27;s more accurate to say that good developers need to have some level of systems expertise which probably means OS &amp; networking fundamentals + practical Linux experience. This is because you need to have some understanding of how your program will execute on (or across) machine(s), and some know-how to get your program up and running on a real machine available to users on a real network. Whether you need to be an expert in the latest best practices for managing a fleet of servers compliant with whatever regulations, that&#x27;s more for the sys admin (though it never hurts to learn more, there&#x27;s just so many things we developers could spend our time learning).
fnjover 8 years ago
It&#x27;s hard for me to &quot;get into&quot; developers who don&#x27;t know how the system works. I don&#x27;t mean they have to be full-fledged sysadmins, but it just seems natural to me that they should have a decent understanding of the electronics, how to assemble the hardware, and what&#x27;s involved in tuning it and keeping it all running.<p>It may have something to do with my education being EE, but that&#x27;s not the heart of it. I never practiced EE. I taught myself (more or less in order) building circuits, programming, building PCs from pieces, and elementary system administration. I know there are lots who never &quot;get&quot; all three, but the best I knew did.
saosebastiaoover 8 years ago
I get the sentiment. Personally, I could say the same thing about control theory, systems theory, economics, operations research, and statistics. And I could also share dozens of anecdotes of bugs and suboptimal software that was produced as a result of this lack of experience.<p>But at some point, you have to acknowledge that developers can&#x27;t know everything. Go deep or go broad, but accept that both paths have their limitations and benefits. Maybe in some cases you&#x27;ll run into a bug you created due to your lack of sysadmin experience, but don&#x27;t feel bad about it...your experience has led you to other types of expertise. Fix it, learn from it, and move on.
al2o3crover 8 years ago
In the case described by the author, it seems like most of the trouble would be resolved by the developer understanding the concept of &quot;distributed file storage is hard&quot; and coordinating with ops early - the dev doesn&#x27;t necessarily need to know the details of setting that up, but they need enough understanding of the domain to realize there&#x27;s work to be done on the issue.<p>For that matter, I suspect most sysadmin &#x2F; ops teams would be happier if developers could only strip &quot;just&quot; out of their vocabulary (and vice versa) - nothing more annoying than &quot;could you JUST do {thing asker doesn&#x27;t understand and thinks is trivial}&quot;
protomythover 8 years ago
Actually, I would rather language and tool designers have a go at a true System Admin job. There is a reason PHP gets installed by default and Ruby on Rails does not.<p>Frankly, at the end of the day, most businesses don&#x27;t want something that was &quot;working&quot; breaking. All the conflict comes from that one desire. There is a reason System Admins have to do the &quot;Patch Tuesday&quot; dance and despise any software that just updates by itself. Getting yelled at because Chrome updated to a version not supported by your ISV or your local developers is an amazingly fun experience.<p>Yep, System Admins have to be the company nanny. It sucks, but the apps need to keep working.
评论 #13358174 未加载
jordacheover 8 years ago
Just like how you typically distinguish front and back end developers, went wouldn&#x27;t you distinguish dev from devops?<p>Sure, it helps if an individual is not entirely myopic, and would be great if that person can wear multiple hats with skillset spanning all those disciplines.<p>However those different labels exist primarily they align to unique mindsets and solutions to a problem in a mutually exclusive manner.<p>Do you want a jack of all, but master of non?<p>Seriously, how many individual out there do you think have the capacity to develop their career so they have deep capabilities for all of those disciplines?
joshwcomeauover 8 years ago
I think that there&#x27;s good advice in this article, but a couple counter-points:<p>- Plenty of software developers work for small- to mid-size companies where the Node or Rails server running on your machine is nearly identical to the one running in production.<p>- Plenty of software developers work exclusively on the front-end, which means that the computer your code runs on in production is very very similar to the one you develop on (although then you have the added concern of testing on lower-end mobile devices, other browsers, etc)
seanwilsonover 8 years ago
I think everyone would benefit if they learned just a small amount about what their team members have to deal with. For example, project managers should learn a little about what programming involves while managing programmers and graphics+UX designers should learn a little about coding to make their designs more practical (i.e. identifying cases where a slightly better design requires a massive amount more coding effort and probably isn&#x27;t worth it).
teekertover 8 years ago
I think many professions should have sysadmin knowledge. In my work (biology) there is a lot of unnecessary clicking, not only in Excel (where Python&#x2F;R would be much more efficient) but also in web interfaces to transfer data daily. The latter is often completely automatable using cron and rsync over ssh. Many repetitive actions are a shell script or command line pipe away. But people don&#x27;t know.
评论 #13357793 未加载
kapilkaleover 8 years ago
I understand the point here; it can be useful to understand the abstractions on top of which you are working.<p>But given most engineering projects are crud apps without scaling problems, and PaaS companies like Heroku totally abstract away the system adminstration piece for those projects... I certainly wouldn&#x27;t recommend a new engineer spend any time learning system administration.
vickychijwaniover 8 years ago
I&#x27;m moving from dev to devops for a while, because of similar reasons. Excited to learn all these new things and peer beneath all the tools and abstractions I rely on! I have no doubt in my mind that it will make me a better developer - most abstractions have a tendency to leak, eventually.
pantulisover 8 years ago
This seems obvious to me. You&#x27;ll never see your code again with the same attitude after being on call.
mrskeltalover 8 years ago
As a developer with practically no sysadmin skills, how would I go about improving these skills?
评论 #13356771 未加载
0xC0DECAFE2020over 8 years ago
I have been at this since the early nineties and can&#x27;t comprehend that someone can be a developer and not have any experience with How to set up and administer the environment. I guess it&#x27;s a normal thing now?
golergkaover 8 years ago
Just a small nitpick with the article: it probably meant &quot;backend software developers&quot;. Because a graphics developer who mostly writes shader code would surely benefit more from artist experience.
评论 #13355659 未加载
mmartinsonover 8 years ago
Careful with that should, or you might take someone&#x27;s eye out.
partycoderover 8 years ago
As a developer the way we resolve this is by including the sysops department as a stakeholder that produces requirements for us to negotiate and implement.
adamdonahueover 8 years ago
The general trend in software development appears to be toward generalization over specialization. I&#x27;m not sure this is a good thing.
nanodanoover 8 years ago
Funny, I always see this from the other side. I always find myself thinking &quot;I wish sysadmins had more programming background.&quot;
EliRiversover 8 years ago
Yes, sysadmin experience will give me vital insight into my work writing embedded code for dishwashers.<p>Snark aside, if the post was headed &quot;Software developers need to understand the environment where their code will be running or they may not build it properly&quot;, which is the point being badly made by conflating &quot;environment&quot; with &quot;some kind of commodity x86 with a standard OS&quot;, it would be a lot more applicable.
dpaluyover 8 years ago
Software developers should have experience in the things they need to get their job done. This is not a revelation.
youdontknowthoover 8 years ago
sysadmin really means &quot;technical debt manager&quot; most of the time these days.<p>I really spend the majority of my time trying to slot something into place in a way that no one will notice that doesn&#x27;t disturb existing stuff...often that means duplicating bad behavior because its become a dependency.
imjustsayingover 8 years ago
Maybe it&#x27;s only because I&#x27;ve only ever worked with myself, but it&#x27;s hard to imagine a software developer who isn&#x27;t also a sysadmin by necessity. How do you debug things otherwise?<p>In what city or company is the barrier to entry for software developers so low?<p>I just can&#x27;t imagine paying above median income for a software developer who can&#x27;t install, for an example given in this thread, Visual Studio.
ultrasaurusover 8 years ago
One thing I like about DevOps is never having to hear &quot;it works on my machine!&quot;
评论 #13355905 未加载
teteover 8 years ago
I think the reverse is much more true, but maybe nobody argues about that anyway.
nierover 8 years ago
This needs to be mandatory for developers working on printer software.
mrmrcolemanover 8 years ago
Shameless plug perhaps, but this is free and will help here: <a href="https:&#x2F;&#x2F;www.cncf.io&#x2F;event&#x2F;webinar-cloudnativenetworking" rel="nofollow">https:&#x2F;&#x2F;www.cncf.io&#x2F;event&#x2F;webinar-cloudnativenetworking</a>
Diederichover 8 years ago
I worked in the home office of WalMart Stores, the retailer, from the mid 90s until 2009 as a hybrid network engineer&#x2F;developer. That is, my team worked in Network Engineering, but we wrote tons of code, because when there are millions of different IP addressable nodes on a centrally managed network, there will be code to manage it. (:<p>In that era, I think WalMart handled this apparent developer&#x2F;sysadmin confliact quite well. Believe it or not, even though we (the whole company) had exceptional uptime, we were also extremely agile. LOTS of code was written and rolled out across thousands of sites, multiple times a day.<p>There were a number of keys to this, and I&#x27;ll enumerate a few.<p>1. Most new hires (those that had minimal previous experience) to development positions had to spend their first six months working at one of the four or so major help desks. Note: they were paid their target salary, but in an hourly fashion. 2. The various operations teams had complete and final say so over what went out, and how incidents were handled. A couple of the big operational areas were Unix Operations, Network Operations, Windows Operations and Mainframe Operations. I called them &#x27;teams&#x27; but each had a number of teams, and their own help desk. 3. Any time there was an impacting problem associated with a program developed by team X, a war room was called by the relevant operational area. A member of team X (a developer) had to stay in that war room (switching off between team members if it ran very long) until the production problem was not only fixed, but also deemed unlikely to recur, except in cases where that dept of a fix would take a long time. 4. Every development team had a pager rotation, with rigorous expectations about responding to such pages. This was primarily to support the previous point. 5. Because of the enormous operational scale, all of the major operational areas had dedicated teams focused on automation. My team was that team for the networking area. Furthermore, most of the rest of the operational folk read&#x2F;wrote code to some extent.<p>In short, incentives were aligned. Teams that wrote externally facing code felt pain if the stuff they wrote and released caused problems. Operational folks wrote&#x2F;managed&#x2F;interacted with tons of their own code in order to manage the enormous infrastructure. Also, ops folk were far more willing to let things move with velocity knowing that the people who actually wrote the code would be required to support it, globally, any time of the day or night.<p>Another, perhaps even more important reason we were so successful during those years (and the years before) was a strong and vibrant esprit de corps. The entirety of Information Systems was, at the time, around 2000 people, and we were facilitating double digit year over year growth over a 150 billion dollar company. We had over 5000 remote sites in 15+ countries, with a diversity of software and infrastructure that was honestly pretty astounding. Each of those sites had quite a surprising amount of infrastructure.<p>We worked hard, and we produced huge velocity with fantastic uptime. For example, the network achieved six 9s of availability a couple of quarters.<p>In end, while things were sometimes contentious, we trusted each other, and only minority of teams were forced to work bad hours for any length of time.
timkofuover 8 years ago
Agreed.
andrewclunnover 8 years ago
Yes, yes. We get it, we should know all things, keep up with all the latest rends, work 80+ hours every week, and do our company&#x27;s IT support on the side for free while we code their software. Or companies could compensate us for our time and hire more god damn people if they need it. You know, just saying.
kahrkunneover 8 years ago
On top of the 20 years of experience we already require of junior developers?
评论 #13354890 未加载
madihaawanover 8 years ago
As software developers, this should notice and kept in mind that all these experiences are very much important. Thanks for sharing an important article.
A_Crazy_Ideaover 8 years ago
Managers should have management experience and less nepotism experience. One step at a time, guys.
necklaceover 8 years ago
Ironically I can&#x27;t see the images at the bottom of that page.
ovdollover 8 years ago
That’s interesting.
评论 #13356105 未加载
whenwillitstopover 8 years ago
Sysadmin is getting automated. There is not reason to know it.
tzakrajsover 8 years ago
I have ten years of experience in systems and have written tools software in Python and Javascript for Netflix, Stanford University and Google (current). I have never been a software engineer officially, but I want to be. I am a PC gamer, anime lover, and a proud SJW. Any takers?
评论 #13356212 未加载
WhitneyLandover 8 years ago
No. Developers do not necessarily need any experience as a sysadmin and I would advise against moving productive developers into this role for the purpose of misguided training.<p>The argument about understanding how code scales beyond one computer is fallacious. In fact it&#x27;s quite easy to learn and practice all kinds of distributed high scale architectures without getting up from a desk.<p>I will agree it&#x27;s important to understand the people and work connected to your job. It&#x27;s probably a good idea to eat lunch with the sysadmin, or QA, or UX person.<p>Would also agree its important to have respect, empathy, and collaboration others, but it doesn&#x27;t mean you need to switch jobs.
评论 #13354808 未加载
评论 #13354836 未加载
评论 #13354893 未加载
评论 #13354900 未加载
cekvenich3over 8 years ago
Nope. SD should not be using DB. They should use API to make REST | Microservices, ex FireBase. SD should not be using servers. They should go serverless, deploy static HTML to CDN&gt;<p>etc. If you are still using docker, puppet or worried about scale&#x2F;security... you are living in the past. (one ex: <a href="https:&#x2F;&#x2F;github.com&#x2F;struts3&#x2F;what-demos" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;struts3&#x2F;what-demos</a> )
评论 #13356769 未加载