The three pitfalls TFA mentions are the reason why I enforce the following rules around operations automation at my company:<p>1. Only that should be automated what can be done manually.<p>Before automating something, a sysadmin must be able to manually carry out the same operation. If they do not understand how it works, without using someone else's scripts, they are ineligible to administrate it.<p>2. What has been automated should always be doable manually.<p>After something has been automated, a new sysadmin should be able to read the automation code and re-do the steps manually. Maybe not at the full scale of repeating actions manually on every server, but enough that if the automation tools are irrecoverably lost they can be rebuilt from knowledge.<p>This also serves as a test for adequate documentation in the automation code. A new sysadmin should not have trouble figuring out why a certain thing is done a certain way in the scripts.<p>3. What happens, even when automated, should not be hidden from the sysadmin<p>Even in full auto, what's going on is always visible, so a sysadmin can jump in (i.e., go full-manual) at any step. This was one of our earliest rules, because we (me & my teammates) wanted to be able to debug without friction if anything went wrong. Also, it looks cool(er).<p>----<p>As a guiding principle behind how we automate operations, we assert that no amount of automation is replacement for a sysadmin's skills; it replaces only manual, error-prone effort.
He didn't forget how to manage a web server. He knew his problems, how to look for solutions and how to apply them. That's an important skillset to. Especially since it enables one to keep up with change. You're not a professional sysadmin if you only mindlessly keep using the same configurations that were taught to you.
Things he feels bad about are strange. I only remember how to allow root with key auth because I searched for it yesterday. "Not remembering how" is not the problem at all. "Not knowing what" is.
The thing is, memorization itself should -not- be the primary skill of a sysadmin.<p>A sysadmin should know how to troubleshoot, how to use tools, and (more importantly) how to learn how to use the tools.<p>I don't want to discount the value of someone who still remembers how to handle seemingly arcane tasks like iptables. Experience has its weight in gold, and memory definitely contributes to its value, but there's more to it than that. It is the capacity and willingness to learn in any situation, however, where a talented sysadmin really shines.
I think it’s ok to forget syntax and the like as long as you still understand the concepts.<p>If this guy had to work on a server for a week without any config manager, I’m sure he’d be right back at home.<p>Opposed to a guy who has never touched iptables. That guy is in it for the long haul.<p>Maybe a better title is “I forgot the commands(and some directory locations) I use to manager a server.”
I don't know how chef or puppet or that sort of scripts look like but my administration scripts are littered with: #Here I do A because B didn't work and we need it for C.
Or in fstab: # This is the disk I bought together with father. Or in crontab: # This syntax looks weird but otherwise it does not select the correct python env.<p>(I'm just a home-server admin ;) but that also means significant stuff only happens every 3-5 years at a new LTS or with some major problem, so notes to self are important.)<p>Such notes to self not only help when things go wrong, they also make for easy repetition of stuff years later. When I reinstall, I first back up the entire old root with all configs and thus all my notes (I could limit myself to /etc probably).
I would simply argue that traditional “system administration” is now the responsibility of development teams rather than ops teams (yeah yeah DevOps, but the “ops” part has changed).<p>Ops today is largely responsible for maintaining build / deploy pipelines, orchestration systems, and ensuring SLAs are met via SRE activities.<p>Most functioning DevOps teams I’ve worked with recently have added a more generalist role for a person who is a mile wide and a foot deep. It’s more of a hybrid sysadmin / development skill set ranging across base OS and package management, logging, scripting / automation, networking, access control, security, a dozen programming languages and whatever ITIL / EA platforms you have to interface with. These folks are a godsend in issue resolution as they know where the skeletons are buried. They also can pinpoint your top 5 tech debt issues of the top of your head.<p>The best version of this person also has some BA skills — they work really well as a demand management / intake person because they usually understand the end-to-end architecture — especially the code behind the integration interfaces — better than anyone on the team. They allow developers to focus on code, ops people to focus on production ops, and architects to focus at the right level of abstraction.
I’m not a sysadmin, but I’m remarkably bad at remembering boring things I don’t care about. I generally forget things like how to create systemd units or sshd config options a couple minutes after completing the task.<p>I don’t think it’s a big problem nowadays. As long as you know how to quickly get the information you need, it’s probably not that much more inefficient.
"If you cannot already do the machine’s job by hand, the machine will outwit you."<p>Raney Nelson
<a href="http://www.daedtoolworks.com/lounge-against-the-machine-daedlab-commandments-1-and-2/" rel="nofollow">http://www.daedtoolworks.com/lounge-against-the-machine-daed...</a>
I wonder if it is something to do with puppet. I've never used it (I wrote my own thingy before puppet existed), but I gather puppet comes with a whole pile of configuration thingies out of the box. For example, maybe you don't have to know how edit the sudoer's file to prevent password prompting because puppet has a puppet way of doing it that edits the config file for you. I could well imagine if you did things by editing puppet files for a few years you might forget the underlying config file syntax, or even what config file ends up being edited.<p>If that's true he'll pick it up soon enough. The muscle memory will start flowing again.<p>If he wants something to panic over try moving to k8s. If effectively includes a puppet like thingy done a completely different way, and worse it builds a very different computing model. Use that for 10 years, then try to re-adjust your thinking back to a single server model that you maintain by editing config files and you will be in for a rough ride.
It could be handy to write down some of this stuff you don't do often in a big org file (or plain text) for later reference. I find that some commands for irssi (tui irc client) are hard to remember, so I started writing a local irssiguide.txt that I look at or add to sometimes. I tend to add a new server or channel to autojoin right away, so then if I have a new thing to add 6 months later I can't recall the exact syntax.
I've been meaning to follow my own advice in other areas. Taking notes as you learn things so that you can relearn them later, or optionally show them to a friend, seems very useful. I also sometimes get stuck on something and end up giving up and forgetting what the problem was, meaning coming back to it later is that much harder. I'd like to get into the habit of writing down problems I have as well. e.g. figuring out how to mount the storage of an android phone easily
I rarely try and remember much more than there being a thing that does a thing, and then skim the reference to get the syntax correct when I need it. Things go in if I am using them regularly and go away again when I don't and I don't worry about it or think that I have forgotten how to do a thing. I also know that learning something again when it has disappeared almost completely, often leads to a deeper understanding of the thing, possibly because new connections have to be made before what is left of the original network gets lit up again, so even complete bewilderment at something I used to do regularly doesn't faze me all that much, I just get back on it in the knowledge that I will discover things I missed the first time round.
What it used to mean to say someone was good at math was that they were very good at quickly solving complicated math in their head. One documentary I saw claimed that this is what they looked for at Oxford and Cambridge for their math programs.<p>As soon as the calculator was invented (okay, and made cheap and portable), this skill was useless. Then you had to set up problems so that the calculator could solve them. (Similarly with the computer).<p>This is a natural and good progression. All the best practices should be learned and mastered by people building the tool, and you should just be proficient at the tool.
This is an honest question for sysadmins. What do you do if you go to a company that doesn't have Puppet or Chef? What do you say in an interview?
I've forgotten how to write assembly, but that doesn't mean I can't make the computer do the same things with higher order abstractions.
I've gone through similar scenarios... you hire employees with fantastic experience which includes config management. Everything is optimal until low level problems occur. A good example of this is software raid failing, and the best playbook/manifest writer doesn't know where to begin(without google).<p>The solution is <i>really really simple</i>. Don't hire people without an RHCE, or that haven't had one in the recent past. Feel free to substitute the RHCE for a comparable certification, which are few and far between.
> I had to Google the correct SSH config syntax to allow root logins, but only via public keys. I had to Google for iptables rule syntax and using ufw to manage them. I forgot where I had to place the configs for supervisor for running jobs, let alone how to write the config.<p>Are these actually things sysadmins know <i>without</i> googling or peeking into manpages?