Thank you for keeping systems available and safe. I've been there many times in the past, including having to fly at the last minute to a non-internet-connected data center in NJ to babysit an emergency production bug fix that took the entire holiday to create, install, verify, and monitor.
Always be kind, and say it’s your fault.<p>If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself:<p>> In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works. You want to know what it is? When I call, I say, “Hello. I am Hans Rosling and I have made a mistake.” People immediately want to help you when you put it this way. You get much more when you don’t offend people.<p>[0]: Unless you are in charge of a developing country’s budget and have to decide between education and healthcare.<p>[1]: <a href="https://blog.ted.com/qa_with_hans_ro_1/" rel="nofollow noreferrer">https://blog.ted.com/qa_with_hans_ro_1/</a>
Yeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january.<p>December tends to be hell for our customers, so stability should be a priority there.<p>And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some unnoticed issues and then just lay down the tools. Use that time for documentation, or shorter days, quite frankly.<p>That way we minimize the on-call situations occuring. Let's hope it goes well for the engineer this year as well. We have a streak to keep.
Yes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)
I get nervous every New Year’s Eve due to date/time issues. I work on emergency 911 software. In our system each time a 911 Call is created, we create an incident number in the format YYYY-NNNNNNNNNNNN where N is an incrementing number. I was oncall a few years ago when a date time bug was introduced that resulting in numbers being created prematurely by a few hours. As each hour passed more customers in a different time zone called in to report the issue. I was the only person working and was getting hammered with cases.<p>It sounds like an easy isssue to correct, but downstream systems that consume those numbers had already processed them and associated reports and other records with the incidents. I spent the next few months sorting out that mess and helping work with partners to clear out data.
On call sucks so badly. At this point of my life, I firmly believe that there's not enough amount of money that can compensate the mental suffering it implies. Even more if the company you work for has this mentality of "deal with it" without making improvements, which was my case in the last period I did on call and what made the camel's back to break for me. Nowadays I simply refuse it. For those who are still on the trenches, stay strong, never resígnate yourself to just "deal with it" and thank you.
Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns:<p>It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
My D-I-L is a nurse and will be working the coming weekend. That's her "price" for getting Thanksgiving (in the US) off. We'll schedule as much as possible around her work hours and make sure there's food left for her when she gets home.<p>Many thanks to all of the health care workers who take care of us over the holidays. (Along with all of the others, of course.)
My mum always worked on most of the public holidays because we never really cared about them but it did mean everyone owed her favors (she was a physician and would be on call, at the hospital etc) so she could get someone to cover for when whenever she wanted.<p>Growing up ignoring holidays is mostly great (fly on xmas and everybody feels sorry for you, even though <i>they</i> are the ones working on xmas). But it causes relationship problems bc even when you genuinely try to participate you’re “doing it wrong”.
And for all those of you trying to hold the world on your shoulders: don't be a hero. If you don't let things fail (that aren't your responsibility), nobody will notice it's at risk of failing, and thus will keep letting you hold it up by yourself.
Is it a US thing to push updates right before holidays and force people to be on call?<p>European companies I've been working on are planing releases till the end of November, first week of December max. All project plans are baked like there is one week in December.<p>While US company tried to force every contractors to work everyday and some weekends too. Are you Indian? You don't celebrate Christmas, you'll be on call. Are you Eastern Slavic? You celebrate Orthodox Christmas in January. You'll be on call.
My uncle used to work for the City. Every holiday season, if it snowed, he would be called away to plow the roads.<p>Here's to those out there plowing the roads so we can get there safe!!
FYI Israelis are not on holiday - our holidays are on whole different dates. Hire Israelis and experience no down time while working with Silicon Valley level talent
I announced a downtime for a smallish GPU Cluster starting from christmas eve just a few hours ago. It is just the perfect time to schedule a day or two of downtime for a system like that. And if IPMI doesn't fail me, I can get a lot of things done without leaving the comfort of my home. I scheduled this without pressure from my boss. It was a totally voluntary decision... While being raised as a Christian, this time of the year is for me more about solstice then about the Christian clelbration. A time to enjoy the comfort of a heated home. A time to celebrate that the days are going to be longer from now on again. A time to reflect on the past year. And all of this is easily done while having a few terminals open and waiting for remote stuff to complete...
Holidays are excellent times for hackers to take advantage. It’s not just Christmas or other Western holidays, either.
Extend this principle to any holiday/world conflict/anniversary of conflict made into holiday/calendar new year and then adjust your time of attack.<p>protip: US companies with offshore groups are usually underfunded, understaffed, and underskilled. Time to see if that disaster recovery environment works!<p>Happy holidays to those who encounter system stress tests. Can’t spell salary without some elements of slavery…
Some people just wake up in the morning wanting to fight. If you're providing support, you might just be the first person they find on a given day.<p>It's crazy to want to fight someone who is trying to help you, but lots of people end up in crazed states when hammering away at projects behind their computer terminals.<p>Taking a step back even further, I think that requiring any form of software support or correspondence with software makers is a completely alien concept. I have done it under 5 times in more than 30 years of using computer software and writing documentation.<p>No doubt that countless people encounter show stopping software issues for which they cannot find alternative tools, workarounds, or workflows, but I find that it's almost always better to just go shopping at these junctures, rather than enter into some kind of chain of correspondence hoping that someone out there in the ether might possibly one day fix your problem.
In a similar vein, I'm grateful for the people who maintain the foundational pieces of our digital world that often go unnoticed like date & time systems.
I worked four years in the military, and three of those I had evening and night shifts during the holidays.<p>Absolutely nothing happened, no activity whatsoever - just babysitting systems deep inside a bunker. Closest I got to new year's eve was watching the fireworks through CCTV.<p>The last year I was on call, which was miles better, but those years definitely cemented my will to get a job with normal work hours.
I remember playing Unreal for 3 days solid while working an IT service desk in the late '90s. Compaq Deskpro in software mode later upgraded to a Matrox G200 which upended my world. Working Christmas and New Year was a bonus as far as I was concerned. Dodged the family drama and got in plenty of gaming. Also tidied the office...
I'm not sure if there is something in the water this year, but this week, Dec 18th to Dec 21st (only a partial week), has been our busiest week all time already.<p>Sweating over here trying to make it through the week and praying that it slows at least for the first half of next week.
Thank you indeed! While I have fond memories of working holiday pager support early in my career, especially before marriage and kids to cover for those with families, I’m very grateful for those able to cover for all of us now! Cheers to you all
Just remember this time of year is often peak vulnerability time. When attackers exploit that teams are at reduced strength and off guard. Slower response times to investigate and fix issues etc.
parent company got hit by cyberattack yesterday. salute to all IT and InfoSec colleagues working around the clock while everyone else is taking the day off.
Managers, if you're reading this, and you have engineers/developers "on call" but not contractually (off book), make it so, because it slightly sucks when you're having Christmas drinks but can't enjoy yourself because you might need to drive somewhere and climb up a ladder to tend to a product.
Firefighting sucks. I salute all of the ways people ensure that on-call runbooks are thoroughly documented, not shipping code on Fridays, and robust, simplicity-based engineering that reduces problems during and especially afterhours.
It tends to be a tradition for me to be oncall over at least one of the winter holidays, this year I (again) get to preside over a vendor vs vendor showdown where the only loser is our hardware!
<p><pre><code> non-internet-connected data center
</code></pre>
Woah. That sounds wild. If you are allowed to share: What industry / company? I would like to hear more about that setup!
If you’re atheist or of a religion that isn’t relevant then why do you care? There’s no spiritual meaning. It’s just a day off. Get off the fence and own your convictions.
As if this week had attempted to take a measure of blood from my body, I'll be on-call next week. Looking forward to all things quiet on the HEP network front.
> Thank you for keeping systems available and safe.<p>theres that word "safe" again. What systems are dangerous otherwise? Do you mean like traffic lights or something? The API serving ads to your mobile game isn't dangerous.
Thanks. Honestly, all I do is check emails once or twice a day and maybe respond to a DM if there's an emergency, but it's nice to be appreciated.
Agreed, there are some gigs that just really require support to exist - I know this first-hand from working at a Zoo (very large exotic animal rescue basically). Animals do not take holidays. They need to eat and do animal things in spite of our costumes that day.<p>On the flip side, having worked Cinema on Christmas Day two years I think, there is no amount of Grace and Patience I can give that is enough to those earning their living. Still have a hat and polo. Why? I had to buy them!
Thank you but I wish you were not doing it during your night time. Let someone in their timezone for whom it's daytime do it. Insist on it, refuse being exploited. Take this opportunity to tell your company to hire the time timezones where it's day during your night. If it's not important enough for them then they anyway don't need to you remain awake at night. Don't harm yourself. It's not just losing sleep, that leads to harmful effects in your body that keep showing for long and some damages might not even be reversible.<p>It's not bravery, it's not being a hero as many are putting in comments - it's just plain exploitation if you are awake at night doing this.<p>Your health and wellbeing is not worth any money and definitely not worth someone's shopping cart and checkout page working smoothy over the holidays when your Western colleagues just took off en masse. Not even close. I know a lot of you would be from a third world country like me (yes, that's a thing!). Stay strong and work for this exploitation to end.