I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated.<p>I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign.<p>There is absolutely no situation in which I will <i>ever</i> participate in another on-call shift. I've been there, I've done it, now that chapter of my life is closed. Find some younger kid, pay them better than you paid me for the miserable intrusion on their life. I'm done.<p>Just wanted to be the voice who says what, hopefully, some of the more seasoned and battle-scarred readers here are thinking.
My team does Wednesday to Wednesday for many of the same reasons mentioned in the article, and it works great. We switch at 11am and hold a hand-off meeting at that time, and invite the whole team.<p>Hand-off meetings with the whole team work really well (in my opinion!) when you have a relatively small team--we have 9 FT teammates. Often someone else may have been delegated the page or bug that arose and can discuss how they handled it, or someone who wasn't involved may have insight for how to handle a situation better the next time. Since we're all going to be on rotation at least once during a quarter, it's great to know what happened in case a similar page pops up later.<p>Finally, we also fill out a running Doc before/during the meeting with links to the pages/bugs, along with short descriptions of how they were handled. This forms a great living memory of how to deal with incidents, and is also often the birthplace of new playbooks for handling new types of incidents.
We do Thursday to Thursday and then you get Friday off after completed on-call.
Being on-call gives you no extra pay by itself, but if you get paged off hours and need to work you get paid 150 to 200% of your normal hourly wage depending on what time of day you need to work.<p>Best on-call I’ve had.
Here in France, we have strict laws, and on-calls MUST be paid in some form or another. When we were bought by a US company, mgmt tried to set up on-call shifts for us - we had never needed them for the 10 years prior -, until they learnt of labor laws and went "fuck it, you're on call mon-fri, 10am - 6pm". I'm forty, have a family, and no amount of money could justify that I can't shutoff my phone at night, or prevent from going on a walk on weekends because "uptime". I've never been so glad of french worker protections.
I hate on-call shifts, but if they must exist, I like the way my team handles them. We have split day and night shifts. 7-18 day shift, and 18-07 night shift.
All non-work hours compensated with standby at 10% of hourly pay. Any pages outside of work hours earn you an additional 150% in base pay. Each page guarantees a minimum of 3 hours of pay even if you spent only 5 mins on it.<p>And since in my country, you must gave at least 11 hours between shifts, if you get paged at night, you get PTO for the next 11 hours on top.
> But websites need to be up 24/7, cron jobs need to run on the weekend and backend servers need to be up to support both<p>Tech entrepreneurs should give more weight to choosing markets that don’t require this
When did on-call become so accepted and demanded from employers?
Currently I am "Release Captain" for a week: So I have to setup any releases and manage all the related tasks, do automated/manual testing of the release, release (enabling toggles and any config changes).
Then Backup to secondary and primary for a week: About once or twice I am asked to help with tickets.
Then for 14 days we alternate primary / secondary.
Thursday to Thursday is our deal. Every ~40 days I am in one of the above.
It's absolutely miserable.<p>I have never had this much time spent doing non-development related tasks. For 4 weeks every 1.5 months I can't have a life at all.
This just screams to me that we are forcing broken software/not complete software out the gate a building huge piles of technical debt that will never get the focus.
I remember a time when I would start at 9am and end at 6pm every day and never heard a peep about production issues unless the support engineers couldn't figure it out. Which maybe happened twice a year.
To make matters worse most things are <i>not</i> allowed to be touched in production with the risk of being fired for making changes. So if you want to "fix" any data or call xyz service you need high ranking approval. It's like being tortured!
I'm not sure if my team is a crazy exception, but here's how our on-call works: We're usually on from Monday-Monday (with exceptions if we say, don't have time on Wednesday or something), but every team decides the time on their own. During work-hours, every team member is responsible for responding to alerts (but usually, only the on-call engineers will carry their company-provided phones and are the first to respond).<p>Outside work-hours? Most alarms (if they happen) are due to bad alarm configurations. Because nothing ever happens. There was one alert this month, and it was because a randomly generated ID contained the string "ERROR" and was logged due to a warning.<p>I know that my company isn't the "biggest" (only a few hundred requests per minute) and traffic amount is mostly correlated to usual business hours in my country, so there's just not much happening at night (but never zero traffic). Still, I'm always surprised that other companies seem to have really stressful on-call shifts, because the most annoying part to me is having to carry my laptop if I leave my home for more than 20 minutes.
Here's my on call schedule: never, and don't ask. It's my responsibility to do my job when I am scheduled, and it's management's responsibility to staff properly. If we can't agree, then we can't have a business relationship.
We do daily shifts with a follow the sun rotation, makes it easier to handle persistent commitments and ensures a bad week doesn't all land on the same person.
I work in a small team, we have ~30 active clients and other ~100 or so that are low maintenance or dormant.<p>We don't have an on-call rotation but I desperately want one. Because if no one is on-call - then all of us are on-call. Any one of us could be called at any time if one of our larger commerce projects falls over.<p>To me on-call is a necessary burden that means when I'm not on duty I am completely free to ignore my phone.<p>I'll certainly feel more positive about helping outside of the 9-5. I do like to be helpful, but perhaps that'll wear off like some kind of honeymoon period, juxtaposed to my current situation.<p>I'm always looking for more positives in such a system because I want it to work. Tuesday to Tuesday sounds great. Other comments here highlight the difference between critical fixes and patch it laters. Any other insights are welcome.
I’ve always been partial for Friday night through Friday night.<p>You start off over the weekend, when you have energy and can survive the two days alone. Ideally no Friday releases so the transition is calm, but as the writer says the batches might fail.<p>You spend the week fixing whatever breaks. You’re cleanly off the Monday to Monday sprint, just doing on-call/ops.<p>You finish Friday evening and immediately get Friday night and the weekend to recover when you need it most.
Each person on my team has a day of the week they own, and then we have a rotation for weekends, and negotiate holiday/pto trades. I guess it really only maps correctly for a 5 person team.<p>We previously had a week long rotation, and some folks were initially skeptical of the idea to change, saying they were worried they'd feel like they were "oncall all the time". But, they agreed to try it for a month. That was a bit over a year ago now, and no complaints.<p>I think it ends up being a lower stress configuration, because it just becomes part of your normal expected work-week routine, and generally isn't as mentally draining.
It does make end of year PTO/holiday time a bit more complex to work out, but so far my team has been okay with that tradeoff.
On call should be reasonably compensated. IMHO all other discussions of on call should take place after that is resolved. Instead, developers are expected to work unreasonable hours and are then fired when they start to burn out.
I’ve had to schedule on calls for a team and this would make it much harder to make a schedule. Every week’s vacation now means two weeks that person can’t be scheduled for on-call.
I have occasionally convinced teams to adopt <i>both</i> oncall and sprint cycles aligned with Tuesday [1] - the dev teams all loved it. Management was a harder sell, but by and large were happier with the extra days to communicate results/get metrics before their own Friday deadlines.<p>[1] also Wednesday/Thursdays. Wednesdays were my favorite in good working environments, it felt like running a successful marathon, but it was more prone to falling apart due to short-term thinking.
At a previous employer we had a pair on call for each of: front end, back end, and infra. We had on-call lasting from Monday midday - Friday midday. Handing off to a "weekend on-call" from the same pool of people from Friday midday to Monday midday. Weekend on-call paid 100 per day, weekday on-call paid 50 per day. You were generally expected to take normal time "off" (but still on call) if paged off hours. Many people would still work if it was just a blip (rare).<p>I thought this was a pretty good system and despite the cycles being shorter, we had enough engineers to fill a rotation pretty well so that at most you were on call once a month, alternating months between weekend and weekday on-call cycles.<p>I still do not enjoy being forced into on call and wish I could opt-in. We traded weeks a lot but with smaller rotations or really finicky paging its awful. I still have a sinking feeling in my gut when I hear the work phone ringtone from somebody else's phone in public, and murphy's law definitely applies to being on call -- you always get paged the minute after your beer gets delivered at a restaurant.
I never understood why companies didn't simply leverage 24x7 internet MSPs.<p>They are able to staff 24x7 by spreading the cost over multiple customers and working through the process of making your application manageable by a 3rd party is super beneficial.<p>Most of these companies will also do performance monitoring and analysis as well.<p>They see issues and optimization opportunities across multiple applications and know more than a single team who's only built one.
This is a strong positive imhe:<p>"- Step 1: handling it<p>- Step 2: making sure it doesn’t happen again<p>So when a major issue happens over the weekend only Step 1 happens during the weekend. Step 2 involves following up with other teams, creating new alarms and updating the runbook. And all that usually happen during the week. The oncall is going to spend at minimum their Monday doing that so it’s better if the schedule reflects that."
On a past team I set up on-call to be:<p>- Mon/Tue
- Wed/Thu
- Fri
- Sat/Sun<p>Original reason for this schedule was that on-call was paid by days per quarter in a tiered system so this guaranteed that all members got the 5% on-call for 10 days/quarter rather than one person hitting 9 days and dropping to 3%, but I stand by this as a better on-call rotation.<p>The number of people does need to be not wholly divisible so the days rotate so if you run into this you can combine Fri into Sat/Sun or break Sat/Sun apart. It’s a bit complex to set up but the mental impact of on-call is greatly reduced and if you need a week for vacation you can much more easily find someone to cover your shift for a couple days in a nearby week rather than ending up with 2 weeks back to back 6 weeks from now. And if you pull a weekend you get the week off rather than losing your weekend to on-call and going into a work week still on-call.
I'm not IT, but the hospital I work for has the resident physicans on call Tuesday to Tuesday and the attending physicians Monday to Monday. I have found it to be a good system, with the weekends being covered by people who have a good idea of their patients.
We are on-call for 48hrs at a time, about once every 12 days or so, one day as backup, and one as primary. It's nice because it doesn't interrupt your week too much. The downside being that complex issues might require extra work while not on-call
The point about holidays resonates with me. Our setup you get paid extra for being on-call for a public holiday, but given we do Mon-Mon shifts in practice that means two people can't take advantage of a long weekend and only one of them gets extra compensation for it.<p>Different people deal with being on-call differently but personally I don't do what I normally would when on-call, whether that's long motorbike rides or hiking etc because it's not practical to guarantee cell coverage and also the threat of a page ruins the experience. A "day off" whilst on-call isn't equal to a day off
I always do Wednesday to Wednesday. I think I’d rather fix something than having it hanging over my head. That said, I prefer systems that don’t require constant attendance. But when I was young, I reveled in that. You get to have a lot of fun firefighting and using each chance to make sure the fire can’t be lit next time. Thoroughly enjoyable and eventually the fires reduce.<p>I still think of each error as a possibility of improvement to asymptotic zero error and I prefer working with people like that.<p>Others prefer other systems and I think that it’s fine for each of these groups to select for members appropriately aligned.
I’ve also done Wednesday to Wednesday, though Tuesday seems better mentally if only because there is one less day after the new week starts.<p>What is much better, though, is splitting the week into a 4/3 or 5/2 split, with a primary and backup on-call. Primary takes the weekdays, then switches with Backup for the weekend. You’re still sharp and aware of any current issues should the need arise, but the odds of a weekend page are (hopefully) lower, so you can relax a bit.<p>This of course requires enough people to have a reasonable rotation; 6 at a minimum, but 8 is better.
I’ve been in so many meetings endlessly searching for the Best Day for on-call shifts to start/end, and feel I have heard it all at this point, every day of the week seems to have pros and cons. Current team just does Monday, because that’s when the week usually starts, and I really don’t want to talk about it anymore lol. If it conflicts with one of the abysmally few US holidays, adjust accordingly, you’re a bunch of smart clever adults, you’ll figure it out.
In my 20ish years I’ve done every possible day for oncall schedules. I would say each have pros/cons but overall I found it to be a minor difference.<p>Mon-Mon is nice because it’s a logical time to start something fresh at the start of the week.
Tuesday is good for the reasons in the post, Wednesday is similar.
Thursday is nice because after you’re done you can relax on Friday.
Friday-Friday is less common but can be nice because you get the satisfaction of being done on the last day of the week.
We used to do two week on-call rotations (to follow along with sprint/release cadence). At the beginning of '23, one of our devs asked to be full-time on-call, both because they had sleep issues and so would regularly be awake at 2 am, but also because they didn't like that the other team members wouldn't follow the documented processes.<p>They don't get paid extra, but they seem to be very happy with the setup.
We started a split shift for a really busy oncall and it works out really well. It's Th->Tu, Tu->Th. So basically weekend+2 working days vs 3 working days.<p>Expectation is you are 100% oncall during the working day, so it works out pretty well between weekend vs non-weekend shifts.<p>I much prefer the shorter shifts to a full week. A full week on-call usually means delaying important project work, etc. for a full week.
When I just started out, I used to take everybody's on-call service because it paid well. Also had the advantage that I learned the ropes quicker. Now-a-days I'm very happy to <i>not</i> be awoken at 3am, even if 1 weekend incident would amount to a couple grand.
Oncall for barely any calls. If I have to do anything I just add that as overtime, simple as. Was a bit odd to have to carry my work phone everywhere for the first two weeks or so but now it's just standard procedure
A few months ago we switched from Tuesday-through-Monday to Monday-through-Sunday and on call stress decreased.<p>After a weekend of on call, it sucks to have yet another day of on call on Monday. This overpowered all other reasons (most of them listed in the blog post) for us.
100% agree, especially when you have to deal with distributed teams in the UK with all their "bank holidays" which all seem to land on Mondays.
> Highly scrum/agile focused people have brought up that sprints start on Monday and that starting the oncall on Tuesday makes sprint planning harder<p>Wait, what? Don't run your sprints Monday to Monday either. That's been the eventual conclusion on all scrum teams I've been on.