Expect this to happen on more services, too. Rackspace customer are reporting uncontrollable downtimes (because the physical hardware is down) ranging from "a few minutes" to "six hours." Search twitter for [rackspace reboot].<p>Also, since Rackspace didn't communicate it that effectively: if you're on their first-gen infrastructure (your servers have an asterix next to them in the list), you won't get rebooted. I had to hear this from a fellow customer who heard it from support, as I was searching my inbox frantically wondering why they hadn't given me a heads-up about an incoming-any-minute-now reboot yet.
I got bit by this over this weekend, too. I received the initial email September 27 @ 1:35 AM local time. I called Rackspace to see if they would be sending out emails prior to each VPS being rebooted 30 minutes or so before just as a heads up for human monitoring purposes. The representative told me that it "was not feasible" for them to do that for every VPS. Instead... I've been camped by my computer all of today (Sunday) to monitor the reboots since I have servers at DFW & ORD and they have a 24 hour time window for those regions.<p>While their status page was somewhat helpful, I find it absolutely absurd that they can only update it once every 60 minutes to cue customers in. In addition, their Cloud Control Panel doesn't reflect the reboots. When a VPS goes down for a reboot... the CP shows the server as "Active - Online and functioning as properly". Thankfully great third party monitoring services (using Scout) exist so they can notify in-place of Rackspace's incompetency.<p>As someone who shells out a significant amount of money to them each month, this is pretty disheartening. That being said... I seem to have survived the Great Rackspace Reboot of 2014 and can only hope they handle the next event better.
We're a MG user via Heroku.<p>Our Heroku MG logs indicate that all messages are getting "Delivered", but that doesn't match reality. We've been testing with our own accounts -- ones that receive copies of all automated emails, as well as our personal accounts.<p>The Heroku MG logs says "Delivered" for all the emails.... but using 4 different addresses across 4 different carriers confirms: a VAST number of emails (since noon today) were not actually delivered. The only change in configuration: MG's downtime. I seriously hope all of these "Delivered" emails are re-sent. If someone from MG could weigh in, that would be fantastic! (We have a ticket filed, but email also in profile).
Mailgunner here: usually we just redirect traffic from one environment to another, but this time we are having unexpected networking issues that are preventing us from that. We are still debugging the issue, stay tuned on status.mailgun.com
I don't get it. If a server restarts, don't you just get a different one? Why is that disruptive?<p>At work every machine reboots at least every month. Everything is designed to cope with that reality.
Ah, Rackspace with your 'fanatical' support!<p>I suppose it takes a fanatic to justify working week scheduled down time.<p>Anyone recommend a good dedicated server Rackspace competitor?
This is when I enjoy our new Google Compute Engine instances with transparent maintenance (thanks to live migration).<p><a href="https://cloud.google.com/compute/docs/zones#maintenance" rel="nofollow">https://cloud.google.com/compute/docs/zones#maintenance</a>
Hmm... is it just a coincidence that our school services portal had a semi-scheduled downtime this sunday from 6am-6pm, but we weren't notified of this until late friday... Also there is never maintenance for this and when there is the official time is 2-5am on friday....
To the admin who had to hit <Enter> on the script that kicked off these reboots.... I imagine this guy turning and looking the other way before clicking.<p>What poor execution on this...Xen updates can't be that rare that this is their first rollout.
I understand reboots are necessary, but why not migrate the instances from a node to a different node, reboot the node now that there are no instances running on it, and do this in a rolling manner?