"Out of Band" network management is not trivial

In the entire history of the Bell System, no electromechanical exchange was ever down for more than 30 minutes for any reason other than a natural disaster. With one exception, a major fire in New York City. Three weeks of downtime for 170,000 phones for that.[1] The Bell System pulled in resources and people from all over the system to replace and rewire several floors of equipment and cabling.<p>That record has not been maintained in the digital era.<p>The long distance system did not originally need the Bedminster, NJ network control center to operate. Bedminster sent routing updates periodically to the regional centers, but they could fall back to static routing if necessary. There was, by design, no single point of failure. Not even close. That was a basic design criterion in telecom prior to electronic switching. The system was designed to have less capacity but still keep running if parts of it went down.<p>[1] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=f_AWAmGi-g8" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=f_AWAmGi-g8</a>

One thing that was fascinating about the Rogers outage was on the wireless side: because &quot;just&quot; the core was down, the towers were still up.<p>So mobile phones would try to make a connection to the tower just enough to connect but not be able to do anything, like call 9-1-1 without trying to fail-over to other mobile networks. Devices showed zero bars, but field test mode would show some handshake succeeding.<p>(The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we&#x27;ll never know)<p>Supposedly you could remove your SIM card (who carries that tool doohickey with them at all times?), or disable that eSIM, but you&#x27;d have to know that you can do that. Unsure if you&#x27;d still be at the mercy of Rogers being the most powerful signal and still failing to get your 9-1-1 call through.<p>Rogers claimed to have no ability to power down the towers without a truck-roll (which is how another aspect where widespread OOB could have come in handy).<p>Various stories of radio stations (which Rogers also owns a lot of) not being able to connect the studio to the transmitter, so some tech went with an mp3 player to play pre-recorded &quot;evergreen&quot; content. Others just went off-air.<p><a href="https:&#x2F;&#x2F;www.theregister.com&#x2F;2022&#x2F;07&#x2F;25&#x2F;canadian_isp_rogers_outage&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.theregister.com&#x2F;2022&#x2F;07&#x2F;25&#x2F;canadian_isp_rogers_o...</a>

I’m reminded of when an old AT&amp;T building went on sale as a house, and one of its selling points was that you could get power from two different power companies if you wanted. This highlighted to me the level of redundancy required to take such things seriously. It probably cost the company a lot to hook up the wires, and I doubt the second power company paid anything for the hookup. Big Bell did it there, and I’m sure they did it everywhere else too.<p>Edit: I bet it had diesel generators when it was in service with AT&amp;T to boot.

It’s trivial when you have the resources that come from being one of Canada’s 3 telecom oligopoly members.<p>Unfortunately the CRTC is run by former execs&#x2F;management of Bell, Telus, and Rogers, and our anti-competition bureau doesn’t seem to understand their purpose when they consistently allow these 3 to buy up and any all small competitors that gain even a regional market share.<p>Meanwhile their service is mediocre and overpriced, which they’ll chalk up to geographical challenges of operating in Canada while all offering the exact same plans at the exact same prices, buying sports teams, and paying a reliable dividend.

There is OOB for carriers and OOB for non-carriers. OOB for carriers is significantly more complex and resource intensive than OOB for non-carriers. This topic (OOB or to forgo) has been beat to death over the last 20 years in the operator circles; the responsible consensus is trying to shave a % off operating expenses by cheaping out on your OOB is wrong. That said it does shock me that one of the tier-1 carriers in Canada was this... ignorant? Did they never expect it to rain or something? Wild.

When I see out of band management at remote locations (usually for a dedicated doctors network run by the health authority that gets deployed at offices and clinics) it&#x27;s generally analog phone line -&gt; modem -&gt; console port. Dialup is more than enough if all you need to do is reset a router config.<p>Not 100% out of band for a telco though, unless they made sure to use a competitors lines.

Reminds me of a data center that said they had a backup connection and I pointed out that only one fiber was coming into the data center. They said, &quot;Oh its on a different lambda[1]&quot; :-)<p>[1] Wave division multiplexing sends multiple signals over the same fiber by using different wavelengths for different channels. Each wavelength is sometimes referred to as a lambda.

My previous org OOB used a data only SIM card from a different service provider. Curious why that wouldn&#x27;t be a good solution?

From TFA:<p>&gt; If your OOB network is your only way of managing things, you not only have to build a separate network, you have to make sure it is fully redundant, because otherwise you&#x27;ve created a single point of failure for (some) management.<p>I&#x27;m not sure I necessarily agree with that. You can set up the network in such a way that you can route over the main network as a backup if your OOB network was down but the main network was up. Obviously, it&#x27;s not quite as simple as sticking a patch cable between the two networks, but it can be close - you have a machine that&#x27;s always on your OOB network, and it has an additional port that either configures itself over DHCP or has a hard-coded IP for the main net. But the important thing is that you never have that patched in, except for emergencies like your OOB network cable being severed but you still have access to the main network. If that does happen, you plug it in temporarily and use that machine as a proxy. There&#x27;s no real reason for extra redundancy in the OOB, because if your main uplink is also severed, there&#x27;s not really much you&#x27;re going to be usefully configuring anyway!

Apart from Rogers et alike, the main OOB&#x2F;LOM issue is that&#x27;s mostly only very old iron very few know, finding people who knows and finding non-hacky homegrown and not much tested solutions it&#x27;s damn hard.

With launch costs dropping, I wonder if there’s a market for a low bandwidth “ssh via satellite” service. Could use AWS Ground Station to connect to your VPC.

<i>&gt; hardened in-band management</i><p>What would this look like in practice? Management interfaces like BSPs don&#x27;t have a great security track record.

I love ChrisO so much, and it&#x27;s funny but often he&#x27;s talking about something I&#x27;m currently working on too.<p>Thank you to Chris and to whoever posts his articles here.

The blog post is weird. &quot;Rogers didn&#x27;t even try, so OOB is hard.&quot;<p>Also this sentence makes me question his IQ:<p>&quot;Some people have gone so far as to suggest that out of band network management is an obvious thing that everyone should have&quot;<p>Yes Chris, Rogers, the monopoly telco company of Canada, should have OOB network! They can afford it.<p>Talking about the challenges of OOB is great, but the point the blog post is wrong and dumb.<p>The report says &quot;Rogers had a management network that relied on the Rogers IP core network&quot;. They had no OOB network. They didn&#x27;t even try.<p>This is a a symptom of Rogers status as a monopoly, negligence on the behalf of Rogers, and negligence on the behalf of the government who should have regulated OOB into existence. This is some serious clown car shit.<p>One of the advantages that competitor networks provides is redundancy. Canada doesn&#x27;t have that, so their networks will remain weak. This will probably happen again some day.<p>Yes OOB is hard, but not even trying and then throwing up your hands and defending the negligent is stupid.

Ham radio. Meshtastic. Knowing your neighborhood.

I disagree, Out of Band Network Management (OOBM) is extremely trivial to implement. Most companies however don&#x27;t see the value of OOBM until they have a major fault. The setup costs can be high, and the ongoing operational costs of OOBM infrastructure and links is also significant. I&#x27;ve built dozens of OOBM networks using fibre and 4G with the likes of Opengear. In instances, often deploying OOBM ahead of infrastructure rollouts so hardware can be delivered to site directly from factory, rather than go through a staging environment which adds time, cost and complexity.

I worry that this misses the point a little. All the OOB in the world will not help you if you cannot reach the management entity (eg IP-enabled PSU, terminal server, etc). It is also insufficient to protect against second order thundering-herd-type problems (e.g.: you log in, stop a worker process, and upstream, traffic is directed away from the node to the others, and starts causing new problems).<p>In telco operations, every MoP should have: an unambiguous linear sequence of steps, a procedure to verify that the desired result has been achieved, and a backout plan if things do go bad. This is drilled into you at every telco I ever worked at. Rogers&#x27; cardinal sin on the day of the outage was that they didn&#x27;t have a backout plan at each step of the MoP.<p>More structurally, networks have a dependency graph that you ignore at your peril. X depends on Y depend on Z, and so on. And yes, loops are quite possible! OOB management is an attempt to add new links to the graph that only get used in a crisis. These kind of pull-it-out-when-you-need-it solutions are fine, but have a tendency to fail just when you need them. For one, they don&#x27;t get exercised enough, and two, they may have their own dependencies on the graph that are not realized until too late.<p>So, what would this Internet rando prescribe? First order of business is to enumerate the dependency graph. I would wager that BGP, DNS, and the identity system are at or near the very top. Notice the deadly embrace of DNS and ID: if DNS is down, ID fails.<p>Next, study the failure modes of the elements. In the Rogers outage, a lack of route filters crashed a core router. That&#x27;s a vague word, &quot;crashed&quot;. Are we talking core dumps and SEGVs? Are we talking response times that skyrocketed, leading to peers timing out? Rogers really need to understand that. Typically in telco networks when nodes get &quot;congested&quot; like this there are escape valves built into the control plane protocol, eg a response that says &quot;please back off and retry in rand(300)&quot;. They need to have a conversation with Cisco&#x2F;Juniper etc and their router gurus about this.<p>Finally, the telco industry (or what&#x27;s left of it) needs to do some introspection about the direction it is pulling vendors. For the last 15 years, telcos have been convinced that if only that can ingest some of that sweet, sweet cloud juice, their software costs will drop, they can slash operations costs, and watch the share price go brrr. Problem is, replacing legacy systems with ones cobbled together by vendors from a patchwork of kubernetes and prayers is guaranteed not to lead to the level of reliability that telcos and their regulators expect. If I&#x27;m a Rogers&#x27; operations manager and my network dies, I don&#x27;t want to hear that some dude in India has to spend the next week picking through a service mesh and experimenting with multus to decide if turning if off and on again is gonna work.

Who said it is trivial?... Edit: The article take a title and describe some straightforward technical and business investments to make oob management network work.