I have designed/coded two redundant systems before.<p>First one is super complex redundant router done by 100+ engineers. All TCP states, BGP states, complete configurations were replicated to backup CPU. If the primary failed, power off, etc, the standby will take over, etc. When the failed unit comes backup, it become the new stanby. Demo works fine, but the company sold just 1 system for $200K. The company did sold for four hundred millions. It paid off my mortgage.<p>2nd is relatively simple done by me in 3 weeks including testing, etc. It is a system convert 48 channels digital MPEG streams to analog NTSC signals for analog cable plant such as Comcast. Xilinx power PC running linux. All configurations were sync. Auto switch over on power off, network disconnect, etc. The system can detect all failure conditions in 50 milliseconds. Very cool to demo - if you blink your eye and you will miss the system failure over event. Support in service SW upgrade - update backup to new SW, sync config over, force switch the primary and update the new backup to new SW with ZERO downtime.<p>The product was very successful - only one firmware bug found after two years in the field. Sold $100 millions+ to various cable companies. But the VC air drop a "professional CEO" and "scale" the company from 16 people to 300+ people. Did two more rounds and manage to burn the company to the ground and sold it for pennies.<p>Redundant system design are not that hard and can be simple. It is best to remember KISS - "Keep It Simple Stupid".
I was called in to troubleshoot system once, a small 2-node cluster with a SAN with no single point of failure in the fabric.<p>They had both their 1500va UPSs plugged into a single 15amp circuit.
Yes, it is indeed a problem when all replicas run the same software.<p>One approach that's been suggested is to use multiple independent implementations of the program. Or, if that's too onerous, N-Version Programming (<a href="https://en.wikipedia.org/wiki/N-version_programming" rel="nofollow">https://en.wikipedia.org/wiki/N-version_programming</a>).
I like the word <i>antifragile</i> instead of redundancy. I don't operate a datacenter, but in my own personal computing space, I try to build antifragile systems that can bounce back from failure rather rapidly. It is astonishing for example, just how many times I have to roll back a Windows 10 install because it likes to randomly break for whatever reason. It could be anything: a faulty update that corrupts the whole OS. The hard-drive gets filled up really quickly when doing gaming video captures, or games that do really heavy writes to the SSD and wearing it out within a year of use.<p>Currently I use virtual machines to mitigate this, and you need a beefy setup to do this. If a virtual machines fails, I have a 'template' virtual disk image where I can start afresh. One caveat to VMs is you can't do gaming since you're emulating an OS, so when I can I use a bare metal solution for gaming. You can do basic gaming for games that use low resources (like Minecraft). But forget about playing Crysis or other monsters in a VM!