At one place I remember that we had the usual scheduled backups for our MySQL databases. We also wanted a quicker way to bring systems back with less data loss, so we also had a lagged replica that purposely applied changes delayed by I think it was about 4 hours. The other handy thing was that swapping a replica for the write leader was a common operation that we did whenever we upgraded hardware or software versions so it was a well exercised procedure.
W. Curtis Preston, author of the O'Reilly book "Backup and Recovery" has said:<p>> Backups are useless. Restores are priceless.<p>I also like:<p>> The state of a backup is unknown and unknowable, until such time as it has been tested with a restore.
Sounds legit. This goes double on the cloud. In my early days of using AWS, it was so easy to create and destroy EC2s, I created some EC2s for testing - then terminated them... then realized I terminated the wrong ones.<p>Whoops