Facebook’s backups were written in PHP. Well, the second version was, anyway. (The first version didn’t work so well.) By backups, I mean the central MySQL databases with profile and post information; not media or messages. The “Crown Jewels,” so to speak.<p>It was written by an engineer in a week or so and then handed off to me. I was a storage guy with some programming chops. Fewer chops than I thought.<p>Multiprocess, forking, long-lived, server-side, I/O intensive PHP. And it worked brilliantly. Over the following years, it scaled from a backup set of a few hundred TB to many, many petabytes.<p>I extended the code quite a bit, fixed a bunch of bugs (it was version 0 when it launched, no hate to the guy who wrote it), and added statistics for reports and coverage.<p>The main reliability problem ended up being when we switched to HPHP, the predecessor to HipHop. Since the primary and largest customer of this code was the web front-end, changes there would often break our backend code. One I remember was changing fork() to only start a new green thread; saved $millions on the front end, completely broke backups. We ended up getting a special posix_fork() function or something from the HPHP team that restored the old behavior.<p>Eventually I rewrote it in Python 2.7. It took me two complete attempts and then a further six months of tweaking to get it anywhere near as stable as the PHP version, even with brilliant guys like Haiping Zhao constantly re-weaving the carpet under our feet. I never did like it as much as the PHP version.