Dribbble offline due to Postgres problem

This looks like disk page corruption.<p>In such a case, the <i>very first</i> thing you do — aside from verifying that your backups, replication, and WAL archiving are all working — before you even try diagnosing your problem any further than "something weird is happening" is <i>make a filesystem-level copy of your PostgreSQL cluster</i>. (If you're running LVM, ZFS, a SAN, or any other thing that lets you take an instantaneous filesystem snapshot, so much the better. Do that, and then copy <i>it</i>.) Then, and only then, should you even contemplate trying to un-fuck your database.<p>I cannot stress this point enough. That means make a tarball, or cp the directory, or rsync it, or whatever <i>disk file level</i> tool you prefer, and specifically and emphatically <i>not</i> pg_dump. (It's probably not going to make things any worse, but if you do have corrupted disk pages, pg_dump isn't likely work, anyway.)<p>Flailing around trying to fix things can sometimes make them worse. If you're working on your already broken data, and break it further without the safety net of a fs-level backup, you're ... well, you're worse off than you were five minutes prior, aren't you?

Since "dribbble.com" appears to be "taking a timeout for maintenance" I can't at a glance determine what, exactly, they do. So maybe they're a bunch of hobbyists. But if not ...<p>I don't understand how a "real" company, even in todays overheated environment of soon-to-fail hipster startups, doesn't do (at least) one of these two things: a) have competent employees on staff that are intimately familiar with their critical infrastructure and how to support it or b) pay, yes, gasp, pay some other company for professional <i>support</i> services. A glance at the Wikipedia entry for Postgres shows some possible companies that do that. Or, as already mentioned, there are mailing lists.<p>But twitter? Really? That's support? For someone other than a hobbyist running a website in their spare time?<p>Go ahead, flame me. But my first impression is "amateur hour". My apologies if dribbble.com really is an amateur effort.

Everyone should be running pg_basebackup + pg_receivexlog on a separate machine, preferably at a different data center.<p>These let you go back to any point in time. If you ran 'delete from orders where id=id', you can restore to the transaction before you ran that command.<p><a href="http://www.packtpub.com/how-to-postgresql-backup-and-restore/book" rel="nofollow">http://www.packtpub.com/how-to-postgresql-backup-and-restore...</a> contains more information.<p>Also, postgresql 9.3 (out in a few months) supports disk page checksums which can detect filesystem failures immediately.

The error in question is raised at <a href="http://doxygen.postgresql.org/sequence_8c_source.html#l01083" rel="nofollow">http://doxygen.postgresql.org/sequence_8c_source.html#l01083</a><p>My <i>guess</i> is hard drive failure. Hope they have good backups.

Did they try to post about their problem on postgresql mailing list (both pgsql bugs and general apply here)?<p>Postgres community is quite responsive and usually give very effective advice, as long as you use the proper channel to communicate.

I feel for you guys, but it sounds like a disk error. If you're not experts, you need to do one of the following A) rollback to a snapshot B) restore from backup. If you have neither of those, take a sql backup, and pray you can recreate the broken data.

Might as well comment here as tweets to the account seem to do nothing. Every 25th click or so has always resulted in a Chrome "no content" error. It's a frequent problem, and it's been happening for years.

It's back but I think one day of data is lost!