TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Our Server’s Hard Drive is Dead. We didn’t have a backup.

55 点作者 rhdoenges将近 12 年前

16 条评论

JPKab将近 12 年前
It's noble of you to come clean and own your mistake, but let me say this over and over:<p>You should never, ever provide an environment that stores people's hard work without having professionals who know how to safeguard it.<p>If it makes you feel any better, I recently had to clean up a mess in a huge enterprise IT shop, (if I were to name the organization you would immediately know them) involving hundreds of thousands of man-hours of work lost due to a lazy, incompetent DBA and the clueless management above her.<p>This "DBA" was the kind of person who came in at 9:45AM, took a 2 hour lunch at noon, and left at 3:30. Did I mention she refused a work from home option?<p>She didn't know how to do chron jobs, so all of her backup scripts had to be run manually. If she was on vacation, they didn't get run. Surprise Surprise, the DB died after her long pre-Christmas vacation. Zero backups for the first 3 weeks of December.<p>Even "professionals" can be suspect sometimes.
评论 #5760938 未加载
评论 #5760889 未加载
评论 #5760829 未加载
gregd将近 12 年前
<i>"We need to get programming talent on-board."</i> Sounds to me like they still haven't learned their lesson...
评论 #5760849 未加载
评论 #5760765 未加载
评论 #5760728 未加载
ashray将近 12 年前
I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts.<p>EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary hard disk had failed over a year ago - but he didn't know about that (the host informed him of this... now?). That points to a RAID setup where the mirror was basically working all this while. That's what I'm talking about in this post.
评论 #5760855 未加载
评论 #5760790 未加载
评论 #5760766 未加载
评论 #5760865 未加载
评论 #5760861 未加载
femto将近 12 年前
Chances are that most of the information is still physically there, just that it is inaccessible. First thing I would do is physically obtain the drive, so even if it's inaccessible you have the information in your possession.<p>From your blog post, I'd assume you don't have the knowledge to attempt recovery yourself, so call in an expert to handle the data recovery for you. At this stage, it is a matter of what the information is worth to you, compared to the cost of recovery. Almost any intervention is possible, for a price.
pdeuchler将近 12 年前
I feel for you. I really do. You did a lot of things right: learned how to program, bootstrapped your startup, released a product (!), got users, went viral, etc. etc.<p>But.<p>1) All of this could have been solved with money, specifically money used to pay professionals. You got 30 <i>THOUSAND</i> signups and you didn't think of trying to get funding? I'm surprised VC's weren't pounding at your door. At the very least, that might even be enough for a bank loan from a savvy lender. Hell, you could probably find a recently graduated ('tis the season) CSCI student willing to just take sweat equity with those numbers. This is especially frustrating for me as I currently have a startup that recently garnered a whopping 400 (count 'em!) <i>hits</i> on it's signup page, and yet I still got emails from people trying to invest. Not nearly platinum tier, and thus far none have panned out, but still!!!<p>2) You claim to have worked in web design/development for a while, and you didn't hear about 1&#38;1's horrific reputation? That's hard for me to believe. In fact, of any community, the PHP/JS crowd is probably most familiar with being burned by 1&#38;1. (Not even going into the slimy overselling).<p>I hate to say it, but you should have known better. That said, I sincerely wish you the best of luck. You've succeeded pretty spectacularly thus far, and in the big scheme of things this is a pretty minor setback. Just keep shipping and you'll get it eventually.<p>Edit: I realize that it might seem foolish to some to go after funding when it's not needed, but I would argue that if you are making it up as you go along (not an indictment, it's how we learn) and you get these kind of numbers, you should feel at least a little obligated to your users to secure your product. If that requires money that you don't have, get funding.
sc00ter将近 12 年前
Not to be depended on as a substitute for backup, but 'dead' doesn't necessarily mean <i>dead</i>. Forensic recovery (either DIY or professional, depending on the nature of the failure) may still be an option.<p>Logic board failures are common, and replacements cheap (the cost of a new HD of the same model), data can be highly recoverable from soft failures. Mechanical failure is the worst case, but as long as the platter(s) is/are in tact, not insurmountable.
tempestn将近 12 年前
"Technical support informed me that my first HD died 20 days into my contract. The backup HD hummed along for a year."<p>That sounds more like RAID than a backup HD.
评论 #5760812 未加载
jefe78将近 12 年前
"We messed up bad. We launched without having a backup procedure in place, and without the resources to make it happen. This was a hard-learned lesson that won’t happen again. We have no one except ourselves to blame."<p>You know how you messed up? By not using something like AWS - EC2 - Snapshots. Or even S3 or Glacier. What is this trend of devs doing Operations? As a Sysadmin with a Compsci/dev background, it blows my mind constantly.<p>Great, you know how to move around the CLI, but are you versed in how to maintain a proper and robust system?<p>Also, why weren't you using something like SES for your email alerts?
评论 #5761120 未加载
thaumaturgy将近 12 年前
a. My business has a partnership with a good data recovery outfit. We might be able to get you a good deal on a data recovery if you want to try going that route.<p>b. It takes a particular kind of personality to be good at sysadmin work. (And a lot of trial-and-error -- I just recently had to do an emergency server build due to a Debian update whoops, and I've been doing this stuff for a while.)<p>c. I usually recommend BackupPC (<a href="http://backuppc.sourceforge.net/" rel="nofollow">http://backuppc.sourceforge.net/</a>) for easy set-it-and-forget-it backup infrastructure. It's compatible with everything, it will notify you if there are problems, it does pooling and de-duplication and compression, it's fast and reliable, and you can usually store months of backups on a small offsite server. I store 12 months of all hosted and customer data with it, and we've used it to meet other clients' needs too.<p>d. If you need affordable help, let me know. I'm way too cheap, and I do this stuff all day, every day. I opened a business specifically to address problems like this: needs something, money is a problem.<p>That goes for anybody else too. If your lack of backups is keeping you awake at night, or if you've suddenly outgrown your infrastructure, or if looking at config files gives you an ulcer, get in touch with me. I'll help you out.
Sealy将近 12 年前
How many people did it affect? I would be too ashamed to admit it if i was a company offering services for programmers but didn't back up my server.
nwilkens将近 12 年前
I see this too many times.. and have read about this more than once on HN in recent memory.<p>Hire a proper system administration company early to work with you on these types of things. There are many companies out there that do this. I happen to run a company that does this, so I know that you can add an expert admin to your team for $100-200/mo.
评论 #5760835 未加载
评论 #5760877 未加载
foobarbazqux将近 12 年前
This site is an excellent way to find out if you've covered all your bases in your backup protocol:<p><a href="http://www.taobackup.com/" rel="nofollow">http://www.taobackup.com/</a>
评论 #5761004 未加载
hoodoof将近 12 年前
<a href="http://www.codinghorror.com/blog/2009/12/international-backup-awareness-day.html" rel="nofollow">http://www.codinghorror.com/blog/2009/12/international-backu...</a>
ironchef将近 12 年前
On the plus side, this will probably only ever happen to you once. Once you've felt the pain, you'll never let it happen again.
评论 #5760740 未加载
cdvonstinkpot将近 12 年前
I wonder if it still would've happened if they were SSDs...<p>I tend to think they're safer due to no moving parts.
评论 #5760927 未加载
macarthy12将近 12 年前
Spinrite !