Amazon Glacier

I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage.<p>With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,vud,vmdk,vmsn,vmsd,hdd,vdi,vmwarevm,nvram,vmx,vmem,iso,dmg,sparseimage,sys,cab,exe,msi,dll,dl_,wim,ost,o,log,m4v" files. They also don't want to backup your /applications, /library, /etc, and so on locations. They also make it clear that backing up a NAS is not the target case for their service.<p>I can live with that - because, honestly, it's $4/month, and my goal is to keep my working files backed up. System Image backups, I've been using Super Duper to a $50 external hard drive.<p>Glacier + a product like <a href="http://www.haystacksoftware.com/arq/" rel="nofollow">http://www.haystacksoftware.com/arq/</a> means I get the benefit of both worlds - Amazon will be fine with me dropping my entire 256 Gigabyte Drive onto Glacier (total cost - $2.56/month) and I get the benefit of off site backup.<p>The world is about to get a whole lot simpler (and inexpensive) for backups.

This is a really good offering for media that you typically will keep locally for instant access, yet you want to have an off-site backup in a way that lives for a very long time.<p>Dropbox should work here, but it's simply too expensive. My photo library is 175GB. That isn't excessive when considering I store the digital negatives and this represents over a decade.<p>I don't mind not being able to access it for a few hours, I'm thinking disaster recovery of highly sentimental digital memories here.<p>If my flat burns down destroying my local copy, and my personal off-site backup (HDD at an old friends' house) is also destroyed... then nothing would be lost if Amazon have a copy.<p>In fact I very much doubt anyone I know that isn't a tech actually keeps all of their data backed-up even in that manner.<p>I find myself already wondering: My 12TB NAS, how much is used (4TB)... could I backup all of it remotely? It's crazy that this approaches being feasible. It's under 30GBP per month for Ireland storage for my all of the data on my NAS.<p>To be able to say, "All of my photos are safe for years and it's easy to append to the backup.". That would be something.<p>A service offering a simple consumer interface for this could really do well.

I think one of the most interesting parts of this is how they plan to ensure that people do not use it for transient backup: <a href="https://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_deleting_data_that_is_less_than_3_months_old" rel="nofollow">https://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_de...</a><p><i>Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months.</i>

Storage experts: I'd love to know more about what might be backing this service.<p>What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?

<i>Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month.</i><p>What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics.<p>Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.

What I find fascinating with Amazon's infrastructure push is the the successful 'homonymization' of their brand name Amazon.<p>Amazon simultaneously stands for ecommerce and web infrastructure depending on the context. e.g "Hey I want to host my server".. "Why don't you try Amazon". "Do you know where I can get a fair priced laptop?" "Check Amazon".<p>Is there any other brand that has done this successfully?<p>Edit: I should have specified internet brand.

I know a lot of people seem to have jumped on the <i>backup</i> options of Glacier here and, whilst there is some potential for home users to make use of this product for back, that is not what Glacier is intended for.<p>Glacier is an <i>archive</i> product. It's for data you don't really see yourself ever needing to access in the general course of business ever again.<p>If you're a company and you have lots of invoice/purchase transactional information that's 2+ years old that you never use for anything, but you still have to keep it for 5 - 10 years for compliance reasons, Glacier is the perfect product for you.<p>Even its pricing is designed to take into account that the average use case is to only access small portions of the total archive store at a reasonable price (5% prorated for free in the pricing page).

This is fantastic. I've long searched for a solution like that. This is really suitable for a remote backup that only needs to be accessed if something really bad happens (i.e. a fire breaking out, etc). I'm a lone entrepreneur, so I do have backup hard disks here, but being able to additionally save this data in the cloud is great.<p>I'm often creating pretty big media assets, so Dropbox doesn't necessarily offer enough space or is - for me - too expensive in the 500gb version (i.e. $50 a month).<p>Glacier would be $10 a month for 1 terabyte. Fantastic.

I'm not sure if cost is right. Each project I work on is approx. 50-60 TB in size (video). Recent one got backed up on 20 LTO 5 tapes times three. That's $600 for tapes per project. Each tape set went to a separate location - two secure ones for about $20/year and one at studio archive for immediate access, if needed. I find this method extremely reliable and it cost ~$700 initially to back all up and virtually non existent further fees. With Glacier it would cost $600 per month.

Interestingly they penalise you for short-term storage:<p><i>Amazon Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months</i>

I had a quick skim through the marketing stuff and the FAQs and didn't see anywhere that actually details what the backend of this is. I'd be curious if they're actually using tape, older machines, Backblaze pods, etc. I guess if it's the latter, the time to recover could be an artificial barrier to prevent people from getting cute.

Beware that retrieval fee!<p>The retrieval fee for 3TB could be <i>as high as</i> $22,082 based on my reading of their FAQ [1].<p>It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour?<p>This last case seems most plausible to me [6] -- that the retrieval rate is based solely on the rate of your requests.<p>In that case, the math would work as follows:<p>After uploading 3TB (3 * 2^40 bytes) as a single archive, your retrieval allowance would be 153.6 GB/mo (3TB * 5%), or 5.12 GB/day (3TB * 5% / 30). Assuming this one retrieval was the only retrieval of the day, and as it's a single archive you can't break it into smaller pieces, your billable peak hourly retrieval would be 3072 GB - 5.12 GB = 3066.88 GB.<p>Thus your retrieval fee would be 3066.88 * 720 * .01 = $22081.535 (719x your monthly storage fee).<p>That would be a wake-up call for someone just doing some testing.<p>--<p>[1] <a href="http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_when_retrieving_large_amounts_of_data_from_Amazon_Glacier" rel="nofollow">http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...</a><p>[2] After paying that fee, you might be reminded of S4: <a href="http://www.supersimplestorageservice.com/" rel="nofollow">http://www.supersimplestorageservice.com/</a><p>[3] How do you think this interacts with AWS Export? It seems that AWS Export would maximize your financial pain by making retrieval requests at an extraordinarily fast rate.<p>[(edit) 4] Once you make a retrieval request the data is only available for 24 hours. So even in the best case, that they charge you based on how long it takes you to download it (and you're careful to throttle accurately), the charge would be $920 ($0.2995/GB) -- that's the lower bound here. Which is better, of course, but I wouldn't rely on it until they clarify how they calculate. My calculations above represent an upper bound ("as high as"). Also note that they charge separately for bandwidth out of AWS ($368.52 in this case).<p>[(edit) 5] Answering an objection below, I looked at the docs and it doesn't appear that you can make a ranged retrieval request. It appears you have to grab an entire archive at once. You can make a ranged GET request, but that only helps if they charge based on the download rate and not based on the request rate.<p>[(edit) 6] I think charging this way is more plausible because they incur their cost during the retrieval regardless of whether or how fast you download the result during the 24 hour period it's available to you (retrieval is the dominant expense, not internal network bandwidth). As for the other alternative, charging based on how long it takes them to retrieve it would seem odd as you have no control over that.

This is the best thing ever! I've been dreaming about such a service for a very long time -- posted about how I needed this here 6 months ago:<p><a href="http://news.ycombinator.com/item?id=3560952" rel="nofollow">http://news.ycombinator.com/item?id=3560952</a><p>Rotating hard drives on the NAS in my attic is going to get a LOT simpler...

that's certainly interesting. as there will be migration from s3 to glacier, it would be nice if tarsnap had an option to store only the (say) last week in s3 (with .3$/gb/month) and the rest in glacier (with, say, .03$/gb/month).<p>that would certainly be very nice. cperciva, what do you think?

Here's to hoping that <i>duplicity</i> and <i>git-annex</i> could somehow make use of this service. I'm far more optimistic about <i>duplicity</i> support though, as incremental archives seem to fit the glacier storage model much better. A <i>git-annex</i> special remote [1] might turn out to be much more challenging, if at all possible.<p>[1] <a href="http://git-annex.branchable.com/special_remotes/" rel="nofollow">http://git-annex.branchable.com/special_remotes/</a>

From the retrieval times they are giving, it seems plausible that they could be only booting the servers 5 or 6 times a day, to run the upload and retrieval jobs stored in a buffer system of sorts. Having the servers turned off for the majority of the time would same an immense amount of power, although I wonder about the wear on drives spinning up and down compared to being always on.<p>Any other theories on how this works on the backend while still being profitable?

This sounds really appealing as a NAS backup solution, but I'm a bit concerned about security and privacy. Let's say I want to backup and upload my CDs and movies, would Amazon be monitoring what I upload and assume I'm doing something illegal?

Wow, order of magnitude cheaper than S3 (1 cent per month vs. 12-14 cents for S3). Data transfer priced the same.<p>Drawback is "jobs typically complete in 3.5 to 4.5 hours."<p>Seeing as how people tend to be pack rats, I can see this being huge.

One recurrent issue with Amazon services is that they charge in US$ and currently do not accept euros. European banks charge an arm and a leg for micropayment conversions: last time I got a bill from AWS for 0.02$ it ended up costing me 20 euros or so. Pretty much kills the deal. One solution could be to pre-pay 100$ on an account and let them debit as needed.

What does "99.999999999% durability" mean? Does it mean they allow themselves to lose one byte per terabyte on average?

Interesting to see that in some cases it probably makes sense to just stop paying the bills, rather than pay the early deletion fee[1].<p>[1]<a href="https://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_deleting_data_that_is_less_than_3_months_old" rel="nofollow">https://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_de...</a>

I'm currently using an app called Arq that backs everything up to S3. If I had to guess, I'd say there's about 50-60 gigs or more on there. Last months bill was something like .60 cents. How does glacier compare or contrast to this setup (the app does something similar with the archive concept)?

Crashplan is still cheaper for storage larger than 400GB.<p>Crashplan+ Unlimited is USD 2.92/month if you take the 4 year package. When I upload 300GB to Amazon and pay 0.01 * 300 = USD 3/month. Amazon would be even more expensive for larger amounts of data.<p>Is there some fine print I'm missing with Crashplan unlimited?

I just started a project where I'm keeping a raspberry pi in my backpack and am archiving a constant stream of jpgs to the cloud. I've been looking at all the available cloud archival options over the last few days and have been horrified at the pricing models. This is a blessing!

I am a bit confused with the pricing of retrieval.<p>Could some good soul tell me how much would cost to:<p>Store 150 GB as one big file for 5 years. To this I will add 10 GB (also as one file) every year. And lets say I will need to retrieve the whole archive (original file + additions) at the end of year 2 and 5.<p>How much will it cost?

They should provide a "time capsule" option - pay X dollars, and after a set number of years, your data archive will be opened to the public for a given amount of time.<p>There'd be no better way to ensure that information would eventually be made public.

Great. Now if someone would simply create a badass client for this for Mac (like Backblaze or Mozy) … we'd be in business. :)

What's particularly awesome, is that this likely represents an upper bound on cost. It will only go down as time goes on.

Try this link<p><a href="http://aws.amazon.com/glacier/?utm_source=AWS&#38;utm_medium=website&#38;utm_campaign=BA_glacier_launch" rel="nofollow">http://aws.amazon.com/glacier/?utm_source=AWS&#38;utm_medium...</a>

Looks like the perfect solution to backup all my photos. Considering 100 GB of photos, that's still just 100 * 0.01 * 12 = $12 a year! I'm sold on this!

This looks like exactly what I need. I'm currently using S3 for backup and archiving by regularly running s3cmd to sync my data on my NAS.<p>And while not super-duper expensive, s3 provides much more than I really need, and hence a more limited (but cheaper) service would definitely be appreciated.<p>If there is anything with the easy of use like s3cmd to accompany this service I will be switching in a heartbeat.

I'm with Atlantic.net cloud [AWS competitor, full disclosure]; the price point for storage is great but retrieval seems expensive -- perhaps retrieval is very rare its offset by the savings on the storage. I know you can mail in drives for storage, can you have them mail you drives for retrieval? (for Glacier specifically)<p>Also, prior comments made mention they were using some sort of robotic tape devices, but according to this blog:<p><a href="http://www.zdnet.com/amazon-launches-glacier-cloud-storage-hopes-enterprise-will-go-cold-on-tape-use-7000002926/" rel="nofollow">http://www.zdnet.com/amazon-launches-glacier-cloud-storage-h...</a><p>Its using "commodity hardware components". So, thats why I thought maybe they are loss-leadering on the storage and making up on the retrieval prices?<p>Its definately a interesting product and I love how there's a reason they called it Glacier. AMZN is a wild boar going after everyone!

Since this is built on top of S3, I'd love to be able to store EBS Snapshots in this.

We've just added support for Amazon Glacier to <a href="http://www.PlanForCloud.com" rel="nofollow">http://www.PlanForCloud.com</a> so you quickly forecast your costs and compare it with other options.

This is amazing! Just ~3 weeks ago I finally broke down and started using S3 to store my digital photos and other such crap.<p>Sure hope transferring 10 or 20 gigabytes of data from S3 to Glacier is easy.

Anyone know what the TOS are for this? I couldn't find them on a scan of the announcement.<p>A lot of the consumer-level services refuse any liability for any data loss. Does Amazon do the same for this?

Can anyone make sense of the retrieval fees? Seems like the most confusing thing ever. If I'm storing 4TB and one day I want to restore all 4TB, how much is it going to cost me?

I get a 404, and when I go to <a href="http://aws.amazon.com/" rel="nofollow">http://aws.amazon.com/</a> I see nothing about glacier in the news feed.

Any easy to use Windows clients for Glacier (for backing up an OS and/or other files)?<p>If not, to any developers reading this: there's money in them thar Glacier.

Amazon should complement this service with data contact centers which are connected to their data centers network. Then people could go to these centers in person and hand over their hard drives full of data for back up. It will be like bank lockers but only digital. At this low price people would want to upload terabytes of data which will be pain to upload/download.

I think this is more or less the formula for calculating monthly costs (corrections welcome):<p>0.01S+max(0,7.20*(R-0.0017S)/4)<p>S is number of GB stored<p>R is biggest retrieval in the month<p>4 is the average number of hours a retrieval<p>For an example with 10TB storage (replace 10000 to change): <a href="http://fooplot.com/plot/4pu7u2gpox" rel="nofollow">http://fooplot.com/plot/4pu7u2gpox</a><p>x is biggest retrieval in GB, y is $/month

Would be nice if I could just type my Amazon login details into TimeMachine and magic off-site backups just happened.

This looks awesome! We are currently developing a P2P-based backup solution (<a href="http://degoo.com" rel="nofollow">http://degoo.com</a>) where we are using S3 as fall-back. This will allow us to be much cheaper and I am sure it will enable many other backup providers to lower their prices to.

I hope they can get the access time down from 3-5 hours to about 1 hour - that's the difference for me between it being a viable alternative for storing backups of my client's web sites or not.<p>I might create a script that uploads everything to Glacier and just keeps a couple of the latest backups on S3 though.

Does anyone know if there is a CLI tool to interface with this yet? I see SDK's mentioned on the product homepage but I dont see any simple CLI tools for this yet to upload/download data and query etc.<p><a href="http://aws.amazon.com/developertools/" rel="nofollow">http://aws.amazon.com/developertools/</a>

I'm curious about data security. It says it is encrypted with AES. But is it encrypted locally and the encrypted files are transferred? I.e. does Amazon ever see the encryption keys?<p>Or is the only way to encrypt it yourself, and then transfer it?

It will be interesting to know if they will upgrade AWS Storage Gateway to use this kind of backend instead of S3 <a href="http://aws.amazon.com/storagegateway/" rel="nofollow">http://aws.amazon.com/storagegateway/</a>

It's a bit strange how S3 has a 'US Standard' region option, while Glacier has the usual set of regions (US East, US West, etc). I wonder if this means that unlike S3, Glacier isn't replicated across regions?

&#62; Amazon Glacier is designed to provide average annual durability of 99.999999999% for an archive.<p>That's pretty impressive. I wonder how many bytes they lost to lose that .0000000001% of data.

I just saw this and thought wow finally cheap mass storage. After reading the comments (and the Amazon Glacier web page) it's clear it's cheap archiving but not cheap retrieval.

Any desktop apps out there that will let you add folders for backup to Glacier and have them be automatically synced up to the cloud as they change? That would be quite useful.

Excellent! Is there any good open source client / backup application for this? I would start using it immediately. I'm currently using ridiculously expensive backup solution.

In case anyone is wondering it appears you can only upload from their APIs right now. I wonder if they intend to make it accessible through their web interface at any point?

I wonder if homeowner's insurance would cover the retrieval fee if your computer/hard drives were protected assets on the policy.

Wow, the cadence of releases from aws recently has been amazing. Can't wait to see what else they have in store.

Now if Tim Kay just adds support for glacier and I can leave all the backup scripts as is...

11 9's. Impressive.

Worst. name. ever.

Wake up.