A Storage Crisis

I tried to back up a 60% full Ubuntu Linux RAID to a 4TB USB3 hard drive. I set the backup going and left it a few hours.<p>I was shocked to come back and find the RAID 100% full and only a few GB copied. It took me a few minutes to figure out what happened.<p>Maddeningly, the USB drive has a feature that it goes to sleep after 15 minutes or so, <i>even if</i> Ubuntu is actively using it to write files! Maybe the drivers on Windows do something that keep it awake, but on Linux it just goes to sleep in the middle of being used.<p>Now here&#x27;s the thing, the drive DOES realize it has more to do and wake back up. But this sleep-wake causes a USB disconnect and reconnect. Which causes Ubuntu to unmount and remount the drive.<p>Now here&#x27;s the problem, perhaps because the backup program is still making the the original mount point &quot;busy&quot;, Ubuntu doesn&#x27;t re-mount the media to the path. Instead it, gets mounted at &quot;<i>&#x2F;media&#x2F;path</i>-1&quot;.<p>Since Linux uses regular folders as mount points, the old mount point at &quot;&#x2F;media&#x2F;path&quot; becomes a valid folder on the <i>local</i> disk. So the backup program keeps going, but now it&#x27;s filling up the local disk.<p>I haven&#x27;t found a solution for this problem that will allow me to complete a backup (or even complete a long-duration manual copy).

Maybe the data retention policy of OP is the real problem. Sound like a data hoarder&#x2F;virtual messy to me.<p>When ever I take pictures on a trip or vacation, i go through them at the end of each day, delete most of them and keep maybe 2 or 3 max, beautify them and the rest goes into the bin. No matter how long the trip, i try to only keep at max the 15 best Pictures. All filler no killer. That way I am comfortable to show others pictures of a trip without boring them and I also like to look at them from time to time as I know those are the best moments.<p>At the end of each year we create a calendar with photo collages of 4 to 6 pictures per month. The calendar goes to relatives and we create a photo book from the print outs. That is what gets archived.

There&#x27;s a huge market here for something that professional photographers could use. I know several and they all tell each other to keep buying more external 1tb hard drives then back everything up to random cloud image hosts. Most of them end up with a bunch of zip lock bags laying around full of disks and thumb drives.<p>I made the mistake of building a NAS for one at one point but it turns out that you need to be really diligent about documenting the setup as rebuilding a raid5 array when it inevitably fails can be very difficult if you don&#x27;t remember what the actual configuration was.<p>The crux of the issue is that they need massive storage capacity but whatever the solution is needs to be able to manage merging disjoint Adobe Lightroom catalogs while de-duplicating and without the possibility of data loss, all while needing basically no maintenance because these people tend not to be incredibly computer literate.

If you want more reliable backups, Backblaze personal backup should never be on your list. The client takes anywhere between 1-8 hours to upload the index of files to the server <i>after</i> uploading the actual data for the backup, but it will say that the backup is complete before this happens and give a false assurance. So if your machine dies in that span (data uploaded but index not uploaded), then it’s as good as the backup not happening because the server has data blobs but no metadata, and cannot provide you a way to restore that. This is an intentionally designed behavior (for some other reasons) that adds risks that I belief nobody should take.<p>See this blog post and comment from nearly four years ago for a better description. [1]<p>And here’s Backblaze’s support page on the same, last updated a month ago (June 2021) indicating that this behavior remains. [2]<p>Any backup solution that doesn’t, at a minimum, follow the 3-2-1 rule will cause more instances of regret in the future. If the data is of value, it needs better and constant care.<p>[1]: <a href="https:&#x2F;&#x2F;mjtsai.com&#x2F;blog&#x2F;2014&#x2F;05&#x2F;22&#x2F;what-backblaze-doesnt-back-up&#x2F;#what-backblaze-doesnt-back-up-update-2017-08-24" rel="nofollow">https:&#x2F;&#x2F;mjtsai.com&#x2F;blog&#x2F;2014&#x2F;05&#x2F;22&#x2F;what-backblaze-doesnt-bac...</a><p>[2]: <a href="https:&#x2F;&#x2F;help.backblaze.com&#x2F;hc&#x2F;en-us&#x2F;articles&#x2F;217665498-Why-hasn-t-Backblaze-backed-up-my-new-files-yet-" rel="nofollow">https:&#x2F;&#x2F;help.backblaze.com&#x2F;hc&#x2F;en-us&#x2F;articles&#x2F;217665498-Why-h...</a>

Assuming you have family living elsewhere but reachable through a fast internet connection you can do what I do by making a deal with them: they hang your backup box off their net and you will do the same for them. The backup box is some piece of computing equipment with storage media attached, e.g. a single board computer hooked up to a JBOD tower. Depending on the level of trust between you and your family you can use the thing as an rsnapshot target - giving you fine-grained direct access to time-based snapshots (I use 4-hour intervals for my rsnapshot targets which are located on-premises in different buildings spread over the farm) or as a repository of encrypted tarballs, or something in between. Allow the drives to spin down to save power, they&#x27;ll be active only a fraction of the day. The average power consumption of the whole contraption does not need to exceed 10-15W making electricity costs negligible. You can have as much storage capacity as you want&#x2F;can afford at the moment, keep for for as long as you want or until it breaks without having to pay any fees (other than hosting their contraption on your network - possibly including building it for them if they&#x27;re not that computer-savvy).

I recently did a comparison of cloud storage options:<p>- If you want &quot;personal storage&quot; (no programmatic access) the options are generally around $5 per TB&#x2F;month - but that&#x27;s a lot of drag-and-dropping and praying the connection doesn&#x27;t die during transfer.<p>- If you want object storage (think S3) the cheapest is $5 per TB&#x2F;month, the average is $10 per TB&#x2F;month, and the &quot;high end&quot; is $20 per TB&#x2F;month, with extra costs for bandwidth ranging from $10 to $120 per TB&#x2F;month.<p>Honestly, just store less crap. Marie Condo your digital life. Does it not spark joy? Have you not looked at it in the last 6 months? Does it not serve a useful purpose, such as tax records? Ditch it.<p>Even if you want to keep some pictures&#x2F;video, either print a copy in original-quality, or compress&#x2F;downscale it. I took a 3 minute video on my phone and it&#x27;s 425MB. And it was still grainy! I used to download two-hour movies that were 725MB and looked like a DVD! errrrrrr... I mean, I heard about a guy that did that.

A few people have raised this question in the thread already, but why are we compelled to store such unreasonable numbers of images?<p>I&#x27;m definitely not immune and I find that the satisfaction I get from my photo collection is inversely proportional to the size of my collection. At this point, I&#x27;m just lugging around this huge mass of data &quot;just in case&quot;. There&#x27;s no way I will ever have time to sort my images. There are probably 2000 wedding images alone, let alone the tens of thousands thousands of random snapshots that may or may not be something I ever care to see again.<p>At this point, I would almost prefer a smart solution opt-in solution similar to what Google photos provides for smart albums: &quot;We found these images that would be good long-term. Save them indefinitely?&quot;

I back up about 15tb of photos with Arq to Wasabi cloud storage.<p>It has been running with zero maintenance (other than occasional partial restores) since late 2018.<p>I transitioned off of AWS cloud storage when they raised the prices.<p>I’m not sure if Wasabi is still the cheapest and fastest, but they have been great to deal with. And Arq is an excellent set-it-and-forget-it encrypted cloud backup app.<p>I also run a server with a 16tb RAID 1 array and a set of local backup drives. Sadly it is almost full, and the volume of data makes it a hassle to upgrade (not to mention the cost).<p>I’ve found standard 1Gig Ethernet to be just barely fast enough for editing photos over the local network. However, for my own sanity, I usually do the initial editing on a local drive before sending the files to the server (and from there to the cloud backup).

I think $6 a month for &quot;unlimited data&quot; is going to end badly.<p>The AWS or Azure price is high, but it&#x27;s a scalable price, a real price.

10mbps up will upload about 30GB in 8 hours overnight while you sleep. Most people are not taking 30GB of new photos every day. Or using their Internet connection all day when they are awake either. So cloud storage is totally viable for most people with 10mbps upload speeds.<p>Now, a person might have a big backlog of data. This guy has 7TB of photos. If you start uploading, you’ll get caught up eventually! If you want to get caught up faster, you just need to find a faster pipe and park your computer&#x2F;HDs on it for a little while. For me, it was a friend with FiOS. For many professionals, it might be their office.<p>The point is, once you’re through the backlog, 10mbps is probably enough to maintain. Again: for most people.

Amazon Snowball? <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;snowball&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;snowball&#x2F;</a>

Lots of people talking about photo retention policies here, but for me the issue is videos. We have little kids, and that means lots of cute videos. It&#x27;s harder to scan through them and see what&#x27;s useful versus what isn&#x27;t. I try to remember to delete videos that didn&#x27;t turn out when I take them, but that doesn&#x27;t help with the videos that I shoot that are 2 mins long but only have 20 seconds of something worth keeping.<p>So far I&#x27;ve been able to get by with retail HDDs; I have a 5TB drive at home and another that we keep off-site and refresh a couple times a year. This seems to be a sustainable strategy for me, as affordable (~$100) portable HDDs are growing at a rate that is faster than my storage needs. I don&#x27;t know if it would be cheaper to pay for iCloud, but for whatever reason I feel safer having two HDDs with my media than trusting Apple&#x27;s (or anyone else&#x27;s) cloud.

I ended up buying a 4 disk synology NAS unit&#x2F;appliance for about $400 a couple of years ago. It&#x27;s been remarkably reliable, and has a built-in feature to backup to Glacier, although there&#x27;s third party tools for most any online data backup system you can think of.<p>Prior to that I had this monsterous, 4U server that I had to maintain, update, debug, not to mention build and move from house to house, for 8 years. At the time it seemed like a good move, but over time, as my time got shorter to work on personal products, I started to look for other ways to solve the storage problem.<p>I much prefer the Synology NAS to the 4U custom built system. It&#x27;s about as close to a toaster-style appliance as I can think of. If it died tomorrow, I&#x27;d buy another one (then restore from my Glacier backups).

This storage talk abut hybrid drives sounds a lot like 2010. This isnt new and SSDs are really not much more expensive than your regular HDD and they offer much more benefits. Its just that Apple charges you for 1 drive the price of at least 4 same drives if you were buying it on your own.

If you don&#x27;t want to pay for the cloud, then there is only one sane option: You keep upgrading your local storage so that you never have more than a few devices. Every six months, buy a new giant disk and decommission as many smaller disks as you can.

The real question being, of course, given that your life is finite ... how often will you actually look at any of these pics.<p>The more pix there are, the less likely it is you&#x27;ll ever even open any of these.

I think there is another point worth mentioning. The Data we own or create are increasing. Taking more photos than ever, higher resolution, Video in 4K or 60 frames. Friends sending their media over. So not only does the usage increases, the actual size of those unit increases as well.<p>And yet Hardrive price &#x2F; GB hasn&#x27;t fallen much at all. From 2013 to 2021 now, when it was $0.035 to $0.04 &#x2F;GB. Mostly because much HDD patter density has stalled.

I previously had a 4-bay Synology and loved the UI but couldn&#x27;t deal with the price jump to an 8-bay. I discovered xpenology.org which allows you to bootload Synology software on x86 hardware.<p>I turned an old desktop PC into an 8-bay storage machine with a PCIe SATA card and now have all the flexibility I want, without the price penalty for Synology hardware.<p>The best place to find hard drives is <a href="https:&#x2F;&#x2F;diskprices.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;diskprices.com&#x2F;</a> which scrapes Amazon and a few other sites to show TB per £&#x2F;$. It&#x27;s almost always cheaper to buy a retail packaged external disk and extract the hard drive.<p>Note: Some hard drives once shucked from external cases won&#x27;t turn on when attached to regular SATA power adapters due to mismatches in power connector specification. Easily solved by using a SATA power to Molex adapter, and connecting that to a Molex to SATA power adapter (SATA-&gt;Molex-&gt;SATA) as Molex does not support the 3.3V pin which is the issue.

Spend a few hundred to a few thousand buck and build a chunky NAS. Get a bunch of 12TB SSDs and put them in a RAIDZn.<p>Also, what shitty phone takes 108MP photos? That&#x27;s guaranteed to be some stupid Android phone gimmick. There&#x27;s no way having that many pixels with a teeny optical path and a teeny sensor is useful. I&#x27;d only want above 100MP on a medium format sensor.

What do folks typically use for storing media you don&#x27;t access very often, but when you do access it you want it relatively quickly and at high transfer speeds?<p>I like to backup my UHD blurays losslessly and stream them from my PC via plex (~100mbps). I&#x27;m thinking local storage via a NAS is probably the cheapest option here, right?

i am using this jpeg recompressor:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;danielgtaylor&#x2F;jpeg-archive" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;danielgtaylor&#x2F;jpeg-archive</a><p>Tried a lot of combinations of quality, and settled for -q medium --max 75% --accurate<p>Anything higher makes no sense&#x2F;difference, even if you print it at A3. And my images are 8-18-24Mpix.<p>i&#x27;m not a pro-foto but somewhat of an old pre-print school - Your level of pickiness may vary :&#x2F;<p>Effectively it is like 4-8 times down - Total went from 150Gb into 30Gb - that&#x27;s about 40,000 images.<p>Which still doesn&#x27;t solve the OP&#x27;s or anyone&#x27;s problem of too-easy-producible digital &quot;assets&quot;.. but is more manageable.

What are the requirements? Large capacity, redundancy, reasonable access speed, always on but maintenance downtime of even a few hours a year tolerable?<p>Easy but your data isn&#x27;t yours: sync your data to GDrive or Apple or whatever, and sync a NAS to that.<p>A little harder but still doable: get a Hetzner and set that up as your storage, set up your own access, sync to local NAS. A Hetz is also really useful for running a load of other services, so for 50 bucks or so it seems pretty reasonable.<p>You could just buy a huge disk and run a server in your house, but it gets annoying in various ways. Kids unplug the power, it creates heat, maybe noise, multiple disks end up needing management, that kind of thing.

How much of the actual information in the high resolution photos is just noise?<p>So we buy sensors with more megapixels that just save more noise. Then we need to buy more storage capacity, more bandwidth, more processing, more battery etc.<p>Since the cameras have been hitting closer and closer to physical limits but something still needs to be sold...<p>It&#x27;s like people buying clothes from the mall and taking them straight to a storage building. They don&#x27;t have enough space at home. And they won&#x27;t have time to wear so many clothes anyway.

&gt; The best new phones come with the ability to shoot 108 megapixel photos, record 4K video with stereo sound, and pack the results into a terabyte of onboard storage. But what do you do when that storage fills up?<p>A good starting point for me is to not keep everything I shoot. I have often noticed that every 50-100 photos I shoot I only want to keep 5-10. Rest are bad&#x2F;weird shots, pics I’m not interested in etc.<p>I never faced this when I used film.

Almost everything I create doesn’t really need to be saved and I could never find it if I needed to. I just let Google Photos keep them. It’s good enough.

I use borgbase (<a href="https:&#x2F;&#x2F;www.borgbase.com" rel="nofollow">https:&#x2F;&#x2F;www.borgbase.com</a>) for my online backups. It took a few days to make the initial upload, but now it&#x27;s it only takes a couple of minute every hour. And you can select the pruning strategy you want.<p>I haven&#x27;t had to use it yet, but I check the files from time to time, and so far I&#x27;m satisfied with the service.

I used Backblaze B2 as an offsite backup for RAW digital film. 1TB took about a month to completely backup, which is not ideal.<p>B2 worked great and the pricing is indeed excellent, but the volume of raw data to backup over broadband is just not realistic.<p>I think it would be better to physically mail a drive for archiving raw footage, or use tap drives.

I am faced with this issue of dealing with hundreds of GB of family video and pictures everytime I upgrade my wife’s phone.<p>The newer phones always have a higher resolution camera so file sizes keep increasing.<p>I would love a simple easy to use tape drive that just works. Something I could backup files to for 20-25 years.

I can&#x27;t fathom how you get to 7TB of photos without being a photographer, or recording video all the time. Granted I haven&#x27;t been too prolific in the last few years, but my entire collection from 3 DSLRs + iPhones going back to ~2005 totals 300GB. All backed up in B2 + google cloud.

The short answer to &quot;can I replace physical storage with cloud storage?&quot; is no.

The solution is either NAS or cloud archive (not backup) such as SmugMug or ExpanDrive.

I’m still happy with my setup of Drobo + 2 rotating time machine backup HDs. I’ve probably been using it close to 10 years now.

I recommend looking into decentralized cloud storage solutions, which are really the next generation of AWS S3-like services.<p>Check out storj.io

I remember when my life&#x27;s professional work fit on a CD. It was a strange feeling.

Isn’t this the business model that Dropbox, iCloud and Google Drive, etc. are based upon?

I want to replace those external drives with cloud storage. Is that possible?

Never heard about cloud storage?

Hoarding is a mental illness