OpenZFS deduplication is good now and you shouldn't use it

I want &quot;offline&quot; dedupe, or &quot;lazy&quot; dedupe that doesn&#x27;t require the pool to be fully offline, but doesn&#x27;t happen immediately.<p>Because:<p>&gt; When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool.<p>To me, this is &quot;obviously&quot; the wrong approach in most cases. When I&#x27;m writing data, I want that write to complete as fast as possible, even at the cost of disk space. That&#x27;s why I don&#x27;t save files I&#x27;m actively working on in 7zip archives.<p>But later on, when the system is quiet, I would love for ZFS to go back and figure out which data is duplicated, and use the BRT or whatever to reclaim space. This could be part of a normal scrub operation.

&quot;And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads.&quot;<p>This struck me as a very odd claim. I&#x27;ve worked with Pure and Dell&#x2F;EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe&#x2F;compression savings. Only storing one copy of the base VM image works extremely well. Dedupe&#x2F;compression works really well on syslog servers where I&#x27;ve seen 6:1 savings.<p>The effectiveness of dedupe is strongly affected by the size of the blocks being hashed, with the smaller the better. As the blocks get smaller the odds of having a matching block grow rapidly. In my experience 4KB is my preferred block size.

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

I&#x27;m so excited about fast dedup. I&#x27;ve been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools compress within a crawl to create wacz or warc.gz files, but I don&#x27;t think anyone has tried to do compression across the entire database of all snapshots ever taken by a tool.<p>Big thank you to all the people that worked on it!<p>BTW has anyone tried a probabilistic dedup approach using soemthing like a bloom filter so you don&#x27;t have to store the entire dedup table of hashes verbatim? Collect groups of ~100 block hashes into a bucket each, and store a hyper compressed representation in a bloom filter. On write, lookup the hash of the block to write in the bloom filter, and if a potential dedup hit is detected, walk the 100 blocks in the matching bucket manually to look for any identical hashes. In theory you could do this with layers of bloom filters with different resolutions and dynamically swap out the heavier ones to disk when memory pressure is too high to keep the high resolution ones in RAM. Allowing the accuracy of the bloom filter to be changed as a tunable parameter would let people choose their preference around CPU time&#x2F;overhead:bytes saved ratio.

I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with &quot;little&quot; RAM (compared to the amount of disk storage attached).<p>Just store fingerprints in a database and run through that at night and fixup the block pointers...

You should use:<p><pre><code> cp --reflink=auto </code></pre> You get file level deduplication. The command above performs a lightweight copy (ZFS clone in file level), where the data blocks are copied only when modified. Its a copy, not a hard link. The same should work in other copy-on-write transactional filesystems as well if they have reflink support.

I wanted to use ZFS badly, but of course all data must be encrypted. It was surprising to see how usage gets much more complicated than expected and so many people just don’t encrypt their data because things get wild then.<p>Look, even Proxmox, which I totally expected to support encryption with default installation (it has „Enterprise“ on the website) does loose important features when trying to use with encryption.<p>Also please study the issue tracker, there are a few surprising things I would not have expected to exist in a productive file system.

I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.

Forget dedupe just use zfs compression, a lot more bang for your buck

General-purpose deduplication sounds good in theory but tends not to work out in practice. IPFS uses a rolling hash with variable-sized pieces, in an attempt to deduplicate data rysnc-style. However, in practice, it doesn&#x27;t actually make a difference, and adds complexity for no reason.

If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?

I&#x27;d love if dedicated hardware existing in disk controllers for calculating stuff like ECC could be enhanced to expose hashes of blocks to the system. Getting this for free for all your I&#x2F;O would allow some pretty awesome things.

When the lookup key is a hash, there&#x27;s no locality over the megabytes of the table. So don&#x27;t all the extra memory accesses to support dedup affect the L1 and L2 caches? Has anyone at OpenZFS measured that?<p>It also occurs to me that spacial locality on spinning rust disks might be affected, also affecting performance.

So if a sweet spot exists where dedup is widely beneficial then:<p>Is there an easy way to analyze your dataset to find if you&#x27;re in this sweet spot?<p>If so, is anyone working on some kind of automated partial dedup system where only portions of the filesystem are dedupped based on analysis of how beneficial it would be?

We have a read-heavy zpool with some data that&#x27;s used as part of our build process, on which we see a roughly 8x savings with dedup - and because of this ZFS dedup makes it economically viable for us to store the pool on NVMe rather than spinning rust.

What are the use cases where it makes sense to use de-dup? Backup comes to mind. What else?

I&#x27;ve used ZFS dedupe for a personal archive since dedupe was first introduced.<p>Currently, it seems to be reducing on-disk footprint by a factor of 3.<p>When I first started this project, 2TB hard drives were the largest available.<p>My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache.<p>Every few years, I try to do a better job of things but at this point, the best improvement would be radical simplification.<p>ZFS has served very well in terms of reliability. I haven&#x27;t lost data, and I&#x27;ve been able to catch lots of episodes of almost losing data. Or writing the wrong data.<p>Not entirely sure how I&#x27;d replace it, if I want something that can spot bit rot and correct it. ZFS scrub.

What happened to the issue with ZFS which occurred around half a year go?<p>I never changed a thing (because it also had some cons) and am believing that as long as a ZFS scrub shows no errors, all is OK. Could I be not seeing a problem?

In addition to the copy_file_range discussion at the end, it would be great to be able to applying deduplication to selected files, identified by searching the filesystem for say &gt;1MB files which have identical hash.

Off topic, any tool to deduplicate files across different external Hard disks?<p>Over the years I made multiple copies of my laptop HDD to different external HDDs, ended up with lots of duplicate copies of files.

My dream Git successor would use either dedupe or a simple cache plus copy-on-write so that repos can commit toolchains and dependencies and users wouldn’t need to worry about disk drive bloat.<p>Maybe someday…

Knowing that your storage has really good inline dedupe is awesome and will affect how you design systems. Solid dedupe lets you effectively treat multiple copies of data as symlinks.

Why are enterprise SANs so good at dedupe, but filesystems so bad? We use HPE Nimble (yeah, they changed the name recently but I can&#x27;t be bothered to remember it), and the space savings are insane for the large filesystems we work with. And there is no performance hit.<p>Some of this is straight up VM storage volumes for ESX virtual disks, some direct LUNs for our file servers. Our gains are upwards of 70%.

I&#x27;m storing a lot of text documents (.html) which contain long similiar sections and are thus not copies but &quot;partial copies&quot;.<p>Would someone know if the fast dedup works also for this? Anything else I could be using instead?

Any timing attacks possible on a virtualized system using dedupe?<p>Eg find out what my neighbours have installed.<p>Or if the data before an SSH key is predictable, keep writing that out to disk guessing the next byte or something like that.

So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He&#x2F;she&#x2F;whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe.<p>No, you won&#x27;t save much on a client system. That isn&#x27;t what the feature is made for.

can someone smarter than me explain what happens when instead of the regular 4kb block size in kernel builds we use 16kb or 64kb block size or is that only for the memory part, i am confused. Will a larger block size make this thing good or bad?

OT: does anyone have a good way to dedupe iCloud Photos. Or my Dropbox photos?

I don&#x27;t get it - many people here claim in this thread that VM base image deduplication is great use case for this. So lets assume there are couple of hundreds of VMs on a ZFS dataset with dedupe on, each of them ran by different people for different purposes entirely - some databases, some web frontends &#x2F; backends, minio S3 storage or backups ect - this might save you those measly hundreds of megabytes for linux system files those VMs might have in common ( even though knowing how many linux versions are out there with different patch levels - unlikely ) it will still not be worth it considering ZFS will keep track of each users individual files - databases and backup files and whatnot - data which is almost guaranteed to be unique between users so it will completely miss the point of ZFS deduplication. What am I missing?

Edit: disregard this, I was wrong and missed the comment deletion window.

I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn&#x27;t derail this discussion with opinions on contrast ratios...)<p>For less patient readers, note that the concise summary is at the bottom of the post, not the top.

Already don&#x27;t use ZoL because of their history of arms shrug-level support coupled with a lack of QA. ZoL != Solaris ZFS. It is mostly an aspirational cargo cult. Only a few fses like XFS and Ext4 have meaningful real-world, enterprise deployment hours. Technically, btrfs has significant (web ops instead of IT ops) deployment exposure due to its use on 10M boxes at Meta. Many non-mainstream fses also aren&#x27;t assured to be trustworthy because of their low usage and prevalent lack of thorough, formalized QA. There&#x27;s nothing wrong with experimentation, but it&#x27;s necessary to have an accurate understanding of the risk budget for a given technology for a given use-case.