Hi,
just discovered that the (sometimes excessive large) golang mod disk cache is - in opposite to the compiler cache - not (yet) properly deduplicated.<p>Try yourself: Made a little hack/tool to deduplicate data via hardlinks (unix/i-node-filesystems tested only, not windows yet).<p><a href="https://github.com/paepckehh/fsdd">https://github.com/paepckehh/fsdd</a><p>(less than 1kLOC, 100% golang stdlib, no external deps)<p><pre><code> go install paepcke.de/fsdd/cmd/fsdd@latest
cd $GOMODCACHE && fsdd --hard-link .
FSDD [start] [/usr/store/gomod][hash:MAPHASH]
FSDD [_done] [time: 39.000221ms]
FSDD [stats] [files:13329] [inode(s): 8680] [sym.valid: 0] [sym.invalid: 0] [data blocks: 277.1 Mbytes]
FSDD [_info] [new deduplication savings][inode(s): 4649] [data blocks: 66.9 Mbytes]
</code></pre>
If you can reproduce and has it shows no downsides (remove/add should auto-dereference) it should be part of src/cmd/go/internal/mod* pack?<p>If you can confirm/field test, please report back!
I would love to write a quick stdlib/proposal.<p>At golang scale, it could make a real resource / enviromental dent!<p>It least for me, it cut my disk cache in half and reduced vfs fs mem-cache required as well, so should scale even on deduplicated fs like zfs/btrfs. Made my batch builds fast and snappy again!<p>Michael
(edit:first-post-ever-style-fixes)