Mtime comparison considered harmful (2018)

Weird that there is such detailed technical information alongside this statement:<p>&gt; And anyway, purists might argue that the &quot;content&quot; of a directory doesn&#x27;t change when the files it points to change; the content is merely a list of filenames and inode numbers, after all, and those stay the same, no matter what happens inside those inodes. Purists make me sad.<p>Or maybe Unix makes the author sad? You can’t wish the OS into doing something different with directory files when the set of contained files has not changed (but a file was modified) so that these Make troubles could be handled differently.<p>Am I missing something?

&gt; Annoyingly, when you update the content of a file, the mtime of its containing directory is not changed. All sorts of very convenient tree traversals would be possible if the directory mtime were updated (recursively to the root) when contained files changed, but no. This is probably because of hardlinks:<p>He lost me here. As much as that&#x27;s a <i>would be cool</i> kinda feature it smacks of a special kind of tunnel vision: what&#x27;s the point of anything but what <i>you</i> happened to be working on?<p>Consider how many writes happen on your system. Consider what sliver of a fraction of them even come under mtime scrutiny. The idea is to amplify those writes all the way up the directory hierarchy? For every write? And contend on ever write to &#x2F;home&#x2F;?<p>We already needed a noatime, if it wasn&#x27;t for make I could imagine a nomtime too.

Good news is that there is work on making the i_version field available via statx in Linux: <a href="https:&#x2F;&#x2F;lore.kernel.org&#x2F;linux-nfs&#x2F;20220826214703.134870-1-jlayton@kernel.org&#x2F;T&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lore.kernel.org&#x2F;linux-nfs&#x2F;20220826214703.134870-1-jl...</a><p>So we might have a reliable mtime alternative in Linux soon in 2022.

The title should be: “I don’t like Mtime comparison. Here’s why.”<p>If your name is Dijkstra and it’s 1968 feel free to use the phrase “considered harmful”, otherwise use plain language.

We&#x27;re currently dealing with crashes across all Qt applications using QML on NixOS [1], since Qt utilizes the binary&#x27;s mtime to invalidate the cache of embedded QML resources.<p>Since all builds have an mtime of 0 as timestamps are the biggest source of reproducibility issues [2], QML loads outdated cache objects which will then load invalid bytecode at runtime and therefore causing a crash.<p>Our initial plan was to utilize a hash of the binary [3] which should be IMHO the most straightforward, unlikely to break and future-proof way to solve the problem. The currently suggested implementation&#x27;s performance could most likely be improved by generating the hash once on startup instead of every time when a QML resource is loaded, but I believe since it&#x27;s already cached in memory, hashing is not that expensive, binary sizes are usually relatively small and resource loading (of embedded ones) doesn&#x27;t happen that often, it should be already reasonably fast (real performance test haven&#x27;t been done yet).<p>The big challenge will be upstreaming it, once we&#x27;ve proven it to properly work. The current approach has been apparently rejected [4] and was declared to be a downstream issue, which I personally don&#x27;t agree with, since sooner or later reproducible builds will become the norm and therefore will affect everyone. Current ideas to work around it require individual solutions per distribution&#x2F;ISV, as this would mean they&#x27;d have to come up with domain specific criteria for cache invalidation (as the store path&#x2F;derivation hash on NixOS) and to maintain a downstream patch for this solution and furthermore wouldn&#x27;t work for local build processes (e.g. from within an IDE).<p>Lesson of the day: never use mtimes, they&#x27;ll bite you in the ass sooner or later!<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;NixOS&#x2F;nixpkgs&#x2F;issues&#x2F;177720" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;NixOS&#x2F;nixpkgs&#x2F;issues&#x2F;177720</a><p>[2] <a href="https:&#x2F;&#x2F;reproducible-builds.org&#x2F;docs&#x2F;timestamps&#x2F;" rel="nofollow">https:&#x2F;&#x2F;reproducible-builds.org&#x2F;docs&#x2F;timestamps&#x2F;</a><p>[3] <a href="https:&#x2F;&#x2F;github.com&#x2F;qt&#x2F;qtdeclarative&#x2F;commit&#x2F;5106afcd76e377a6bbcdda578b9efd5167447ec0" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;qt&#x2F;qtdeclarative&#x2F;commit&#x2F;5106afcd76e377a6b...</a><p>[4] <a href="https:&#x2F;&#x2F;github.com&#x2F;NixOS&#x2F;nixpkgs&#x2F;issues&#x2F;177720#issuecomment-1244763504" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;NixOS&#x2F;nixpkgs&#x2F;issues&#x2F;177720#issuecomment-...</a>

I really do think it&#x27;s unfortunate that the OS cannot keep something like a &quot;write count&quot; for each file. That would be enough. Possibly it would even be good enough to know the number of times a file had been opened for writing.

&gt; Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures. You normally don&#x27;t see any symptoms of this except that hardlinks are suspiciously slow on MacOS. But in exchange for the slowness, the kernel actually can look up all filenames of a hardlink if it wants. I think this has something to do with Aliases and finding .app files even if they move around, or something.<p>This reads weird to me.<p>As the author points out, a directory entry (hardlink) is just a filename pointing to an inode. When you first create a file, and there is only one filename for that file, that entry is a hardlink. So in that text the first occurrence of &quot;hardlink&quot; should probably read &quot;the kernel does know all the filenames of an inode&quot;.<p>But then it gets weird. When the author says &quot;hardlinks are suspiciously slow on MacOS&quot;, it sounds like they&#x27;re saying &quot;accessing files is suspiciously slow on MacOS&quot; - because accessing any file in the filesystem goes through hardlinks. All links to a file, including the first, are hardlinks.

It would be neat if filesystems like btrfs or ZFS could expose their internal checksum of the file data to userspace, to quickly see if two files are having identical content (eg. think of rsync). (Assuming the hashes computed internally the filesystem are actually purely based on the file data, and not for example metadata like block pointers. But, dedup-capable filesystems should surely have such checksum internally...).

apropos<p>&gt; Linux added an O_NOATIME<p>!!Con 2016 - How I fixed UNIX atime! With 10 lines of code and feminism!!! By Valerie Aurora <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=fHjsdyN4UK0" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=fHjsdyN4UK0</a>

&gt; mtime is one of the inode fields, so whenever mtime changes, ctime also changes<p>Not in my experience.