TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Faster than the filesystem (2021)

216 pointsby madmax108over 2 years ago

20 comments

ChuckMcMover 2 years ago
That a database is faster than a nominal file system has been known for quite a while. It can&#x27;t quite replace them though. Microsoft tried really hard to have database for their root filesystem, investing a lot of time and effort but ultimately it fizzled, why? Important edge cases (like swap) that databases do really poorly.<p>That said, if you&#x27;re using a file system abstraction for complex and compound documents using a database is a really stellar way to go. In part because it doesn&#x27;t have the &quot;chunking&quot; problem where allocation blocks are used for both data and metadata in file systems so you pick a size that is least bad for both, versus one &quot;optimum&quot; size for disk io&#x27;s to keep the disk channel bandwidth highly utilized and the naming&#x2F;chunking part as records inside that.<p>I wrote a YAML &lt;=&gt; SQLite tool so that apps could use an efficient API to get at component types but the file could be &quot;exported&quot; to pure text. And it worked well in non-UTF string applications. (this was fine for the app which was orchestrating back end processes) At some point it would be interesting to move it to UTF-8 to see how that worked out.
评论 #34387799 未加载
评论 #34388181 未加载
评论 #34388164 未加载
评论 #34392434 未加载
评论 #34387917 未加载
评论 #34391926 未加载
评论 #34388005 未加载
评论 #34393293 未加载
评论 #34388288 未加载
polyrandover 2 years ago
As others have pointed out, that benchmark is from 2017. However, the &quot;SQLite: Past, Present, and Future&quot; paper[0] has an updated version of this benchmark (see section 4.3 Blob manipulation), and also compares it with DuckDB.<p>Edit:<p>Another thing that is sometimes forgotten when comparing SQLite to the filesystem is that files are hard[1]. It&#x27;s not only about performance, but also about all the guarantees that you get &quot;for free&quot;, and all the complexity you can remove from your codebase if you need ACID interactions with your files.<p>[0]: <a href="https:&#x2F;&#x2F;www.vldb.org&#x2F;pvldb&#x2F;vol15&#x2F;p3535-gaffney.pdf" rel="nofollow">https:&#x2F;&#x2F;www.vldb.org&#x2F;pvldb&#x2F;vol15&#x2F;p3535-gaffney.pdf</a><p>[1]: <a href="https:&#x2F;&#x2F;danluu.com&#x2F;file-consistency&#x2F;" rel="nofollow">https:&#x2F;&#x2F;danluu.com&#x2F;file-consistency&#x2F;</a>
评论 #34387935 未加载
评论 #34389528 未加载
summerlightover 2 years ago
This reminds me of WinFS, which was probably one of the most ambitious architectural project ever in the history of Windows yet failed to materialize. The vision was so attractive, encode all the semantic knowledge of file schema and metadata into relational filesystem. So you can programmatically query on whatever information about the filesystem and its content.<p>IIRC the problem was its performance. I don&#x27;t have any insider knowledge so cannot pinpoint the culprit but I suppose that the performance issue was probably not something fundamental tradeoff (as this article suggests) but more of its immature implementation. The storage technologies got much better nowadays so many of its problem could be tackled differently. Of course the question it has to answer is also different; is it a still worth problem to solve?
评论 #34387794 未加载
评论 #34388300 未加载
评论 #34387929 未加载
mklover 2 years ago
On Windows in my experience it&#x27;s at least a factor of 10. I worked on a script recently that reads ~20000 files a few kb each and extracts some info to generate a web page. I sped it up enormously just by putting the file contents into SQLite tables.
评论 #34387645 未加载
评论 #34387675 未加载
评论 #34387803 未加载
评论 #34391753 未加载
评论 #34387847 未加载
karmakazeover 2 years ago
The summary says:<p>&gt; SQLite reads and writes small blobs (for example, thumbnail images) 35% faster¹ than the same blobs can be read from or written to individual files on disk using fread() or fwrite().<p>&gt; The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database.<p>This should come as a surprise to no one. The rest of the article is only of interest in terms of how benchmarking is done, if that&#x27;s your thing.
mgover 2 years ago
Did they read and write all the data a filesystem handles?<p>Off the top of my head, the typical filesystem stores:<p><pre><code> - content - creation time - modification time - last access time - read&#x2F;write&#x2F;execute permissions - owner - group - position in the dir hirarchy</code></pre>
评论 #34387798 未加载
评论 #34388645 未加载
scotty79over 2 years ago
Maybe node_modules could move to SQLite.
评论 #34392070 未加载
BartjeDover 2 years ago
In 2017 you didnt have io uring yet. Though that doesn&#x27;t explain it for windows and android.
评论 #34389028 未加载
评论 #34387734 未加载
paulgbover 2 years ago
This could use (2017) (original benchmark date) or (2021) (last modified time) in the title.
cdbattagsover 2 years ago
Anyone else think maybe AWS (S3) has made this optimization already? Or would it just be a whole team of kernel engineers optimizing it there?<p>The overhead on CPU cycles this would save cloud storage systems... Can someone help me quantify the potential savings?<p>Edit:<p>They specifically don&#x27;t list their storage medium on their marketing:<p><a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;storage-classes&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;storage-classes&#x2F;</a>
评论 #34387638 未加载
评论 #34387787 未加载
评论 #34387899 未加载
评论 #34387662 未加载
detritesover 2 years ago
Title should have (2017)?
gnufxover 2 years ago
Long ago we had a contrasting experience. It had been assumed that the implemented &quot;database&quot; (not a general database) for storing nuclear spectroscopy data was needed because the filesystem was too slow. However, one of our &quot;physicist programmers&quot; decided to do the experiment, and found it wasn&#x27;t so, and the system was re-designed around directories of files of spectra.<p>On the other hand, the other facility in the lab had consulted Logica on storage of similar data, who viewed x-y (or multiple dependent variables) data as tables suitable for storage in their early RDB, Rapport. That wasn&#x27;t actually used in production, and the storage format for the table model was unfortunately usually mangled by data acquisition systems writing files.
k__over 2 years ago
That claim sounds more preposterous than it actually is, considering that databases and filesystems are both just software using the same hardware in a different manner.
mpweiherover 2 years ago
More accurate title: &quot;Operations within a file are faster than directory operations&quot;
评论 #34388970 未加载
shrubbleover 2 years ago
25 years ago this was done&#x2F;tried with Pgfs, an NFS server with Postgres underneath it... <a href="https:&#x2F;&#x2F;www.linuxjournal.com&#x2F;article&#x2F;1383" rel="nofollow">https:&#x2F;&#x2F;www.linuxjournal.com&#x2F;article&#x2F;1383</a>
eddsh1994over 2 years ago
Would it make sense to store images as blobs if I’m building a web app using SQLite then? Or is this specifically small images only? Saves having to do backups of data and images separately :)
评论 #34387606 未加载
评论 #34388142 未加载
评论 #34387714 未加载
enriqutoover 2 years ago
Is it possible to mount a database so that its rows (or columns, or whatever you have selected upon the call to mount) are accessible as regular files?
评论 #34388807 未加载
评论 #34389328 未加载
fexecveover 2 years ago
How does it do this? Does it cache changes and only write them every so often? Does it keep a file open to write uncommitted changes?
评论 #34393216 未加载
crabboneover 2 years ago
I&#x27;ll start with an anecdote. Scroll down if you want just the conclusion.<p>I work in test automation. In the general storage area. I worked for some years with a distributed filesystem (think something like Lustre, but modern and fast), then worked with something like DRBD, but, again more modern. Not sure how fast (never ran any benchmarks on DRBD). I had to deal with filesystems like Ceph&#x27;s filesystem, BeeGFS...<p>Anyways. When I worked on the DRBD analog, let&#x27;s call the product &quot;R&quot;, one of my tasks was to figure out how well would a database work on top of R. Well, &quot;database&quot; is a very broad term. I figured I&#x27;d concentrate on using couple of well-known relational databases. PostgreSQL turned out to have the most to offer in terms of insight into its performance. Next, I&#x27;d have to find a suitable configuration for the database. And that&#x27;s where things got really complicated. To spare you the gory details: the more memory you can spare for the benefit of your database server -- the better, the more you can relax the requirement of synchronizing with persistent storage (i.e. fsync and friends) -- even better.<p>In the end of the day, I had to abandon this kind of testing because, essentially, if given enough memory, or enough replicas (allows not to care about destaging to persistent storage) the bigger numbers I could produce, which made the question &quot;how well does R compare to a plain block device?&quot; irrelevant.<p>---<p>Fast-forward to this article. It gives out a vibe of &quot;filesystems are not efficient, if you re-arrange something in the logic of doing I&#x2F;O you can gain more performance!&quot;. And that really reads like &quot;10 things doctors don&#x27;t want you to know!&quot; advertorial. It&#x27;s similar to the misguided idea I often encounter with people who aren&#x27;t system programmers that &quot;mmap is faster&quot;.<p>Now, understanding why &quot;mmap is faster&quot; is nonsense will help understand why benchmarking a database on top of a filesystem and comparing performance doesn&#x27;t make a lot of sense. So, in order to properly compare the speed, we need to make sure we compare both good and bad paths. What happens when I&#x2F;O errors occur when memory-mapping files and when using other system API? I invite you to explore this question on your own. Another question you need to ask yourself in this comparison is &quot;how well does this process scale&quot;, on a single CPU, single block device, multiple CPUs single block device, single CPU multiple block devices... And on top of this, what if we consider Harvard architecture (to an extend, IBM&#x27;s mainframes are it, at least there general I&#x2F;O is separate from the rest of computing). In other words, what if our hardware knows &quot;some tricks&quot;? (Other examples include the kinds of drivers and protocols used to talk to the hardware and whether the storage software, even if running on top of a filesystem will know &#x2F; be able to take advantage of these, i.e. NVMe allows big degree of parallelism, but will &quot;mmap&quot; be able to utilize that? Especially if multiple files are mapped at the same time?)<p>And, of course, there is a difference between what&#x27;s been actually done and what guarantees can be given about the state of data during and after the I&#x2F;O completes. And these details will greatly depend on the details of the specific filesystem being used. For example, if you wanted to write to the device directly (as in with &quot;ODIRECT | OSYNC&quot;), but the filesystem is something like ZFS or Btrfs (i.e. it needs to checksum your data beside other things), then you might get confused about what you are actually comparing (direct I&#x2F;O would imply less I&#x2F;O than is actually necessary to give durability guarantees that may not be given by an alternative storage).<p>---<p>So, a better title for the article would have been &quot;It&#x27;s possible to carve out a use-case, where SQLite works faster than a similar process designed to only use system API to access a filesystem&quot;. Which is essentially saying: you, the programmer, don&#x27;t know how to use a filesystem as well as we do... And, in my experience, most programmers are clueless when it comes to using filesystem or any other kind of storage really. So, that&#x27;s not surprising, but is still not as sensationalist as the original title.
dan-robertsonover 2 years ago
I wonder how using SQLite like this compares to something like squashFS.
评论 #34387605 未加载