I am going to write this comment with a large preface: I don't think it is ever helpful to be an absolutist. For every best-practice/"right way" to do things, there are circumstances when doing it another way makes sense. That can be a ton of reasons for that, be it technical, money/time, etc. The best engineering teams aren't those that just blindly follow what others say is a best practice but understand the options and make an informed choice. None of the following comment is at all commentary on questDB, as they mention in the article, <i>many</i> databases use similar tools.<p>With that said, after reading the first paragraph I immediately searched the article for "mmap" and had a good sense of where the rest of this was going. Put simply, it is just really hard to consider what the OS is going to do in all situations when using mmap. Based on my experience, I would guess that a <i>ton</i> of people reading this comment have hit issues that, I would argue, is due to using mmap. (Particularly looking at you prometheus).<p>All things told, this is a pretty innocuous incident of mmap causing problems, but I would encourage any aspiring DB engineers to read <a href="https://db.cs.cmu.edu/mmap-cidr2022" rel="nofollow">https://db.cs.cmu.edu/mmap-cidr2022</a> as it gives a great overview of the range of problems that can occur when using mmap<p>I think some would argue that mmap is "fine" for append only workloads (and is certainly more reasonable compared to a DB with arbitrary updates) but even here, lots of factors like metadata, scaling number of tables, etc will <i>eventually</i> bring you to hit some fundamental problems when using mmap.<p>The interesting opportunity in my mind, especially with improvements in async IO (both at FS level and in tools like rust), is to build higher level abstractions that bring the "simplicity" of mmap, but with more purpose-built semantics ideal for databases.
Going through mmap for bulk-ingest sucks because the kernel has to fault in the contents to make what's in-core reflect what's on-disk before your write access to the mapped memory occurs. It's basically a read-modify-write pattern even when all you intended to do was write the entire page.<p>When you just use a write call you provide a unit of arbitrary size, and if you've done your homework that size is a multiple of page size and the offset page-aligned. Then there's no need for the kernel to load anything in for the written pages; you're providing everything in the single call. Then you go down the O_DIRECT rabbithole every fast linux database has historically gone down.
Am I the only one surprised to read that this database relies on periodic flushing (every 30s by default) with no manual syncs at all? I guess it’s metrics so 30s of data loss is fine? I dunno about that. Data loss is usually due to a power failure, and the metrics collected right before a power failure are important.
Seems like using memory mapped files for a write-only load is the sub optimal choice. Maybe I’m mistaken but surely using an append-only file handle would be simpler than changing the behaviour of how memory mapped files are cached like they did for their solution?
I know sharing ChatGPT/GPT/AI generated text in comments here can be unappealing, but I would like to share this one as I feel that I managed to get ChatGPT-4 to summarize this article using a non-computer analogy pretty well:<p>"Imagine you work in a library where you store books on shelves. Your primary task is to take new books and put them on the shelves (write-only load). You don't expect to read the books often, so the number of times you need to open and read the books should be minimal.<p>One day, you notice that several books are being opened and read more often than expected, even though your main task is to put away new books. This is confusing and unexpected, so you start investigating why this is happening.<p>After some investigation, you find out that the library assistant (the operating system) is trying to be helpful by anticipating which books might be needed next and opening them ahead of time (readahead). This anticipation works well when there is plenty of shelf space (memory) available. However, when the library gets crowded (memory pressure), the assistant starts anticipating the wrong books, causing unnecessary book openings (phantom reads).<p>To resolve this issue, you tell the library assistant to stop anticipating which books to open (disabling readahead) when you're just putting away new books. This solves the problem and reduces the number of unnecessary book openings. The experience teaches you the importance of understanding how the library assistant works and shows that addressing unexpected issues can lead to improvements in the overall library system."
TLDR; "Ingestion of a high number of column files under memory pressure led to the kernel starting readahead disk read operations, which you wouldn't expect from a write-only load. The rest was as simple as using madvise in our code to disable the readahead in table writers."
There are other methods you can use to increase performance under memory pressure, but you'd end up handling i/o directly and maintaining your own index of memory and disk accesses, page-aligned reads/writes, etc. It would be easier to just require your users buy more memory, but when there's a hack like this available, that seems preferable to implementing your own VMM and disk i/o subsystem.