I’ve used LMDB as a simpler alternative to SQLite as “an alternative to fopen”. The goal was simply robust file writes in the face of unpredictable server reboots for a tiny Python program writing data to be processed later by a tiny C++ program.<p>That’s harder than it sounds to roll by hand with fopen. SQLite with write ahead logging is pretty much as good as it gets for reliablity, but SQL at all was overkill for the task. LMDB is a close second and it’s memory mapped key-value interface is much simpler. . Would write again.<p><a href="https://lwn.net/Articles/457667/" rel="nofollow">https://lwn.net/Articles/457667/</a>
I like LMDB, but why does ~most sql/nosql use LSM/rocksdb compared to it ? At least the ones going for read-speed ? Cause of missing WAL ?<p>There is also a fork? who claims is better/more-features than LMDB: <a href="https://github.com/leo-yuriev/libmdbx" rel="nofollow">https://github.com/leo-yuriev/libmdbx</a>
LMDB is a very good choice for many well-known reasons. I don't need to expand here, the advantages are well documented, and more and more projects are choosing LMDB.<p>However LMDB does not solve all problems, and can be a bad choice for some, and I couldn't find this documented anywhere. Specifically write-intensive workload. Why?<p>- LMDB by default provides full ACID semantics, which means that after every key-value write committed, it needs to sync to disk. Apparently if this happens tens of times per second, your system performance will suffer.<p>- LMDB provides a super-fast asynchronous mode (`MDB_NOSYNC`), and this is the one most often benchmarked. Writes are super-fast with this. But a little known fact is that you lose all of ACID, meaning that a system crash can cause total loss of the database. Only use `MDB_NOSYNC` if your data is expendable.<p>In short, I would advise against LMDB if you are expecting to have more than a couple of independent writes per second. In this case, consider choosing a database that syncs to disk only occasionally, offering just ACI semantics (without Durability, which means that a system crash can cause loss of only the last seconds of data).
Is there a design doc or talk about the internals?<p>In particular are there any good resources about the details of using memory mapping?<p>I know how to implement persistent data structures (and it seems like lmdb is just a persistent b+-tree). But I don't know how to make it persist to disk. Is it as simple as using a memory mapped file for all memory allocations? Can all data structures be turned into a "database" in this way? If your workload fits in memory is there any performance difference between in-memory data structures? When do writes actually flush? What happens if multiple processes use the same file? etc
> Data pages use a copy-on- write strategy so no active data pages are ever overwritten, which also provides resistance to corruption and eliminates the need of any special recovery procedures after a system crash.<p>But I imagine this is somewhat slower than keeping a log (and rewinding it if necessary)?
The web page seems to suggest that robust POSIX semaphores are Linux-specific, while they've been available in FreeBSD for quite some time. I wonder if they detect it properly, or is there some actual problem in FreeBSD's implementation?