> How often the translog is fsynced to disk. Defaults to 5s. [...] In this test we kill random nodes and restart them. [...] In Elasticsearch, write acknowledgement takes place before the transaction is flushed to disk, which means you can lose up to five seconds of writes by default. In this particular run, ES lost about 10% of acknowledged writes.<p>Something bothers me about this: if the bug was merely a failure to call fsync() before acknowledging an operation, then killing processes shouldn't be enough to cause data loss. Once you write to a file and the syscall returns, the written data goes into the OS's buffers, and even if the process is killed it won't be lost. The only time fsync matters is if the entire machine dies (because of power loss or a kernel panic, for instance) before those buffers can be flushed.<p>So is the data actually not even making it to the OS before being acked to the client? Or is Jepsen doing something more sophisticated, like running each node in a VM with its own block device instead of sharing the host's filesystem?
The 'Recap' section has good advice to address the issue of data loss:<p><i></i><i>My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.</i><i></i>
Elasticsearch has its flaws, but are there really any alternatives that do what it does? In my case, I need to do fulltext search across millions of documents -- nearly a TB of text in all, and still growing. Many database engines can do basic fulltext, but what else can scale like Elasticsearch does while offering powerful fulltext features like fuzzy matching and "more like this?"
I bet the people who work for these database companies have nightmares about getting bug reports from Kyle.<p>But seriously, his work is super impressive. Kudos to Stripe for funding this.
waaaay back in the mid 2000s when search engines were large, complex, expensive pieces of enterprise software not unlike a commercial database, it was always assumed (and the customer told) that the search engine cuts corners for the sake of speed, in both indexing and searching, and that it should never be counted on as a database, or any kind of authoritative data store. it does one thing: search staggering amounts of information quickly at acceptable levels of performance and accuracy.<p>looks like expectations have moved beyond that, which is good, but it's not an easy problem to solve... especially with billions or trillions of documents in the index. leaving out all the the important stuff that made a database slow is what made a search engine fast.
Key takeaway for me:<p>"crash-restart durable storage is a prerequisite for most consensus algorithms, I have a hunch we’ll see stronger guarantees as Elasticsearch moves towards consensus on writes."<p>It seems like ElasticSearch is inching towards a proper consensus algorithm, at least optionally, which makes me wonder yet again, why not just implement RAFT? While I won't speculate I will point out that the answer for other systems appears to have been related to ego (ie, distributed systems are easy).
I'm pretty pleased with Elasticsearch's progress on durability. The snapshot/restore feature has been pretty nice to work with!<p>That said, having a "single source of truth" and regularly refreshing Elasticsearch is a <i>huge</i> pain in the butt. I currently maintain a ~500 line syncing script that takes ~15 minutes to run.<p>Adding a new field in a doc_type means:<p>- Adding a column in Postgres<p>- Adding a field in the doc_type mapping (I think being explicit with the field is better practice)<p>- Adding code to the syncer to update the field.<p>Ouch.<p>Also, the syncing scripts required a steep learning curve. At first, I was upserting everything and the syncing script took forever. To solve this, I made the syncer fetch all the documents in Postgres and all the documents in Elasticsearch then update the specific changes (thankfully, the dataset is small enough that it easily fits in memory).<p>I'd really love to scrap this portion of my infrastructure...
One more thing to add to the Elastic reliability:<p>> We recommend installing the Java 8 update 20 or later, or Java 7 update 55 or later. Previous versions of Java 7 are known to have bugs that can cause index corruption and data loss. Elasticsearch will refuse to start if a known-bad version of Java is used.<p><a href="http://www.elastic.co/guide/en/elasticsearch/reference/1.5/setup.html" rel="nofollow">http://www.elastic.co/guide/en/elasticsearch/reference/1.5/s...</a>
Kind of OT, but losing data in these cases is not even what I'm most concerned about with ES: I had to recreate the ES cluster way too many times now that I'm really glad that PostgreSQL keeps on running, or just restarts without bricking the data files, even after OOM or out of disk errors.
Also <a href="https://news.ycombinator.com/item?id=9475620" rel="nofollow">https://news.ycombinator.com/item?id=9475620</a>.