TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Call me maybe: Elasticsearch 1.5.0

276 点作者 tylertreat大约 10 年前

11 条评论

teraflop大约 10 年前
&gt; How often the translog is fsynced to disk. Defaults to 5s. [...] In this test we kill random nodes and restart them. [...] In Elasticsearch, write acknowledgement takes place before the transaction is flushed to disk, which means you can lose up to five seconds of writes by default. In this particular run, ES lost about 10% of acknowledged writes.<p>Something bothers me about this: if the bug was merely a failure to call fsync() before acknowledging an operation, then killing processes shouldn&#x27;t be enough to cause data loss. Once you write to a file and the syscall returns, the written data goes into the OS&#x27;s buffers, and even if the process is killed it won&#x27;t be lost. The only time fsync matters is if the entire machine dies (because of power loss or a kernel panic, for instance) before those buffers can be flushed.<p>So is the data actually not even making it to the OS before being acked to the client? Or is Jepsen doing something more sophisticated, like running each node in a VM with its own block device instead of sharing the host&#x27;s filesystem?
评论 #9490448 未加载
评论 #9490329 未加载
评论 #9490610 未加载
评论 #9490117 未加载
w8rbt大约 10 年前
The &#x27;Recap&#x27; section has good advice to address the issue of data loss:<p><i></i><i>My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.</i><i></i>
评论 #9490141 未加载
Meekro大约 10 年前
Elasticsearch has its flaws, but are there really any alternatives that do what it does? In my case, I need to do fulltext search across millions of documents -- nearly a TB of text in all, and still growing. Many database engines can do basic fulltext, but what else can scale like Elasticsearch does while offering powerful fulltext features like fuzzy matching and &quot;more like this?&quot;
评论 #9489925 未加载
评论 #9489957 未加载
评论 #9489946 未加载
评论 #9490303 未加载
评论 #9489962 未加载
评论 #9490062 未加载
评论 #9490367 未加载
评论 #9490589 未加载
评论 #9491239 未加载
phamilton大约 10 年前
TL;DR; All previous data loss scenarios still exist, however the window during which data can be lost is much smaller.
评论 #9489853 未加载
larrywright大约 10 年前
I bet the people who work for these database companies have nightmares about getting bug reports from Kyle.<p>But seriously, his work is super impressive. Kudos to Stripe for funding this.
评论 #9491328 未加载
beachstartup大约 10 年前
waaaay back in the mid 2000s when search engines were large, complex, expensive pieces of enterprise software not unlike a commercial database, it was always assumed (and the customer told) that the search engine cuts corners for the sake of speed, in both indexing and searching, and that it should never be counted on as a database, or any kind of authoritative data store. it does one thing: search staggering amounts of information quickly at acceptable levels of performance and accuracy.<p>looks like expectations have moved beyond that, which is good, but it&#x27;s not an easy problem to solve... especially with billions or trillions of documents in the index. leaving out all the the important stuff that made a database slow is what made a search engine fast.
jodah大约 10 年前
Key takeaway for me:<p>&quot;crash-restart durable storage is a prerequisite for most consensus algorithms, I have a hunch we’ll see stronger guarantees as Elasticsearch moves towards consensus on writes.&quot;<p>It seems like ElasticSearch is inching towards a proper consensus algorithm, at least optionally, which makes me wonder yet again, why not just implement RAFT? While I won&#x27;t speculate I will point out that the answer for other systems appears to have been related to ego (ie, distributed systems are easy).
评论 #9491638 未加载
temuze大约 10 年前
I&#x27;m pretty pleased with Elasticsearch&#x27;s progress on durability. The snapshot&#x2F;restore feature has been pretty nice to work with!<p>That said, having a &quot;single source of truth&quot; and regularly refreshing Elasticsearch is a <i>huge</i> pain in the butt. I currently maintain a ~500 line syncing script that takes ~15 minutes to run.<p>Adding a new field in a doc_type means:<p>- Adding a column in Postgres<p>- Adding a field in the doc_type mapping (I think being explicit with the field is better practice)<p>- Adding code to the syncer to update the field.<p>Ouch.<p>Also, the syncing scripts required a steep learning curve. At first, I was upserting everything and the syncing script took forever. To solve this, I made the syncer fetch all the documents in Postgres and all the documents in Elasticsearch then update the specific changes (thankfully, the dataset is small enough that it easily fits in memory).<p>I&#x27;d really love to scrap this portion of my infrastructure...
评论 #9490175 未加载
评论 #9490145 未加载
评论 #9490394 未加载
Jarlakxen大约 10 年前
One more thing to add to the Elastic reliability:<p>&gt; We recommend installing the Java 8 update 20 or later, or Java 7 update 55 or later. Previous versions of Java 7 are known to have bugs that can cause index corruption and data loss. Elasticsearch will refuse to start if a known-bad version of Java is used.<p><a href="http:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;elasticsearch&#x2F;reference&#x2F;1.5&#x2F;setup.html" rel="nofollow">http:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;elasticsearch&#x2F;reference&#x2F;1.5&#x2F;s...</a>
nyir大约 10 年前
Kind of OT, but losing data in these cases is not even what I&#x27;m most concerned about with ES: I had to recreate the ES cluster way too many times now that I&#x27;m really glad that PostgreSQL keeps on running, or just restarts without bricking the data files, even after OOM or out of disk errors.
dang大约 10 年前
Also <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9475620" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9475620</a>.