Postmortem: Outage due to Elasticsearch’s flexibility and our carelessness

It&#x27;s not the first time I read about service (near) outages and post mortem that involve ElasticSearch. It&#x27;s marketed as NoSQL solution, but some devs don&#x27;t read the details in the documentation like &quot;Elasticsearch is <i>not</i> schema-less&quot;.<p>Knowing Lucene and its index structure for years, I wouldn&#x27;t advise to use a Lucene database as primary storage. Always keep the original data at least in a log-queue or in a separate database - so that you can rebuild the Lucene database.

Very interesting article, and a few lessons learned. And very well written. I feel like I can feel the tenseness in the team when you are debugging and searching for a solution under stress :)<p>Just wanted to tip you off to what I assume is a typo:<p><pre><code> But lack of automated regression testing would have caught this issue. </code></pre> I understand what you mean, but if you mean that literally then that is indeed a new and interesting paradigm in software testing :)

Excellent write up! Would configuring ES to alert instead of silently default type on new fields have helped ?<p>&quot;dynamic&quot;: &quot;strict&quot; <a href="https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;elasticsearch&#x2F;guide&#x2F;current&#x2F;dynamic-mapping.html" rel="nofollow">https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;elasticsearch&#x2F;guide&#x2F;current&#x2F;...</a>

This is something we encountered at Snowplow with our real-time loading of events into Elasticsearch [1]. It&#x27;s not an issue for us because we schema all our events, but it was interesting to observe. Here&#x27;s a summary in one slide:<p><a href="http:&#x2F;&#x2F;www.slideshare.net&#x2F;alexanderdean&#x2F;snowplow-analytics-from-nosql-to-sql-and-back-again&#x2F;27?src=clipshare" rel="nofollow">http:&#x2F;&#x2F;www.slideshare.net&#x2F;alexanderdean&#x2F;snowplow-analytics-f...</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;snowplow&#x2F;snowplow&#x2F;tree&#x2F;master&#x2F;4-storage&#x2F;kinesis-elasticsearch-sink" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;snowplow&#x2F;snowplow&#x2F;tree&#x2F;master&#x2F;4-storage&#x2F;k...</a>

Really awesome to read such a nice writeup of an error. It got me tangentially thinking about why we do not read stuff like:<p>When the patient was brought in, he was barely breathing. Due to the spots on his face, we assumed it was X. Then we made an incision in his pelvis to fix Y. Suddenly, he died. Gosh, we <i>really</i> should have kept an eye on the meter in the corner!<p>I understand things like litigation, but it makes me wonder to what extent we are being limited in medical achievements. From my outsider point of view, it seems that post mortems (ha!) are not used as much in the medical field other than for insurance &#x2F; legal reasons.

You&#x27;ve got what appears to be a vestigial comment in your article. It reads:<p><pre><code> (add more details here) </code></pre> But it looks like you&#x27;ve got plenty of details. Just thought I&#x27;d let you know ;-)

&gt; The values for field price in products mapping and the values for field price in promotions mapping (in the same index) will essentially be added to the same list at Lucene Segment level. And it will not fail or throw an exception.<p>Probably a stupid question but why can&#x27;t ElasticSearch translate both mappings into something like, &quot;products_price&quot;, and &quot;promotions_price&quot; before adding to the &#x27;list&#x27;?

It is for this reason Elasticsearch 1.x advocates using &#x27;copy_to&#x27; on mapping attributes that may encounter collision. Elasticsearch 2.x explicitly throws exception whenever it detects mapping collision.

I ran into the same issue on one of our apps too. Nice to see it written down so thoroughly!

nice writeup.. it strikes a good balance between tech details and narrative.

I&#x27;m not sure I understand the topic correctly, are you referring to carelessness in Elasticsearch or on your own behalf?