Using ElasticSearch and Logstash to Serve Billions of Searchable Events

I&#x27;ve used ElasticSearch &amp; Logstash (&amp; Kibana, also shown here), and I&#x27;ve been nothing but impressed.<p>My primary concern was how it would cope when data exceeded RAM (I like to call this the MongoDB fallacy). However, based on this post it looks like the ElasticSearch setup works just fine.

Have you considered using any Graphite alternatives? I found it to be very slow with rendering data, especially when multiple arrays are being used as sources. Also, it lacks some features that graphs rendering software is generally supposed to have, at least the half-a-year old version.

Elasticsearch is really awesome for keeping large amount of searchable data. I used it in a previous application where we stored millions of items a week.<p>For data retention I had different indexes with different TTLs, depending on the type of queries that hit them (queries that only dealt with frequent items were sent to an index with a very short TTL).<p>For graphing I also used Graphite, with metrics (<a href="http://metrics.codahale.com/" rel="nofollow">http:&#x2F;&#x2F;metrics.codahale.com&#x2F;</a>) for sending data from Java programs and scales (<a href="https://github.com/Cue/scales" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Cue&#x2F;scales</a>) for sending data from Python applications.<p>The only problem I had was tuning it for faceting (Faceting consumed lots of RAM).

Here is a blogpost we (<a href="https://commando.io" rel="nofollow">https:&#x2F;&#x2F;commando.io</a>) wrote on shipping nginx access logs using LogStash and ElasticSearch<p><a href="https://medium.com/devops-programming/b01bd0876e82" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;devops-programming&#x2F;b01bd0876e82</a>

After reading this article, I checked out logstash.net today nearly after an year.... I am happy to see they added so many connectors and it has grown so much.... This open source software can be serious competition to splunk..

How much time was involved in building this setup? Just curious.

Is this a compelling OSS replacement for Splunk?

Also, are you using local statsd installations at each server that is sending stats, or you keep statsd separately?

Anyone used elasticsearch as a database, like mongodb ?