Heka by Mozilla is another alternative - <a href="http://hekad.readthedocs.org/en/latest/" rel="nofollow">http://hekad.readthedocs.org/en/latest/</a>. I have been running in production to ship docker logs to our Elasticsearch cluster.
> We will go through the installation process, basic setup, listen to events through the HTTP interface, and look at a simple use case of storing HTTP events into a MongoDB database.<p>...and I'm out.
I'm just starting to adopt fluentd but I'm scared by the fact that the "official" drivers have different interfaces and unclear leadership.<p>e.g.<p><a href="https://github.com/fluent/fluent-logger-php" rel="nofollow">https://github.com/fluent/fluent-logger-php</a>
<a href="https://github.com/fluent/fluent-logger-python" rel="nofollow">https://github.com/fluent/fluent-logger-python</a><p>and a bit of drama<p><a href="https://github.com/fluent/fluent-logger-php/issues/36" rel="nofollow">https://github.com/fluent/fluent-logger-php/issues/36</a>
> Its built-in reliability through memory and file-based buffering to prevent inter-node data loss have...<p>I wonder if we can expect "Call me maybe - fluentd" from Aphyr soon. ;)
This feels like they are trying very hard to avoid calling their syslog a syslog, probably with good reason. A lot of distributions provide a very minimalist build of whichever syslog daemon they've decided to use, and as such, people get the idea that syslogs can't write to databases, or parse json, or listen on pipes or any other number of things modern syslogs can do.
For years I've been considering coding a tamper proof logger, something where each entry has a hash that depends on the entry's log and the hash of the previous entry. This could help detect potential system compromise.<p>I haven't really taken the time to look for a logger with such a feature, it would be nice to know if fluentd has something like this.
I've been using Fluentd for years, and it's a super useful tool.<p>It initially had some memory leaks that prevented us to use it in production, but it's now very stable. Writing new input/output plugins is extremely simple. And yes, it's written in Ruby, but give it a spin before judging; it's fast enough for most needs.
Is there any benefit/disadvantage between fluentd and logstash ? I am not using either one, but I'll need to centralize my logs soon. My understanding tells me that these are two very similar projects, but I might be wrong.
Isn't Fluentd in Ruby though? It's 2015 and we need something like this in Go [0] [1] or Rust [2].<p>[0] Heka: <a href="https://hekad.readthedocs.org/" rel="nofollow">https://hekad.readthedocs.org/</a><p>[1] Chainsd: <a href="https://github.com/mikeszltd/chainsd" rel="nofollow">https://github.com/mikeszltd/chainsd</a><p>[2] Flogger: <a href="https://github.com/jedisct1/flowgger" rel="nofollow">https://github.com/jedisct1/flowgger</a>
We'd love to use a ruby-based solution like this, but the docs say it will lose data whenever the receiving end crashes. Any plans to fix that?<p>The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?
I started building something of similar concept a while ago. But had to pause the development for some time. It is written in Go, so much easier to install etc. It has plugins for Hadoop, Mongo, RabbitMQ, File and stdout :) Can take data from tail, HTTP, RabbitMQ, heartbeat and other chains.<p><a href="https://github.com/mikeszltd/chainsd" rel="nofollow">https://github.com/mikeszltd/chainsd</a>