I was dragging my feet to build a log shipper solution. I was going to use Filebeat -> ElasticSearch -> Kibana.<p>This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations.<p>Now, if I can make a suggestion for your next/additional project.....
A neat system metric collector in Rust that can export to Prometheus with same principles.<p>Low memory footprint,<p>Rust,<p>Single binary,<p>Customizable with a single config file without spending hours in manuals,<p>Stdin, Stderr -> transform -> Prometheus.<p>I’m learning Rust and eventually plan to build such a solution but I think a lot of this project can be repurposed for what I asked much faster than building a new one.<p>Cheers on this open source project. I will contribute whatever I can. Thanks!!
Seems similar to Veneur (like many other projects mentioned in comments here; didn't realize this space was so crowded!) - down to the first two letters of the name: <a href="https://github.com/stripe/veneur" rel="nofollow">https://github.com/stripe/veneur</a><p>Veneur is more metrics-focused, but might offer inspiration as you work on metrics support in Vector - in particular the SSF source, internal aggregation, and Datadog and SignalFX sinks.
We use a rather bespoke syslog -> clickhouse log sink (<a href="https://github.com/discordapp/punt/tree/clickhouse" rel="nofollow">https://github.com/discordapp/punt/tree/clickhouse</a>) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!
Just a heads up: There are several figures in your docs where the entirety of the useful information on the page is in the image and they don't have alt tags or any accessible way to get the information (that I can find anyway). e.g. <a href="https://docs.vector.dev/use-cases/security-and-compliance" rel="nofollow">https://docs.vector.dev/use-cases/security-and-compliance</a>
Could this replace a simple fluentd setup right now or are there still major functionalities missing?<p>Specifically, I'm ingesting nginx logs in JSON format, cleaning up invalid UTF8 bytes (usually sent in the forwarded-for header to exploit security vulnerabilities), and sending to elasticsearch on an automated 90 day retention policy (daily indexes).<p>Seems like a fairly common use case for webservers.
Well logstash is now supporting good persistent queue.
<a href="https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#durability-persistent-queues" rel="nofollow">https://www.elastic.co/guide/en/logstash/current/persistent-...</a><p>I don't know why author didn't put correctness tick mark on it.
How does this compare to telegraf?<p><a href="https://github.com/influxdata/telegraf" rel="nofollow">https://github.com/influxdata/telegraf</a><p>Biggest thing that pops out to me is LUA engine (seems amazing :) )
Off Topic -<p>Does either of Filebeat or Logstash support config hot reload, as mentioned in the Vector's doc?
<a href="https://docs.vector.dev/usage/administration/reloading" rel="nofollow">https://docs.vector.dev/usage/administration/reloading</a><p>Edit -
Found It - <a href="https://www.elastic.co/guide/en/logstash/current/reloading-config.html" rel="nofollow">https://www.elastic.co/guide/en/logstash/current/reloading-c...</a>
For clarification, does "mib/s" mean "Mbit/s" (since lowercase b usually stands for bits, and uppercase B usually for Bytes)?<p>If yes, how comes log processing runs at only so low throughput in general?<p>That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth are capped at (in your case) 76.7Mbit/s. What's the bottleneck?
This is so exciting! An enterprise-grade solution to log workflow. To those unfamiliar with the Rust ecosystem, this project (Vector) addresses the 'L' within the ELK stack, and probably more.
There already are a lot of projects in this space.<p>While better performance is always great, most are already plenty fast for the majority of use cases.<p>The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing.<p>PS: the Logstash numbers seem suspiciously low. I'd bet it's some JVM config issue. Logstash can come to a crawl if it does not have enough memory.
Congratulations to the Timber team! I've had the pleasure of doing a small amount of contract work for them, have nothing but praise for them all.
When people say "high performance" about these things, I wonder how they compare, for instance, with the Sandia tools.* One things that matters there is avoiding system noise (jitter) on the monitored systems with transport over RDMA.<p>* <a href="http://ovis.ca.sandia.gov/" rel="nofollow">http://ovis.ca.sandia.gov/</a>
A pity it does not support at least once or exactly once delivery for the Vector sink.<p>Also, the documentation seems to miss information about which sinks support TLS?<p>We're currently looking for a distributed-over-the-internet logging setup and are interested in evaluating Rsyslog/RELP alternatives.