Show HN: Vector – A High-Performance Log and Metric Router Written in Rust

I was dragging my feet to build a log shipper solution. I was going to use Filebeat -&gt; ElasticSearch -&gt; Kibana.<p>This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations.<p>Now, if I can make a suggestion for your next&#x2F;additional project..... A neat system metric collector in Rust that can export to Prometheus with same principles.<p>Low memory footprint,<p>Rust,<p>Single binary,<p>Customizable with a single config file without spending hours in manuals,<p>Stdin, Stderr -&gt; transform -&gt; Prometheus.<p>I’m learning Rust and eventually plan to build such a solution but I think a lot of this project can be repurposed for what I asked much faster than building a new one.<p>Cheers on this open source project. I will contribute whatever I can. Thanks!!

Seems similar to Veneur (like many other projects mentioned in comments here; didn&#x27;t realize this space was so crowded!) - down to the first two letters of the name: <a href="https:&#x2F;&#x2F;github.com&#x2F;stripe&#x2F;veneur" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;stripe&#x2F;veneur</a><p>Veneur is more metrics-focused, but might offer inspiration as you work on metrics support in Vector - in particular the SSF source, internal aggregation, and Datadog and SignalFX sinks.

We use a rather bespoke syslog -&gt; clickhouse log sink (<a href="https:&#x2F;&#x2F;github.com&#x2F;discordapp&#x2F;punt&#x2F;tree&#x2F;clickhouse" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;discordapp&#x2F;punt&#x2F;tree&#x2F;clickhouse</a>) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!

Just a heads up: There are several figures in your docs where the entirety of the useful information on the page is in the image and they don&#x27;t have alt tags or any accessible way to get the information (that I can find anyway). e.g. <a href="https:&#x2F;&#x2F;docs.vector.dev&#x2F;use-cases&#x2F;security-and-compliance" rel="nofollow">https:&#x2F;&#x2F;docs.vector.dev&#x2F;use-cases&#x2F;security-and-compliance</a>

I really appreciate the comparison table. Very rarely are things totally new and novel so it’s very nice to know what other things it’s _like_

Could this replace a simple fluentd setup right now or are there still major functionalities missing?<p>Specifically, I&#x27;m ingesting nginx logs in JSON format, cleaning up invalid UTF8 bytes (usually sent in the forwarded-for header to exploit security vulnerabilities), and sending to elasticsearch on an automated 90 day retention policy (daily indexes).<p>Seems like a fairly common use case for webservers.

Well logstash is now supporting good persistent queue. <a href="https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;logstash&#x2F;current&#x2F;persistent-queues.html#durability-persistent-queues" rel="nofollow">https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;logstash&#x2F;current&#x2F;persistent-...</a><p>I don&#x27;t know why author didn&#x27;t put correctness tick mark on it.

Freaking A... Rust performance is nuts when done right. Very cool.

How does this compare to telegraf?<p><a href="https:&#x2F;&#x2F;github.com&#x2F;influxdata&#x2F;telegraf" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;influxdata&#x2F;telegraf</a><p>Biggest thing that pops out to me is LUA engine (seems amazing :) )

Off Topic -<p>Does either of Filebeat or Logstash support config hot reload, as mentioned in the Vector&#x27;s doc? <a href="https:&#x2F;&#x2F;docs.vector.dev&#x2F;usage&#x2F;administration&#x2F;reloading" rel="nofollow">https:&#x2F;&#x2F;docs.vector.dev&#x2F;usage&#x2F;administration&#x2F;reloading</a><p>Edit - Found It - <a href="https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;logstash&#x2F;current&#x2F;reloading-config.html" rel="nofollow">https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;logstash&#x2F;current&#x2F;reloading-c...</a>

For clarification, does &quot;mib&#x2F;s&quot; mean &quot;Mbit&#x2F;s&quot; (since lowercase b usually stands for bits, and uppercase B usually for Bytes)?<p>If yes, how comes log processing runs at only so low throughput in general?<p>That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I&#x27;m genuinely curious why modern machines that have 40 Gbit&#x2F;s memory bandwidth are capped at (in your case) 76.7Mbit&#x2F;s. What&#x27;s the bottleneck?

Congrats to the Timber Team on this one. Ben and Zach were a couple of former colleagues on mine.

What is a high level example of why I&#x27;d use this?

Heka, but in Rust instead of Golang?

This is so exciting! An enterprise-grade solution to log workflow. To those unfamiliar with the Rust ecosystem, this project (Vector) addresses the &#x27;L&#x27; within the ELK stack, and probably more.

There already are a lot of projects in this space.<p>While better performance is always great, most are already plenty fast for the majority of use cases.<p>The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing.<p>PS: the Logstash numbers seem suspiciously low. I&#x27;d bet it&#x27;s some JVM config issue. Logstash can come to a crawl if it does not have enough memory.

Congratulations to the Timber team! I&#x27;ve had the pleasure of doing a small amount of contract work for them, have nothing but praise for them all.

When people say &quot;high performance&quot; about these things, I wonder how they compare, for instance, with the Sandia tools.* One things that matters there is avoiding system noise (jitter) on the monitored systems with transport over RDMA.<p>* <a href="http:&#x2F;&#x2F;ovis.ca.sandia.gov&#x2F;" rel="nofollow">http:&#x2F;&#x2F;ovis.ca.sandia.gov&#x2F;</a>

Love Timber.io, so it&#x27;s great to see some of your innovations coming back out as open source projects. Thanks!

Flume, but in Rust?<p><a href="https:&#x2F;&#x2F;flume.apache.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;flume.apache.org&#x2F;</a>

A pity it does not support at least once or exactly once delivery for the Vector sink.<p>Also, the documentation seems to miss information about which sinks support TLS?<p>We&#x27;re currently looking for a distributed-over-the-internet logging setup and are interested in evaluating Rsyslog&#x2F;RELP alternatives.

How do you handle back pressure?<p>Do you have specific module for it?

All this bombast and hubris is very unprofessional.

I wonder how popular posts on hackernews would be if people didn&#x27;t mention &quot;written in rust&quot; in the title.