If you're wondering about the workload, this is the trivial Riemann config this benchmark uses:<p><pre><code> (streams
(rate 5 (comp prn float :metric))
</code></pre>
Which means for each five-second interval, sum all the metrics of the events flowing through this stream, divide by the time elapsed, and print that rate to the console.<p>I'm using this setup to put the heaviest load possible on Riemann's client and TCP server while I optimize those layers--it's not meant to stress the internal stream library, the state index, or pubsub. When I start optimizing those components, I'll have more "real-world" numbers to report.<p>I should also explain that this particular post explores the high-throughput, high-latency range of the client spectrum. End-to-end TCP latencies (not counting wire time) for single-event messages are on the order of ~100 microseconds-1ms, with occasional spikes to ~30ms depending on JVM GC behavior.
>> Throughput here is measured in messages, each containing 100 events, so master is processing 200,000–215,000 events/sec.<p>So in reality it is ~ 2k messages/sec. This is a rather poor throughput, as even off-the-shelf generic web servers (e.g. nginx) have the throughput an order of magnitude higher, and proprietary systems can reach 500k messages/sec over the network.
TLDR: Burned by framework magic. Talk about side effects.<p>Ten layers (and probably buffers) traveled through until your data hits the wire. Layer <i>x</i> decides to change its IO model and your throughput takes a dive. It's exactly why there was a post recently about building an operating system just to run some network daemon.
<p><pre><code> (defn execution-handler
"Creates a new netty execution handler."
[]
(ExecutionHandler.
(OrderedMemoryAwareThreadPoolExecutor.
16 ; Core pool size
1048576 ; 1MB per channel queued
10485760 ; 10MB total queued
)))
</code></pre>
It is a farce, isn't it?))
Offtopic, but it's refreshing to read blogs with clean designs and readable texts, without commercials or widgets inviting to click on things and all that crap.