A coworker pointed me to this interesting post noting that in a browser-facing web server (not the same as hits to a datastore), most users will experience the 99th percentile for at least one resource on the page.<p><a href="http://latencytipoftheday.blogspot.it/2014/06/latencytipoftheday-most-page-loads.html" rel="nofollow">http://latencytipoftheday.blogspot.it/2014/06/latencytipofth...</a>
A nice follow-up and details digging into yesterday's "This is why I can't have conversations using Twitter" post:<p><a href="https://news.ycombinator.com/item?id=8526208" rel="nofollow">https://news.ycombinator.com/item?id=8526208</a>
Why is forking on xen slow? Most google hits for "xen forking slow" seem to point to some discussions about redis, but I guess other software would suffer from that too.
Side note: you can set you Y axis format to "ms" in Grafana to make the values more descriptive and add the "Points" chart option under display styles to make the mean values visible, which are obscured by the 90th percentile bar in your chart.
Also I assume label is wrong, it says 90ths percentile in the chart, but you speak of the 99th percentile.
I'm kind of glad Redis did the fork approach first. It's the reason I went with a userspace COW implementation in my work instead of forking and that paid huge dividends. It's the difference between starting COW in 10-20 milliseconds versus seconds and most of that time is distributed coordination not flipping the COW boolean.<p>When you crank up the density per node to 256 or 512 gigabytes even bare metal is problematic and in some domains like telecommunications they don't care that the spikes are concentrated because they cause cascading failures.<p>I think a userspace COW implementation in Redis would be a big project because you would need a different strategy for every data structure. Being single threaded also makes it challenging to get other/more cores to do the serialization and IO. It's very doable just not within the current philosophy of thou shalt have one thread per process.
fork test by redis labs (2012)
<a href="https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJNvtaiPEI" rel="nofollow">https://redislabs.com/blog/testing-fork-time-on-awsxen-infra...</a>
I know that redis really wants to be a persistent kvstore. I had a problem with a large website when I increased caching by 2 orders of magnitude (enough RAM to play with). When it came time to write a snapshot to disk, everything died for 5 minutes. Turned it off and haven't thought about it since. I'm not sure I'll ever shed my RDBMS predelections.
This is a huge problem on wall street, where trades must have predictable latencies. Stop-the-world garbage collectors are another source of latency "catastrophes".
Does fork not use copy-on-write? I would expect it to add overhead to all overlapped memory operations going forward, but am really surprised if it literally duplicated the entire memory contents.