Author here. We wanted to be able to graph p99, p99.9 metrics with arbitrary ranges, and found the existing solutions were not accurate enough for our needs. Happy to answer any questions.<p>Code here:<p><a href="https://github.com/DataDog/sketches-go" rel="nofollow">https://github.com/DataDog/sketches-go</a><p><a href="https://github.com/DataDog/sketches-py" rel="nofollow">https://github.com/DataDog/sketches-py</a><p><a href="https://github.com/DataDog/sketches-java" rel="nofollow">https://github.com/DataDog/sketches-java</a>
Hey, congratulations, this is a really cool algorithm! Thanks for sharing it.<p>I'm interested in this paper because I worked on a somewhat related problem some time ago, but got stuck on how to handle data that morphs into a mixed-modal distribution. Modes that are close together are no big deal, but modes that are spaced more exponentially apart are tricky to deal with. For an example of something that would be in DataDog's purview, it would be like trying to sketch the histogram of response times from an endpoint that sometimes took a "fast" path (e.g. a request for a query whose result was cached), sometimes took a "normal" path, and sometimes took a "slow" path. (e.g. a query with a flag that requested additional details to be computed) If the response times from the slow path is much bigger than the others, e.g. by an order of magnitude, their statistics might essentially drown-out the data from the other two paths since you're using them to calculate bin size.<p>I noticed you had some results from measuring DDSketch's performance on a mixed-modal distribution that looked pretty good (that "power" distribution on the last page). I was wondering if you had done any more investigation in this area? E.g. how messy/mixed can the data be before the sketch starts to break down?
How does this compare to t-digest?<p><a href="https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf" rel="nofollow">https://github.com/tdunning/t-digest/blob/master/docs/t-dige...</a>