Visualizing AWS Storage with Real-Time Latency Spectrograms

&gt; Every few seconds one of the writes takes forever [~5s]. You can notice the long periods of inactivity, and after that a green dot at the right of the chart: that’s our slow call. What is likely happening is: the local cache saturates and when that happens the application has to wait until the local data is pushed to the remote volume. Boy, you sure don’t want one of your critical code paths to hit one of these slow calls.<p>I&#x27;m surprised that there&#x27;s no asynchronous way that the FS cache will flush itself i.e. when it reaches 50% capacity, and rate-limit incoming requests if it&#x27;s too full. The idea that an FS cache is so dumb that it can&#x27;t do <i>anything</i> while it&#x27;s flushing its entire self is a bit scary - I&#x27;d expect that circular buffers and granular locking mechanisms could be used to great effect here. Is this kernel code? Userspace code? Is there research into this? Fundamental tradeoffs that I&#x27;m missing?

That&#x27;s clever and well executed. Wrong palette though :P<p>Red implies problems, green implies &quot;normality&quot;, but here this association is misplaced. Perhaps a typical &quot;fire&quot; palette would be better - from dark brown to red to orange to yellow and, ultimately, to white for the extremes.

Neat! This is definitely a step forward -- and thanks for the shout-out to our (that is, Sun&#x27;s and Joyent&#x27;s) prior work here. Tempted to also incorporate this into agghist and aggpack, the new DTrace actions I added for this kind of functionality.[1] Anyway, good stuff -- it&#x27;s always good to see new visualizations of system behavior!<p>[1] <a href="http://dtrace.org/blogs/bmc/2013/11/10/agghist-aggzoom-and-aggpack/" rel="nofollow">http:&#x2F;&#x2F;dtrace.org&#x2F;blogs&#x2F;bmc&#x2F;2013&#x2F;11&#x2F;10&#x2F;agghist-aggzoom-and-a...</a>

It would be interesting to run these tests on different instance sizes, specifically for data on the instance store. The larger the instance, the fewer neighbors you have to worry spending those precious IOPS.<p>As for SSD vs Magnetic EBS, I can&#x27;t say that I&#x27;m surprised. I&#x27;d assume that EBS implements some sort of cache in between you and your actual disk on the other side of the network so that the writes can return even faster. Try doing this again with reads and I&#x27;d bet you&#x27;d get some interesting results.<p>Edit: Also, did you pre-warm your EBS volumes? <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html" rel="nofollow">http:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AWSEC2&#x2F;latest&#x2F;UserGuide&#x2F;ebs-prewa...</a>

Nice job on the graphics for the post. Thanks for taking the time to animate and annotate well.

In the world of IOPS provisioned iops application demanding faster and faster iops this tool is handy for devops guy to find the truth of iops being used and how its performing, selecting if there is need to upgrade the storage ..

Calling this visualization a heatmap would be more appropriate than a spectrogram.

I really want to lop off the &#x27;ns&#x27; and &#x27;10 sec&#x27; divisions of all the charts and expand the resolution...

Well done!

cool project