Depending on which kernel version you are, C5 (and M5) instances can be real sources of pain.<p>The disk is exposed as a /dev/nvme* block device, and as such I/O goes through a separate driver. The earlier versions of the driver had a hard limit of 255 seconds before I/O operation times out. [0,1,2]<p>When the timeout triggers, it is treated as a <i>hard failure</i> and the filesystem gets remounted read-only. Meaning: if you have anything that writes intensively to an attached volume, C5/M5 instances are dangerous. We experimented with them for our early prometheus nodes. Not a good idea. Having the alerts for an entire fleet start flapping due to a seemingly nonsensical "out of disk, write error" monitoring node failure is not fun.[ß]<p>If you run stateless, in-memory only applications on them (preferably even without local logging), then you should be fine.<p>0: <a href="https://bugs.launchpad.net/ubuntu/bionic/+source/linux/+bug/1758466" rel="nofollow">https://bugs.launchpad.net/ubuntu/bionic/+source/linux/+bug/...</a><p>1: <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729119" rel="nofollow">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729119</a><p>2: <a href="https://www.reddit.com/r/aws/comments/7s5gui/c5_instances_nvme_storage_stucks/" rel="nofollow">https://www.reddit.com/r/aws/comments/7s5gui/c5_instances_nv...</a><p>ß: We handle nodes dying. The byzantine failure mode of nodes suddenly spewing wrong data is harder to deal with.
This is probably a big deal, if it's anything like the local SSD storage on GCE. The performance of local SSDs on Google Cloud is nearing absurd compared to anything else you can find right now.<p>That being said, I think one of the less compelling parts of this is that it'll probably vary per instance type quite a bit, being limited to C5 to start, so if you have a workload needing way better disk I/O than CPU performance, you might have to waste. That's one thing GCP really does have on AWS, better granularity.
I noticed the following comment in the article:<p>> Encryption – Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key. Each key is destroyed when the instance is stopped or terminated.<p>Does anyone know if the existing i3 EC2 instances NVMe drives are also encrypted in this fashion? I can't find any articles stating this...<p>Thanks!
Great news! I think Amazon should provide more specs on those NVME drivers. what kind of R/W latency/max bandwidth/low and high QD throughput I can expect?
This is really exciting to see! I had assumed that they would be launching most new instance types with only EBS storage so this is awesome that it looks like this will be coming to even more instance types too.<p>The bottom of the article mentions "PS – We will be adding local NVMe storage to other EC2 instance types in the months to come, so stay tuned!"