Ceph: A Journey to 1 TiB/s

Ceph has an interesting history.<p>It was created at Dreamhost (DH), for their internal needs by the founders.<p>DH was doing effectively IaaS &amp; PaaS before those were industry coined words (VPS, managed OS&#x2F;database&#x2F;app-servers).<p>They spun Ceph off and Redhat bought it.<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;DreamHost" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;DreamHost</a>

Nice article! We&#x27;ve also recently reached the mark of 1TB&#x2F;s at CERN, but with EOS (<a href="https:&#x2F;&#x2F;cern.ch&#x2F;eos" rel="nofollow">https:&#x2F;&#x2F;cern.ch&#x2F;eos</a>), not ceph: <a href="https:&#x2F;&#x2F;www.home.cern&#x2F;news&#x2F;news&#x2F;computing&#x2F;exabyte-disk-storage-cern" rel="nofollow">https:&#x2F;&#x2F;www.home.cern&#x2F;news&#x2F;news&#x2F;computing&#x2F;exabyte-disk-stora...</a><p>Our EOS clusters have a lot more nodes, however, and use mostly HDDs. CERN also uses ceph extensively.

I used to love doing experiments like this. I was afforded that luxury as a tech lead back when I was at Cisco setting up Kubernetes on bare metal and getting to play with setting up GlusterFS and Ceph just to learn and see which was better. This was back in 2017&#x2F;2018 if I recall. Good ole days. Loved this writeup!

I wish someone would try to scale the nodes down. The system described here is ~300W&#x2F;node for 10 disks&#x2F;node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all.<p>I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V CPU. Perhaps an eMMC chip or SD slot for boot.<p>This could scale down to just a few nodes, and it reduces the exposure to a single failure taking out 10 disks at a time.<p>I bet a lot of copies of this system could fit in a 4U enclosure. Optionally the same enclosure could contain two entirely independent switches to aggregate the internal nodes.

There was a point in history when the total amount of digital data stored worldwide reached 1TiB for the first time. It is extremely likely this day was within the last sixty years.<p>And here we are moving that amount of data every second on the servers of a fairly random entity. We not talking of a nation state or a supranatural research effort.

This is a fascinating read. We run a Ceph storage cluster for persisting Docker layer cache [0]. We went from using EBS to Ceph and saw a massive difference in throughput. Went from a write throughput of 146 MB&#x2F;s and 3,000 IOPS to 900 MB&#x2F;s and 30,000 IOPS.<p>The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something.<p>It’s been a massive improvement for our caching system.<p>[0] <a href="https:&#x2F;&#x2F;depot.dev&#x2F;blog&#x2F;cache-v2-faster-builds">https:&#x2F;&#x2F;depot.dev&#x2F;blog&#x2F;cache-v2-faster-builds</a>

The worst problems I&#x27;ve had with in-cluster dynamic storage were never strictly IO related, and were more the storage controller software in kubernetes having problems with real-world problems like pods dying and the PVCs not attaching until after very long timeouts expired, with the pod sitting in ContainerCreating until the PVC lock was freed.<p>This has happened in multiple clusters, using rook&#x2F;ceph as well as Longhorn.

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

I wanted to see how 1 TiB&#x2F;s compares to the actual theoretical limits of the hardware. So here is what I found:<p>The cluster has 68 nodes, each a Dell PowerEdge R6615 (<a href="https:&#x2F;&#x2F;www.delltechnologies.com&#x2F;asset&#x2F;en-us&#x2F;products&#x2F;servers&#x2F;technical-support&#x2F;poweredge-r6615-technical-guide.pdf" rel="nofollow">https:&#x2F;&#x2F;www.delltechnologies.com&#x2F;asset&#x2F;en-us&#x2F;products&#x2F;server...</a>). The R6615 configuration they run is the one with 10 U.2 drive bays. The U.2 link carries data over 4 PCIe gen4 lanes. Each PCIe lane is capable of 16 Gbit&#x2F;s. The lanes have negligible ~3% overhead thanks to 128b-132b encoding.<p>This means each U.2 link has a maximum link bandwith of 16 * 4 = 64 Gbit&#x2F;s or 8 Gbyte&#x2F;s. However the U.2 NVMe drives they use are Dell 15.36TB Enterprise NVMe Read Intensive AG, which appear to be capable of 7 Gbyte&#x2F;s read throughput (<a href="https:&#x2F;&#x2F;www.serversupply.com&#x2F;SSD%20W-TRAY&#x2F;NVMe&#x2F;15.36TB&#x2F;DELL&#x2F;182NW_356114.htm" rel="nofollow">https:&#x2F;&#x2F;www.serversupply.com&#x2F;SSD%20W-TRAY&#x2F;NVMe&#x2F;15.36TB&#x2F;DELL&#x2F;...</a>). So they are not bottlenecked by the U.2 link (8 Gbyte&#x2F;s).<p>Each node has 10 U.2 drive, so each node can do local read I&#x2F;O at a maximum of 10 * 7 = 70 Gbyte&#x2F;s.<p>However each node has a network bandwith of only 200 Gbit&#x2F;s (2 x 100GbE Mellanox ConnectX-6) which is only 25 Gbyte&#x2F;s. This implies that remote reads are under-utilizing the drives (capable of 70 Gbyte&#x2F;s). The network is the bottleneck.<p>Assuming no additional network bottlenecks (they don&#x27;t describe the network architecture), this implies the 68 nodes can provide 68 * 25 = 1700 Gbyte&#x2F;s of network reads. The author benchmarked 1 TiB&#x2F;s actually exactly 1025 GiB&#x2F;s = 1101 Gbyte&#x2F;s which is 65% of the maximum theoretical 1700 Gbyte&#x2F;s. That&#x27;s pretty decent, but in theory it&#x27;s still possible to be doing a bit better assuming all nodes can concurrently truly saturate their 200 Gbit&#x2F;s network link.<p>Reading this whole blog post, I got the impression ceph&#x27;s complexity hits the CPU pretty hard. Not compiling a module with -O2 (&quot;Fix Three&quot;: linked by the author: <a href="https:&#x2F;&#x2F;bugs.launchpad.net&#x2F;ubuntu&#x2F;+source&#x2F;ceph&#x2F;+bug&#x2F;1894453" rel="nofollow">https:&#x2F;&#x2F;bugs.launchpad.net&#x2F;ubuntu&#x2F;+source&#x2F;ceph&#x2F;+bug&#x2F;1894453</a>) can reduce performance &quot;up to 5x slower with some workloads&quot; (<a href="https:&#x2F;&#x2F;bugs.gentoo.org&#x2F;733316" rel="nofollow">https:&#x2F;&#x2F;bugs.gentoo.org&#x2F;733316</a>) is pretty unexpected, for a pure I&#x2F;O workload. Also what&#x27;s up with OSD&#x27;s threads causing excessive CPU waste grabbing the IOMMU spinlock? I agree with the conclusion that the OSD threading model is suboptimal. A relatively simple synthetic 100% read benchmark should not expose a threading contention if that part of ceph&#x27;s software architecture was well designed (which is fixable, so I hope the ceph devs prioritize this.)

What surprises me is, why they went with the harder to cool 1U nodes and 10 SSDs&#x2F;2x100Gb NICs instead of 2U nodes with 24 SSDs&#x2F;2x200 or even 400Gb NICs. They could remove the network bottleneck and save on power thanks to larger, lower speed fans and less CPU packages, possibly with more cores per socket though. Also, having a smaller number of nodes increases the blast radius but with even 34 nodes this is probably not such a problem. However, with less nodes they could have a flatter network with 4 switches or so too.

Cool benchmark, and interesting, however it would have read a lot better if abbreviations are explained at first usage. Not everybody is familiar with all terminology used in the post. Nonetheless congrats with results.

Is modern Ceph appropriate for transactional database storage, how is the IO latency? I&#x27;d like to move to a cheaper cfs that can compete with systems like Oracle&#x27;s clustered file system or DBs backed by something like Veritas. Veritas supports multi-petabyte DBs and I haven&#x27;t seen much outside of it or ocfs that similarly scales with acceptable latency

I wrote an intro to Ceph[0] for those who are new to Ceph.<p>It featured in a Jeff Geerling video briefly recently :-)<p>[0]: Understanding Ceph: open-source scalable storage <a href="https:&#x2F;&#x2F;louwrentius.com&#x2F;understanding-ceph-open-source-scalable-storage.html" rel="nofollow">https:&#x2F;&#x2F;louwrentius.com&#x2F;understanding-ceph-open-source-scala...</a>

I&#x27;m playing a lot with MicroCeph. Its aopinionated low TOS, friendly setup of Ceph. Looking forward additional comments. Planning to use it in production and replace lots of NAS servers.

Remember, random IOPs without latency is a meaningless figure.

Does someone knows how Ceph compares to other object storage engine like MinIO&#x2F;Garage&#x2F;...?<p>I would love to see some benchmarks there.

Ceph is interesting... open source software whose only purpose is to implement a distributed file system...<p>Functionally, Linux implements a file system (well, several!) as well (in addition to many other OS features) -- but (usually!) only on top of local hardware.<p>There seems to be some missing software here -- if we examine these two paradigms side-by-side.<p>For example, what if I want a Linux (or more broadly, a general OS) -- but one that doesn&#x27;t manage a local file system or local storage at all?<p>One that operates solely using the network, solely using a distributed file system that Ceph, or software like Ceph, would provide?<p>Conversely, what if I don&#x27;t want to run a full OS on a network machine, a network node that manages its own local storage?<p>The only thing I can think of to solve those types of problems -- is:<p><i>What if the Linux filesystem was written such that it was a completely separate piece of software, and a distributed file system like Ceph, and not dependent on the other kernel source code</i> (although, still complilable into the kernel as most linux components normally are)...<p>A lot of work? Probably!<p>But there seems to be some software need for something between a solely distributed file system as Ceph is, and a completely monolithic &quot;everything baked in&quot; (but not distributed!) OS&#x2F;kernel as Linux is...<p>Note that I am just thinking aloud here -- I probably am wrong and&#x2F;or misinformed on one or more fronts!<p>So, kindly take this random &quot;thinking aloud&quot; post -- with the proverbial &quot;grain of salt!&quot; :-)

My old company ran public and private cloud with Openstack and Ceph. We had 20 Supermicro (24 disks per server) storage nodes and total capacity was 3PB. We learnt some experiences, especially a flapping disk made whole system performance degraded. Solution was removing bad sector disk as soon as possible.

Where can I read about the rationale for ceph as a project? I&#x27;m not familiar with it.

I&#x27;m curious what the performance difference would be on a modern kernel.

What router&#x2F;switch one would use for such speed?

Sure would be nice if you defined some acronyms.

This is an insanely expensive cluster built to show a benchmark. 68 node cluster serving only 15TB storage in total.