Facebook open-sources LogDevice, a distributed storage for sequential data

Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...

Happy to finally see LogDevice open. We have been working on this for years now.

Very interesting! I hadn&#x27;t heard of this before but I&#x27;d love to see it in action.<p>If anyone from the FB team or anyone using LogDevice wants to test performance with Optane SSDs (and compare to a NAND SSD), make a request by submitting an issue on our GitHub page: <a href="https:&#x2F;&#x2F;github.com&#x2F;AccelerateWithOptane&#x2F;lab&#x2F;issues" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;AccelerateWithOptane&#x2F;lab&#x2F;issues</a>. I&#x27;ll hook you up with a server hosted by Packet.

Martin Kleppmann seems to point out technologies for problems of similar patterns already exist - <a href="https:&#x2F;&#x2F;twitter.com&#x2F;martinkl&#x2F;status&#x2F;1039938408393662465" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;martinkl&#x2F;status&#x2F;1039938408393662465</a>

I had just stumbled across <a href="https:&#x2F;&#x2F;github.com&#x2F;facebookincubator&#x2F;python-nubia" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;facebookincubator&#x2F;python-nubia</a> and am anxious to try it out. Was wondering about the internal project it was factored out from. This appears to be it.

The use cases overlap neatly with Kafka&#x27;s. Everything from it&#x27;s usage of zookeeper, time-and-storage-based retention tuning are similar<p>The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn&#x27;t scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writes at scale but this is pure speculation.<p>I don&#x27;t know. If I understand why anyone might use this I&#x27;d contribute to building language bindings for the APIs.

Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage.<p>Kafka has done well so far, especially in making streaming systems more common, but it&#x27;s about time for the next-gen systems.

Awesome, I have been waiting for this since seeing the @scale talk about it. <a href="https:&#x2F;&#x2F;atscaleconference.com&#x2F;videos&#x2F;logdevice-a-file-structured-log-system&#x2F;" rel="nofollow">https:&#x2F;&#x2F;atscaleconference.com&#x2F;videos&#x2F;logdevice-a-file-struct...</a>

The amount of great quality open source projects dein Facebook just keeps growing. I really like the consistency guarantees:<p><a href="https:&#x2F;&#x2F;logdevice.io&#x2F;docs&#x2F;Concepts.html#consistency-guarantees" rel="nofollow">https:&#x2F;&#x2F;logdevice.io&#x2F;docs&#x2F;Concepts.html#consistency-guarante...</a><p>And it uses RocksDB under the hood:<p><a href="https:&#x2F;&#x2F;logdevice.io&#x2F;docs&#x2F;Concepts.html#logsdb-the-local-log-store" rel="nofollow">https:&#x2F;&#x2F;logdevice.io&#x2F;docs&#x2F;Concepts.html#logsdb-the-local-log...</a>

Thank to Open Source that, it looks a great project.<p>Could a LogDevice give a bit of informations about the scale they use that at facebook ?<p>- How many record this thing can injest per day ? - Any limitations on the maximum number of storage nodes ? - What would be your maximum and advise size of record for a production usage ? - ZooKeeper seems to be the center point used as epoch provider. Did you encounter any scaling limitations or max number of client due to that ?

Very interesting!<p>I like the idea of decoupling compute from storage for streaming&#x2F;log data.<p>I wonder if it would be easy to make it run under Consul, instead of ZooKeeper.

Am i the only being puzzled by<p><i>Scalable</i><p><i>Store up to a million logs on a single cluster.</i> ?<p>This sounds pretty confusing &#x2F; low volume.

What benefit to facebook is there from open sourcing technology they have developed?

Awesome to see this finally happening :)<p>Previous discussion in HN: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15142266" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15142266</a>

I don&#x27;t see anything about trust requirements or verification. Does LogDevice assume that all devices in my cluster are trusted?

&quot;bin&#x2F;logdeviced&quot;<p>All daemons and system administration utilities belong into sbin, because bin is for end-user applications.<p>Historically, the &quot;s&quot; in sbin meant something else, but it always contained applications and scripts only root could run.<p>When I see these examples, it&#x27;s depressing to see just how much understanding of UNIX is missing.

External logging service is my favorite way of doing replication. It provides nice features. Specifically:<p>- Cross vendor replication which makes migration much easier.<p>- No dependency on vendor provided replication protocols.<p>- Ability to use in-app databases such RocksDB, SQLite, ...<p>- Upgrading DB nodes becomes way easier since they are totally separated from each other.

How does that fit in a ML training pipeline? (this is mentioned on the page)

Sounds like it might have been influenced by the MSR CORFU project (separate sequencer, write striping). Can anyone confirm?

Is there any comparison with other similar storages?

This lot more similar to apache bookeeper.

this is like....the harder half of a whole blockchain project :D super interesting

Is this a Kafka competitor?