Binance built a 100PB log service with Quickwit

A word of caution here: This is very impressive, but almost entirely wrong for your organisation.<p>Most log messages are useless 99.99% of the time. Best likely outcome is that its turned into a metric. The once in the blue moon outcome is that it tells you what went wrong when something crashed.<p>Before you get to shipping _petabytes_ of logs, you really need to start thinking in metrics. Yes, you should log errors, you should also make sure they are stored centrally and are searchable.<p>But logs shouldn&#x27;t be your primary source of data, metrics should be.<p>things like connection time, upstream service count, memory usage, transactions a second, failed transactions, upsteam&#x2F;downstream end point health should all be metrics emitted by your app(or hosting layer), directly. Don&#x27;t try and derive it from structured logs. Its fragile, slow and fucking expensive.<p>comparing, cutting and slicing metrics across processes or even services is simple, with logs its not.

There was a time at the beginning of the pandemic where my team was asked to build a full text search engine on top of a bunch of SharePoint sites in under 2 weeks and with frustratingly severe infrastructure constraints, (No cloud services, single box on prem for processing, among other things), and we did and it served its purpose for a few years. Absolutely no one should emulate what we built, but it was an interesting puzzle to work on and we were able to cut through a lot of bureaucracy quickly that had held us back for a few years wrt accessing the sensitive data they needed to search.<p>But I was always looking for other options for rebuilding the service within those constraints and found Quickwit when it was under active development. I really admire their work ethic and their engineering. Beautifully simple software that tends to Just Work™. It&#x27;s also one of the first projects that made me really understand people&#x27;s appreciation for Rust as well outside of just loving Cargo.

I wonder how much their setup costs. Naively, if one were to simply feed 100 PB into Google BigQuery without any further engineering efforts, it would cost about 3 million USD per month.

Reminds me of the time Coinbase paid DataDog $65M for storing logs[1]<p>[1] <a href="https:&#x2F;&#x2F;thenewstack.io&#x2F;datadogs-65m-bill-and-why-developers-should-care&#x2F;" rel="nofollow">https:&#x2F;&#x2F;thenewstack.io&#x2F;datadogs-65m-bill-and-why-developers-...</a>

Unfortunate the interesting part is missing.<p>Its not hard at all to scale to PB. Junk your data based on time, scale horizontally. When you can scale horizontally it doesn&#x27;t matter how much it is.<p>Elastic is not something i would use for scaling horizontally basic logs, i would use it for live data which i need live with little latency or if i do constantly a lot of log analysis live again.<p>Did Binance really needed elastic or did they just start pushing everything into elastic without every looking left and right?<p>Did they do any log processing and cleanup before?

What would you use for storing and querying long-term audit logs (e.g. 6 months retention), which should be searchable with subsecond latency and would serve 10k writes per second?<p>AFAICT this system feels like a decent choice. Alternatives?

I am having trouble understand how any organization could ever need a collection of logs larger than the size of the entire Internet Archive. 100PB is staggering, and the idea of filling that with logs, while entirely possible, just seems completely useless given the cost of managing that kind of data.<p>This is on a technical level quite impressive though, don&#x27;t get me wrong, I just don&#x27;t understand the use case.

&gt;Limited Retention: Binance was retaining most logs for only a few days. Their goal was to extend this to months, requiring the storage and management of 100 PB of logs, which was prohibitively expensive and complex with their Elasticsearch setup.<p>Just to give some perspective. The Internet Archive, as of January 2024, attests to have stored ~ 99 petabytes of data.<p>Can someone from Binance&#x2F;quickwit comment on their use case that needed log retention for months? I have rarely seen users try to access actionable _operations_ log data beyond 30 days.<p>I wonder how much $$ can they save more by leveraging tiered storage and engs being mindful of logging.

How? If Binance had a trillion transactions, that’s 100KB per transaction. What all are they logging?

&gt; On a given high-throughput Kafka topic, this figure goes up to 11 MB&#x2F;s per vCPU.<p>There&#x27;s got to be 2x to 10x improvement to be made there, no? No way CPU is the limitation these days and even bad hard drives will support 50+mB&#x2F;s write speeds.

The article talks about 1.6 PB &#x2F; day, which is 150 Gbps of log ingest traffic sustained. That&#x27;s insane.<p>A change of the logging protocol to a more efficient format would yield such a huge improvement that it would be much cheaper than this infrastructure engineering exercise.<p>I suspect that everyone just assumes that the numbers represent the underling data volume, and that this cannot be decreased. Nobody seems to have heard of <i>write amplification</i>.<p>Let&#x27;s say you want to collect a metric. If you do this with a JSON document format you&#x27;d likely end up ingesting records that are like the following made-up example:<p><pre><code> { &quot;timestamp&quot;: &quot;2024-07-12T14:30:00Z&quot;, &quot;serviceName&quot;: &quot;user-service-ba34sd4f14&quot;, &quot;dataCentre&quot;: &quot;eastFoobar-1&quot;, &quot;zone&quot;: 3, &quot;cluster: &quot;stamp-prd-4123&quot;, &quot;instanceId&quot;: &quot;instance-12345&quot;, &quot;object&quot;: &quot;system&#x2F;foo&#x2F;blargh&quot;, &quot;metric: &quot;errors&quot;, &quot;units&quot;: &quot;countPerMicroCentury&quot;, &quot;value&quot;: 391235.23921386 } </code></pre> It wouldn&#x27;t surprise me if in reality this was actually 10x larger. For example, <i>just</i> the &quot;resource id&quot; of something in Azure is about this size, and it&#x27;s just one field of many collected by <i>every</i> logging system in that cloud for <i>every</i> record. Similarly, I&#x27;ve cracked open the protocols and schema formats for competing systems and found 300x or worse write amplification being the typical case.<p>The actual data that needed to be collected was just:<p><pre><code> 391235.23921386 </code></pre> In a binary format that would be 4 bytes, 8 if you think that you need to draw your metric graphs with a vertical precision of a millionth of a pixel and horizontal precision of a minute because you can&#x27;t afford the exabytes of storage a higher collection frequency would require.<p>If you collect 4 bytes per metric in an array and record the start timestamp and the interval, you don&#x27;t even need a timestamp per entry, just one per thousand or whatever. For a metric collected <i>every second</i> that&#x27;s just 10 MB per month <i>before</i> compression. Most metrics change slowly or not at all and would compress down to mere kilobytes.

maybe drop the log level from debug to info...

Why do so many companies insist on shipping their logs via Kafka? I can&#x27;t imagine deliverability semantics are necessary with logs, and if they are, they shouldn&#x27;t be in your logs?

Does QuickWit support regex search now? The underlying store, Tantivy, already does.<p>This is what stopped a PoC cold at an earlier project.

Just browsing the Quickwit documentation it seems like the general architecture here is to write JSON logs but stores them compressed. Is this just something like gzip compression? 20% compressed size does seem to align to ballpark estimates of JSON GZIP compression. This is what Quickwit (and this page) calls a &quot;document&quot;: a single JSON record (just FYI).<p>Additionally you need to store indices because this is what you actually search. Indices have a storage cost when you write them too.<p>When I see a system like this my thoughts go to questions like:<p>- What happens when you alter an index configuration? Or add or remove an index?<p>- How quickly do indexes update when this happens?<p>- What about cold storage?<p>Data retention is another issue. Indexes have config for retention [1]. It&#x27;s not immediately clear to me how document retention works, possibly from S3 expiration?<p>So, network transfer from S3 is relatively expensive ($0.05&#x2F;GB standard pricing [2] to the Internet, less to AWS regions). This will be a big factor in cost. I&#x27;m really curious to know how much all of this actually costs per PB per month.<p>IME you almost never need to log and store this much data and there&#x27;s almost no reason to ever store this much. Most logs are useless and you also have to question what the purpose is of any given log. Even if you&#x27;re logging errors, you&#x27;re likely to get the exact same value out of 1% sampling of logs than you are with logging everything.<p>You might even get more value with 1% sampling because your query and monitoring might be a whole lot easier with substantially less data to deal with.<p>Likewise, metrics tend to work just as well from sampled data.<p>This post suggests 60 day log retention (100PB &#x2F; 1.6PB daily). I would probably divide this into:<p>1. Metrics storage. You can get this from logs but you&#x27;ll often find it useful to write it directly if you can. Getting it from logs can be error-prone (eg a log format changes, the sampling rate changes and so on);<p>2. Sampled data, generally for debugging. I would generally try to keep this at 10TB or less;<p>3. &quot;Offline&quot; data, which you would generally only query if you absolutely had to. This is particularly true on S3, for example, because the write costs are basically zero but the read costs are expensive.<p>Additionally, you&#x27;d want to think about data aggregation as a lot of your logs are only useful when combined in some way<p>[1]: <a href="https:&#x2F;&#x2F;quickwit.io&#x2F;docs&#x2F;overview&#x2F;concepts&#x2F;indexing" rel="nofollow">https:&#x2F;&#x2F;quickwit.io&#x2F;docs&#x2F;overview&#x2F;concepts&#x2F;indexing</a><p>[2]: <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;pricing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;pricing&#x2F;</a>

Lots of storage to log all of the wash trading on their platform.

So people don&#x27;t build this out themselves?<p>Regardless, there&#x27;s some computer somewhere serving this. How do they service 1.6 PB per day? Are we talking tape backup? Disks? I&#x27;ve seen these mechanical arms that can pick tapes from a stack on a shelf, is that what is used? (example: <a href="https:&#x2F;&#x2F;www.osc.edu&#x2F;sites&#x2F;default&#x2F;files&#x2F;press&#x2F;images&#x2F;tapelibrary.jpg" rel="nofollow">https:&#x2F;&#x2F;www.osc.edu&#x2F;sites&#x2F;default&#x2F;files&#x2F;press&#x2F;images&#x2F;tapelib...</a>)<p>For disks that&#x27;s like ~60&#x2F;day without redundancy, do they have people just constantly building out and onlining machines in some giant warehouse?<p>I assume there&#x27;s built in redundancy and someone&#x27;s job to go through and replace failed units?<p>This all sounds like it&#x27;s absurdly expensive.<p>And I&#x27;d have to assume they deal with at least 100x that scale because they have many other customers.<p>Like what is that? 6,000 disks a day? Really?<p>I hear these numbers of petabyte storage frequently. I think Facebook is around 5PB&#x2F;daily. I&#x27;ve never had to deal with anything that large. Back in the colo days I saw a bunch of places but nothing like that.<p>I&#x27;m imagining forklifts moving around pallets of shrink wrapped drives that get constantly delivered<p>Am I missing something here?<p>Places like AWS should run tours. It&#x27;d be like going to the mint.

It&#x27;s always very amusing how all of the blockchain companies wax lyrical about all of the huge supposed benefits of blockchains and how every industry and company is missing out by not adopting them and should definitely run a hyperledger private blockchain buzzword whatever.<p>And then, even when faced with implementing a huge, audit critical, distributed append-only store, the thing they tell us blockchains are so useful for, they just use normal database tech like the rest if us. With one centralized infrastructure where most of the transactions in the network actually take place. Who&#x27;s tech stack looks suspiciously like every other financial institution.<p>I&#x27;m so glad we&#x27;re ignoring 100 years of securities law to let all of this incredible innovation happen.