Do we need to store all that telemetry?

I understand the point but I also advocate for the opposite, it&#x27;s not cool for the planet for sure but having all the data points for at least a couple of months is very useful on any large system and +15months for metrics so you can compare with the year before.<p>I can&#x27;t count the number of times users (or myself) discovered bug after many weeks because something gradually failed over time. Also it saves a lot of time to be able to pin point the exact day a behavior as changed so you can check the deploy of that day and quickly find the bug. Sometimes a trend is not obvious after a deploy but is clearly visible on the graph after a long period of time.<p>And for business intelligence, it&#x27;s always when you badly need a metric that you realize you never tracked it.

Another facet of this is <i>how</i> do we store telemetry data? Fully indexed instantaneously searchable seems to be the &quot;default&quot; these days but who actually needs that?<p>I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale.<p>[1] <a href="https:&#x2F;&#x2F;lists.w3.org&#x2F;Archives&#x2F;Public&#x2F;www-logging&#x2F;1996May&#x2F;0000.html" rel="nofollow">https:&#x2F;&#x2F;lists.w3.org&#x2F;Archives&#x2F;Public&#x2F;www-logging&#x2F;1996May&#x2F;000...</a>

Great post, the observability folks have gone off the rails in the last 5 years. I’ve seen it do more harm than good in terms of dev speed and ironically often make things less observable for the common path.

Isn’t the issue more that off-the-shelf solutions optimize for features and not cost? For instance, if I sell you an observability product, I want to show off all the cool realtime debugging features and such. And since there’s a cost to having all these features available (retention, indexing, sampling), we end up paying for features we don’t need. In a world of usage-based XaaS, there’s very little incentive to be cost-effective. Arguably even a perverse incentive to waste resources.<p>I bet you a full dollar that both in-house and open source solutions, on average, are way more stingy with resources. As they should be.

&gt; we can also add local storage of telemetry data in an efficient circular buffer. Typically, local storage is cheap and underutilized, allowing for “free” storage of a finite amount of historical data, that wraps automatically. Local storage provides the ability to “time travel” when a particular event is hit.<p>I think that this is a good idea when storage is concern for high-volume logs &#x2F; production. Persisting the buffer when high error rates &#x2F; unusual system behavior is observed would be a cool idea.

We&#x27;ve turned off logging &amp; tracing on a bunch of our high volume routes. Ideally I&#x27;d prefer we still sample them, at like 0.1% or what not, to give us some indicator, some chance of seeing anomalies. It just seems easier to gather &amp; use this information than it is to go develop a suite of metrics that can register all issues.<p>OpenTelemetry recently ish gained Open Agent Management Protocol (OpAMP), which allows some runtime control over things generating telemetry. The ability to stay fairly low but then scale up as needed sounds tempting, but gee it also sends shivers down my spine thinking of having such a elastic demands on one&#x27;s telemetry infrastructure, as engineers turn telemetry up as problems are occuring. <a href="https:&#x2F;&#x2F;opentelemetry.io&#x2F;docs&#x2F;specs&#x2F;opamp&#x2F;" rel="nofollow">https:&#x2F;&#x2F;opentelemetry.io&#x2F;docs&#x2F;specs&#x2F;opamp&#x2F;</a><p>The idea of having a local circular buffer sounds excellent to me. Being able to run local queries &amp; aggregate would be sweet. Are there any open otel issues discussing these ideas?

We continue to recreate features of single-computer OSes in distributed systems. This seems like the dtrace&#x2F;bpftrace of microservices world.

“A lot of telemetry doesn’t need to be stored for very long” is the attitude I take. Keeps costs down but gives good visibility.

I think most places don&#x27;t collect enough telemetry in the right formats.<p>It&#x27;s also possible they collect too much in the wrong formats.<p>But the ability to vet a hypothesis (I bet our users are confused about feature X, which we can test by looking at how many times they go to page X, then Y, then X again in 30 second window) in an hour versus 2 sprints is vastly underappreciated&#x2F;underutilized.<p>I feel like this article paints with too broad a brush.

Agree with the article enough that I did something about it which I call &quot;Poor Fred&#x27;s SIEM&quot;. The heart of it is a DNS proxy for Redis (<a href="https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;rkvdns">https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;rkvdns</a>). However it&#x27;s not targeted at environments where everything is in a &quot;bubble&quot; such that there are no ingress &#x2F; egress costs. (Lookin&#x27; at you, Cloud.) Furthermore &quot;control plane&quot; is an important concept, and it&#x27;s well understood in the industrial control world as the Purdue Model.<p>From a systems standpoint do you need to have all resources stored centrally in order to do centralized reporting? No, of course not. Admittedly it&#x27;s handy if bandwidth and storage are free. The alternative is distributed storage, with or without summarization at the edge (and aggregating from distributed storage for reporting).<p>Having it distributed does raise access issues: access needs to be controlled, and management of access needs to be managed. Philosophically the Cloud solutions sell centralized management, but federation is a perfectly viable option. The choice is largely dictated by organizational structure not technology.<p>There is also a difference between diagnostic and evaluative indicators. Trying to evaluate from diagnostics causes fatigue because humans aren&#x27;t built that way; evaluatives can and should be built from diagnostics. Diagnostics can&#x27;t be built from evaluatives.<p>The logging&#x2F;telemetry stack that I propose is:<p>1) Ephemeral logging at the limits of whatever observability you can build. E.g.: systemd journal with a small backing store, similar to a ring buffer.<p>2) Your compliance framework may require shipping some classes of events off of the local host, but I don&#x27;t think any of them require shipping it to the cloud.<p>3) Build evaluatives locally in Redis.<p>4) Use DNS to query those evaluatives from elsewhere for ad hoc as well as historical purposes. This could be a centralized location or it could be true federation where each site accesses all other site&#x27;s evaluatives.<p>I wouldn&#x27;t put Redis on the internet, but I don&#x27;t worry too much about DNS; and there are well-understood ways of securing DNS from tampering, unauthorized access, and even observation. By the way, DNS will handle hundreds or thousands of queries per second you just have to build for it.

&gt; For 30 years how telemetry is produced has not changed: we define all of the data points that we need ahead of time and ship them out of the origin process, typically at large expense. If we apply the control plane &#x2F; data plane split to observability telemetry production we can fundamentally change the status quo for the first time in three decades<p>Has Matt read any prior art in this field? <a href="https:&#x2F;&#x2F;research.google&#x2F;pubs&#x2F;monarch-googles-planet-scale-in-memory-time-series-database&#x2F;" rel="nofollow">https:&#x2F;&#x2F;research.google&#x2F;pubs&#x2F;monarch-googles-planet-scale-in...</a>

NO! You don&#x27;t!<p>I couldn&#x27;t agree with the author more. Keeping historical records of <i>business</i> metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense.<p>If an issue occurs, <i>then</i> turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I&#x27;m just speaking of operational issues.<p>Keeping all of your application logs and telemetry forever is expensive, and I can&#x27;t recall a single time when having more than a day&#x27;s with of history was ever useful in tracking down an operational issue.

In general I think many programmers have internalized the idea that it’s best to waste as many computing resources as we can possibly afford as long as it’s not the bottleneck. Then, in the future, if and when it becomes the bottleneck, we’ll have plenty of headroom to optimize and look like heroes for saving the millions of dollars we never had to spend in the first place. It’s really insane (at best) or genuinely a type of grift at worst.

&quot;Data is the new oil&quot; - if you don&#x27;t collect your customer data, and treat it as an asset, you are guilty of mismanagement . &#x2F;s

YES you do. BUT with varying retention periods for each a) environment b) region c) function d) criticality e) metric namespace&#x2F;name f) team etc.<p>Nobody needs to retain metrics like CPU, Memory for weeks but I may want to see their numbers during an incident, or not long after it is over.