I gotta hand it DataDog's sales team - it's the biggest combination of expensive and useless I've ever seen in my life, but they've somehow managed to convince the people with the checkbooks that they can't live without it.
Datadog is New Oracle and Observability is ripe for Open Source Disruption.<p>Great to see projects like SigNoz, NetData, Coroot appearing and Open Telemetry ecosystem getting more and more usable Opens Source backends and visualization tools not just agents.
I found that "observability platforms" like Datadog, New Relic etc can be really expensive and not that valuable unless you place a centralized "real-time observability pipeline" that's data agonistic between the originating telemetry data sources and the end destination like Datadog, New Relic, Looker, etc.<p>This approach allows you to see the data stream in real-time and then you can reduce, enrich, transform the data and then route it to any end destination like Datadog, New Relic, etc.<p>We did this and saw some pretty significant saving from the observability platform vendors we are using. Not only that the fact we were able to enrich the data made our troubleshooting much faster because we had data ingested now that we can make sense of and act on. We used datable.io who is a startup out of SF accomplish this. We talked with them about a month ago and their service was still in beta but they are allowing us to use it free of charge for now (not sure if they are still doing that or not?). I think the founder is an Ex New Relic guy?
“… and Kibana for metrics logging”<p>I love the ELK stack. Very much; but last I worked with it at a well-oiled company 7ish(?) years ago, we had one full time employee’s worth of time dedicated to keeping it stable, not to mention the infra costs.<p>Was this really a savings when you now you have to manage your own SLA and have an on-call rotation for it?
There's this cycle of posts that are _When other people do things for you, it's expensive_<p>Look at these outrageous AWS prices. Look at these outrageous Datadog prices. Look at these outrageous Snowflake prices...<p>And then of course, they lead to _Look how much cheaper my homegrown solution is_<p>It's a little tiring. You make trade-offs all the time. If the price isn't worth it, don't pay for it.
Why not self host grafana and Loki? It’s not hard and the cost for redundancy and backups running on tons of nodes with fast internet isn’t anything new. Much cheaper
DD was great at the beginning, but over time they moved to value-based pricing.<p>It's sad but understandable that all the good monitoring tools move up the food chain.
"The man who needs a new machine tool, and hasn't bought it, is already paying for it."
Agree though ballooning cost will prove to be Datadog's Achilles heel. But $83k/year is a bargain no? Where do you even find an engineer working less than $150k/year who could provide the same level of service as Datadog?
Its a weird glimpse into DHH’s psychology that he writes such a bombastic take on what is an extremely nuanced decision.<p>If it has any point, it must be to brag about how elite their ops team is to build and maintain obs infrastructure with less than 1/3 of a headcount. THAT would make for interesting reading.
Time to make the move to OTel with an Observability backend like <a href="https://www.kloudmate.com" rel="nofollow">https://www.kloudmate.com</a> (or any other APM of choice) and get most of the stuff done for a fraction. The big Dog has dictated terms for far too long.
Excuse my for rolling my eyes at DHH complaining about the price of software. It's enterprise software. You saw the price when you signed up. DD isn't cheap. Neither are the direct competitors (have you looked at Splunk licenses lately?).<p>Why is the cost so high? Because it's easy to add logging in places it isn't needed or useful. Every place I've ever worked has been able to save costs on these licenses by taking a pass at cleaning up their logs. If you pay by volume and you're logging things that aren't useful and never ever ever get read, you're burning cash. And if you think you're not doing this, I promise you probably are. DD (and others') costs can be reasonable if you don't just spray log data at it indiscriminately.<p>Want it for cheap? Cloudwatch is pennies compared to DD. It's not good, but it works. Want something faster or more featureful? You can stand up ElasticSearch or your favorite log aggregation software, and then you're probably paying a substantial percentage of what you paid for DD in server costs and time required to keep the thing plugging along. And then whatever time you spend dealing with "Grafana started timing out and we don't know why" and similar flavors of incidents.<p>Yes, good software costs about as much as a FTE. Because you'll (hopefully!) save about a FTE's worth of value by using the tool.