Monitoring demystified: A guide for logging, tracing, metrics

A lot of excellent information in that blog post and linked from it... but if you&#x27;re wondering where to start:<p>1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don&#x27;t worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number), and that the message will help you debug.<p>2. Instrument metrics using Prometheus, there are libraries that make this easy: <a href="https:&#x2F;&#x2F;prometheus.io&#x2F;docs&#x2F;instrumenting&#x2F;clientlibs&#x2F;" rel="nofollow">https:&#x2F;&#x2F;prometheus.io&#x2F;docs&#x2F;instrumenting&#x2F;clientlibs&#x2F;</a> . Counts get you started, but you probably want to think in aggregation and to ask about the rate of things and percentiles. Use histograms for this <a href="https:&#x2F;&#x2F;prometheus.io&#x2F;docs&#x2F;practices&#x2F;histograms&#x2F;" rel="nofollow">https:&#x2F;&#x2F;prometheus.io&#x2F;docs&#x2F;practices&#x2F;histograms&#x2F;</a> . Use labels to create a more complex picture, i.e. A histogram of HTTP request times with a label of HTTP method means you can see all reqs, just the POST, or maybe the HEAD, GET together, etc... and then create rates over time, percentiles, etc. Do think about cardinality of label values, HTTP methods is good, but request identifiers are bad in high traffic environments... labels should group not identify.<p>Start with those things, tracing follows good logging and metrics as it takes a little more effort to instrument an entire system whereas logging and metrics are valuable even when only small parts of a system are instrumented.<p>Once you&#x27;ve instrumented... Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) <a href="https:&#x2F;&#x2F;grafana.com&#x2F;products&#x2F;cloud&#x2F;" rel="nofollow">https:&#x2F;&#x2F;grafana.com&#x2F;products&#x2F;cloud&#x2F;</a> so you can see the results of your work immediately.<p>If it&#x27;s a big project, you have a lot of options and I assume you know them already, this is when you start looking at Cortex and Thanos, Datadog and Loki, tracing with Jaegar.

A few things I have learnt along the way:<p>Logs are great, but only once you&#x27;ve identified the problem. If you are searching through logs to _find_ a problem, its far too late.<p>Processing&#x2F;streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after&#x2F;writing&#x2F;decomming (example: dont use access logs to collect 4xx&#x2F;5xx and make a graph, collate and push the metrics directly)<p>Raw metrics are pretty useless. They need to be manipulated into buisness goals: service x is producing 3% 5xx errors vs % of visitors unable to perform action x<p>Alerts must be actionable.<p>Alerts rules must be based on sensible clear cut rules: service x&#x27;s response time is breeching its SLA not service x&#x27;s response time is double its average for this time in may.

The Art of Monitoring [1], covers most of these stuff in a unified manner.<p>You are introduced to some basics (push vs. pull monitoring), then proceeded with simple system metrics collection (cpu, memory) via collectd, then goes to logs ingestion and ends up extracting application-specific metrics from jvm and python applications.<p>I highly recommend it, even for seasoned professionals.<p>[1] <a href="https:&#x2F;&#x2F;artofmonitoring.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;artofmonitoring.com&#x2F;</a>

I never see an important system management principle brought up: If you get a user complaint (for some value of &quot;user&quot;) and not an alert, you should fix the monitoring system so that you don&#x27;t get another occurrence of it or related problems. Obviously that&#x27;s within reason, depending on the circumstances; the effort might not be worth it.

We log extensively. Here are some of my thoughts it<p>- at least in C++, the requirement to be able to log from pretty much anywhere can lead to messy code that either passes a reference to your logger to all classes that might possibly need it, or you&#x27;ve got an extern global somewhere. Yuck.<p>- logging can enable laziness. Being able to log that something weird happened can be considered a sufficient substitute for proper testing.<p>- logs are only as useful as the info they contain. This can mean state needs to be passed around all over the place just so that it can all be eventually logged on one line (it saves your data team from having to do a &#x27;join&#x27;)<p>- if your logger doesn&#x27;t support cycling log files it&#x27;s useless. If something goes wrong you can easily fill a disk.

It’s weird to see the stuff by Jay Kreps (of Kafka ~fame~) listed in the logs section. His writing is specifically _not_ about logs the observability tool, but logs the data structure such as you’d see at the heart of a database.

Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.

&gt; Logging is critical to detecting attacks and intrusions.<p>Yes, but not universally - and just collecting logs will not take you far. Logging everything and trying to approach security via the ’collect all data’ is both expensive and inaccurate, and one of the major inefficiencies in modern cyber.

Recently, I was searching for a service which offers those functionalities on a very basic level. I tried several options and was really disappointed with all of them. The only one that I found to be usable was <a href="https:&#x2F;&#x2F;logdna.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;logdna.com&#x2F;</a>. I&#x27;ve now been using it for a couple of weeks and it works OK. It offers logging, alerts, metrics&#x2F;dashboards, and some other things. And all that for a reasonable pricing.

Am I the only one that can’t reach the “save and exit” privacy button on mobile?<p>It’s hard for me to think that this is not intentional when the “Accept all” is usable but the alternative isn’t...

Nice

If you don&#x27;t need all the fancy metrics, and just want something simple to keep an eye on your services, alert you if they fail, and automatically restart them, check out my stealthcheck service. It&#x27;s all of 150 lines of free range, 0-dependency go:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;anderspitman&#x2F;stealthcheck" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;anderspitman&#x2F;stealthcheck</a>