Logging practices I follow

One thing that&#x27;s an absolute must: Put an ISO-8601 timestamp at the very beginning of every line in your log. No apache format, no &quot;sun&quot; or &quot;jan&quot; or other words. ISO only. Seriously.<p>If the timestamp is in a weird format (or, god help you, multiple formats since some libraries log shit in a special way), it&#x27;ll be just about impossible to tell when things <i>actually happened</i> instead of just when the logging server <i>saw them</i>. In a perfect world these would be milliseconds apart, but lots of bad stuff can happen.<p>Your log-grepping-guy will thank you.

I think all of this sounds fine in theory, but the reality is that most logged information will not be needed - ever. Exactly what information is needed when can be difficult to predict. So, if a developer feels something might be important, they should probably log it. Within reason, I think it is better to have it and not need it than to need it and not have it.<p>It seems the author is putting a heavy emphasis on trying to create readable logs. Finding the signal in the noise. I am biased, but I think this is a failure of the tools used to read the logs rather than the logs themselves. This is why I wrote LogViewPlus (<a href="https:&#x2F;&#x2F;www.logviewplus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.logviewplus.com&#x2F;</a>).

One additional thing I like in structured logs is having some form of context level information be included eith your logs, so that you already know things like tenant id, user id, request id, and basic parameters of the request without having to rewrite all that everytime you get an exception.<p>Unrelated: I live in the Pacific north west and I clicked on this expecting to find a list like &quot;don&#x27;t log old growth for timber, don&#x27;t log the entire area&quot;, fun how your brain can associate a word with a concept and ignore the more context-relevant meaning.

Should add that log messages should answer questions like:<p>- What happened - When it happened? - Where it happened? - Why it happened? - What&#x27;s the next step?<p>If your log doesn&#x27;t answer at least the first 3 questions, then it&#x27;s useless. If you don&#x27;t answer &quot;why&quot;, then you should think harder whether that is useful or not.<p>If I had a cent for every time I see &quot;Something went wrong&quot; optionally followed by stack trace that is nearly entirely in 3rd party code with zero information to correlate it with anything - I would have retired to homestead ages ago.

Something I haven&#x27;t seen discussed very widely: it feels like there&#x27;s not only a balance needed in determine what to log vs what not to log, but also logging in a way that isn&#x27;t a detriment to the readability of the code overall.<p>Over time I&#x27;ve actually found myself logging less just because having to sorta mentally elide logging lines added to the cognitive overhead of reading &amp; understanding code.

I prefer this post which is more detailed: <a href="https:&#x2F;&#x2F;talktotheduck.dev&#x2F;logging-best-practices-mdc-ingestion-and-scale" rel="nofollow">https:&#x2F;&#x2F;talktotheduck.dev&#x2F;logging-best-practices-mdc-ingesti...</a>

One thing I&#x27;d add is the ability to tag certain data or certain loglines as containing personal information so that they can be scrubbed before transmitting or storing the logs. You don&#x27;t want things like credit card numbers, government id numbers, home addresses, and so on sitting out there in your logs, available to any developer reading a bug report (or available to everyone, when your company has a data breach). You&#x27;d log these things during development, skipping the scrubbing step, while the prod logs get scrubbed.

The key is that the person writing the log message is in no position to judge whether it&#x27;s right.<p>The measure of logs is whether you can put them in front of a smart but unfamiliar persons and have them figure out what&#x27;s happening. At a minimum, they should understand generally what&#x27;s happening and specifically what each message is saying (though perhaps not its significance).<p>(i.e., same as when writing code)

Question...why no mention of &quot;change&quot; logs? I&#x27;m curious as to why I don&#x27;t see change logs mentioned often as an important overlay to general system logging.<p>I liken errors and debugs all related to heart beat and breathing rate but without information like &quot;climbing stairs&quot; or &quot;changed medication&quot;, it may be hard to understand context or understand why new errors are being seen. The first question I would expect to ask when seeing logged issues, is what has changed recently that could be related to the new errors. Curious to hear thoughts on this?<p>I actually built software&#x2F;startup related to the logging of changes (architecture change&#x2F;software change&#x2F;server restarts) but just didn&#x27;t get traction and curious to know why it&#x27;s not more interesting to people.

Personally since few yeas I feel more and more issues with logs:<p>- most devs have lost the concept of logging levels, considering normal spitting out crappy giants backtraces and wall of meaningless text;<p>- most devs have lost the idea of &quot;being quiet&quot; or &quot;frugality&quot;, also have NEVER tried to read logs like an application user who do not have nor want to pass gazillion of lines of sources often crappily arranged.<p>In the classic *nix world skimming log for &quot;alerting patterns&quot; was easy, for modern crapplications it&#x27;s a bit of a nightmare. Similarly using logs for debug and mere health check is sometimes useless since many messages should at maximum be debug level logs, others are meaningless and even looking at them from the sources does NOT clarify until you read much more.

I fight log infra all the time. I can&#x27;t win the fight against structured logging anymore, so I&#x27;m now fighting against type systems and allergies to global state to make log output available everywhere. If you&#x27;re going to ram structured logging down devs&#x27; throats, then the least you can do is to make it easy enough to use. I don&#x27;t want to have to pass a logging object everywhere. There&#x27;s like two pieces of information you need in order to make a logger. Just write it to global state somewhere so I don&#x27;t have to worry about it and can call it anywhere.<p>I absolutely <i>loathe</i> reinventing global state by passing &quot;context&quot; objects and the like everywhere. It&#x27;s the dumbest thing in the world but no one ever questions it.

&gt; Whatever service you are using for logging, it costs money, and a fast way to burn money is to log the entire json object that was relatively small on your dev env, but blew up on production.<p>You could also, you know, run your own infrastructure and log to your heart&#x27;s content.

Log frugality and log uniqueness are great concepts. +1 for mentioning both the financial and cognitive costs of excessive logging.

On the levelled logging point, I stopped using levels after switching from Java -&gt; Go and haven&#x27;t looked back: <a href="https:&#x2F;&#x2F;thomshutt.github.io&#x2F;opinionated-logging-in-go.html" rel="nofollow">https:&#x2F;&#x2F;thomshutt.github.io&#x2F;opinionated-logging-in-go.html</a>

One question I always have about logging: how do I log valid and expected but prohibited actions? That is, the system is behaving as designed but the user is seeing an error message because they&#x27;re using the system wrong, and I want to know how often this is happening?

We use SQLite for logging all the things. This sidesteps entire rabbit colonies worth of issues - especially with regard to downstream parsing &amp; reporting.<p>I have found the extra structure and familiar semantics make it a lot easier to talk about what we log, how we log it and why.

Lots of logs can be replaced by metrics. People go crazy with logs.

I was expecting something about sustainable forestry

Unrelated to the content: I really like the phrasing of the title. Not &quot;…you should follow&quot;, not the tired &quot;best practices&quot;, simply &quot;Things I do&quot;.