Log for Machines

NO! No, no, no, a thousand times, no.<p>I spend all day every day troubleshooting broken things. Human readable logs are the very last thing that I have left that leaves me with any energy to keep troubleshooting things. If you take that away from me too, I swear I'm going to shoot somebody. (If I have any sense left when I go 'round the bend, I'll target a software engineer that thought it would be a good idea to fix what wasn't broke.)<p>Here's the thing: I can do anything with a human readable text log. I can scan it visually and look for anomalies; I can use my favorite text editor with it; I can grep it; I can cobble together some magic incantation on a command line to munge it to death until it cries out in mercy and gives me <i>exactly what I'm looking for</i>.<p>If I have to use somebody's special web-based woo-widget to view and search the contents of a log, if I'm restricted to the limits of their imagination when it comes to trying to get useful information out of the log, if I have to debug their dumbass invention while somebody's time-critical server is flopping around on the floor before I can even begin to figure out what in the heck is actually wrong with the server, I'm going to be stupid not to give up completely on support.<p>If you want to log information <i>which is useful to you</i>, you can do that without hardly any trouble at all without replacing perfectly good, reliable logging facilities which are the way they are for damned good reasons. Just write a Ruby whatchamacallit or a Python thingamajig or a PHP flibbedyfloo and log it to your MySQL database or your Postgres database or your no-sql nuh-uh-it's-not-a-database. I don't care.<p>But if I ever open up a server log and get blasted by a jillion lines of JSON that make it a thousand times harder for me to figure out how something broke I'm going to go on a shooting spree.

This falls under "good intention, but not well planned out" IMO. Single line logs are MUCH easier to process with text-oriented tools, which is why they've stuck around for so long.<p>This solves the encoding problem, but not the "how do I read/search it" problem, which is kicked down the line to other tools. And having a different format for every daemon is just a new problem.<p>I'd look at how others have dealt with the problem - for example, Logstash, which is a log parsing and movement engine:<p><a href="http://logstash.net/docs/1.0.17/learn" rel="nofollow">http://logstash.net/docs/1.0.17/learn</a><p>Also, timestamps aren't recommended for storing time data - they're not human readable, and convey less information than something like ISO8601 which covers more edge cases.

Not this again...<p><pre><code> * The first level consumer of a log message is a human. * The programer knows what information is needed to debug an issue. I believe these presumptions are no longer correct in server side software. </code></pre> The author's assumptions are wrong on both cases:<p>There may be any number of automated agents consuming the log, so a human isn't the largest consumer of log messages (in volume), but it is the first consumer of log messages as it is a human that ultimately will need them to troubleshoot issues.<p>Programmers are not the ones who "debug" issues in production environments.<p>Both of these are just symptoms of the same base misconception about how systems administrators go about their work. The same misconception that brought us XML configuration files.<p>When troubleshooting production issues I want to be able to grep for specific strings or regular expressions. I want to be able to tail the log. I do not want to have to parse structure (programatically or mentally), I do not want multi-line log entries. I do not want to be forced to use a tool just to transform the log from whatever format into something I can read.<p>Going even further: often production issues arise when only people less familiar with the system are available locally. In these cases I don't want to explain log structure over the phone.

There's actually a plan to replace syslog with something that's easier to create tools for.<p><a href="https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs&#38;pli=1" rel="nofollow">https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBA...</a>

I wonder if we need text logs at all anymore. It seems silly to expect people to be able to read millions of log entries from tens, hundreds or even thousands of servers. Maybe binary logs using protocol buffers or thrift are the way of the future.

Regardless of the syntax for encoding a server event, having a request id (txnId in the post) is an often overlooked detail. See how Google do this in their systems: <a href="http://highscalability.com/blog/2010/4/27/paper-dapper-googles-large-scale-distributed-systems-tracing.html" rel="nofollow">http://highscalability.com/blog/2010/4/27/paper-dapper-googl...</a>

I'd prefer to have two logs, one for human reading and one for machine consumption.

Before tea, standard sysadmin response: NO<p>After tea and some thought: No thanks

Apache Flume also xfers things in json.

s,JSON,BSON,g<p>These logs are for machines, and should be compact -- why use JSON at all?

Oh christ. I'm pretty sure my sysadmin friends would murder anyone that tried this on their boxen. As an engineer, while I appreciate the amount of shiny going on here, I doubt that adding yet another layer of tooling requirements to dealing with logs is the right answer--this could lead to serious ax-sharpening of picking the best JSON-to-readable scripts, and you can bet your ass that'll be different on every machine, install, site, etc. etc.<p>All that being said, I could see this being hilariously useful in game development. Having little JSON crumbs dribble out during playtesting and QA would be really handy, and that is an area where data mining is actually quite useful. Having a trivial way of sorting through crumbs by issue or tester, and then being able to, say, project the bugs into an HTML5/WebGL-based tool for devs to use would be really nifty.