Grepping logs is still terrible

It&#x27;s beyond me how he doesn&#x27;t understand that text logs are a universal format, easily accessible, that can be instantly turned into whatever binary format you desire with a highly efficient insertion process (Splunk is just one of those that does a great job).<p>Here is the thing he doesn&#x27;t seem to understand - all of us who are sysadmins absolutely understand the value of placing complex and large log files into database so that we can query them efficiently. We also understand why having multi-terabyte text log files is not useful.<p>But what we find totally unacceptable is log files being shoved into binary repositories as the primary storage location. Because you know what <i>everyone</i> has their own idea of what that primary storage location should be, and they are mostly incompatible with each other.<p>The nice thing about text - for the last 40 years it&#x27;s been universally readable, and will be for the next 40 years. Many of these binary repositories will be unreadable within a short period, and will be immediately unreadable to those people who don&#x27;t know the magic tool to open them.

Cool, so which standard binary log storage format should we all switch to?<p>Should I submit patches to jawstats so that it&#x27;ll support google-log-format 1.0 beta, or the newer Amazon Cloud Storage 5 format? Or both? Or just go with the older Microsoft Log Storage Format? Or wait until Gruber releases Fireball Format? Has he decided yet whether to store dates as little-endian Unix 64 bit int timestamps, or is he still thinking about going with the Visual FoxPro date format, y&#x27;know, where the first 4 bytes are a 32-bit little-endian integer representation of the Julian date (so Oct. 15, 1582 = 2299161) and the last 4 bytes are the little-endian integer time of day represented as milliseconds since midnight? (True story, I had to figure that one out once. Without documentation.)<p>Should I write a new plugin for Sublime Text to handle the binary log formats? Or write something that will read the binary storage format and spit out text? Or is that too inefficient? Or should I give up on reading logs in a text form at all and write a GUI for it (maybe in Visual Basic)?<p>Do you know when I should expect suexec to start writing the same binary log format as Apache, or should I give up waiting on that and just write a daemon to read the suexec binary logs and translate them to the Apache binary logs?<p>Should I take the time to write a natural language parsing search engine for my custom binary log format? Do you think that&#x27;s worth the time investment? I would really like to be able to search for common misspellings when users ask about a missing email, you know, like &quot;&#x2F;[^\s]+@domain.com&#x2F;&quot; does now.<p>I look forward to your guidance. I&#x27;ve been eagerly awaiting the day that I can have an urgent situation on my hands and I can dig through server logs with all of the ease and convenience of the Windows system logs.

<i>Binary logs may be fine for you, but don&#x27;t force it on us!</i><p>This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don&#x27;t want will be imposed on them and their objections insultingly dismissed: a consequence of incredibly bad <i>social</i> &quot;change management&quot; by its proponents.<p>They are therefore deploying pre-emptive rhetorical covering fire against the day when greppable logs will be removed from the popular Linux distributions. Plain text is the lingua franca; binary formats bind you to their tools with a particular set of design choices, bugs and disadvantages. My adhoc log grepping workflow has a different set of bugs and disadvantages, but they&#x27;re <i>mine</i>.

Take this philosophy to an extreme and you end up with a dedicated data format and tooling&#x2F;APIs to access the data for every subsystem, not just logging. Essentially, this is Windows.<p>The downside to this is that now you don&#x27;t have a set of global tools which can easily operate across these separate datasets without writing code against an API. I hear PowerShell tackles this; I don&#x27;t know how well. The general principle though harms velocity at just getting something simple done, to the benefit of being able to do extremely complex things more easily. See Event Viewer for a good example of this.<p>Logs don&#x27;t exist in isolation. I want to use generally global tooling to access and manipulate everything. I don&#x27;t want to have to write (non-shell) code, recall a logging-specific API or to have to take the extra step of converting my logs back to the text domain in order to manipulate data from them against text files I have exported from elsewhere for a one-off job. An example might be if I have a bunch of mbox files and need to process them against log files that have message IDs in them. I could have an API to read the emails, and an API to read the logs, or I could just use textutils because I know an exact, validating regexp is not necessary and log format injection would have no consequence in this particular task.<p>I do see the benefits of having logs be better structured data, but I also see downsides of taking plain text logs away. Claiming that there are <i>no</i> downsides, and therefore no trade-off to be made, is futile. It&#x27;s like playing whack-a-mole, because nobody is capable of covering every single use case.

Honestly - I agree about the ELK stack side - piping all your logs into ES &#x2F; Logstash is a great idea. (Or Splunk &#x2F; Greylog &#x2F; Logentries)<p>If you run any sort of distributed system, this is vital. And while that counts as binary logs, I would argue that on the local boxes it should stay text.<p>I would agree, if you are running any sort of complex queries on your data - go to logstash, and do it there - it much nicer than regexes.<p>If on the other hand, you just want to see how a development environment is getting on, or to troubleshoot a known bad component tail&#x27;ing to | grep (or just tail&#x27;ing depending on the verbosity of your logs) is fine.<p>I don&#x27;t have to remember some weird incantation to see the local logs, worry about corruption etc.<p>One problem I will point out with the setup described is syslon-ng can be blocking. If the user is disconnected from the central logstash, and their local one dies, as soon as the FIFO queue in syslog-ng fills, good luck writing to &#x2F;dev&#x2F;log , which means things like &#x27;sudo&#x27; and &#x27;login&#x27; have .... issues.<p>Instead, if you have text files being written out, and something like beaver collecting them and sending them to logstash, you have the best of both worlds.

Windows has had binary logging forever. Is windows administration some wonderland of awesome capability for getting intelligence out of logs? Hell no.<p>For administering Unix like systems, the ability to use a variety of tools to process streams of text is an advantage and valuable capability.<p>That said, your needs do change when you&#x27;re talking about managing 10 vs 10,000 vs 100,000 hosts. I think what you&#x27;re really seeing here is a movement to &quot;industrialize&quot; the operations of these systems and push capabilities from paid management tools into the OS.

Grepping logs is terrible. Reverse engineering a binary format so you can diagnose why you are down&#x2F;crashing&#x2F;losing data is far worse. Logs should be handled as text until they reach their long term storage... then whatever helps analyze and query is fine...

Yeah,in the presence of adequate tooling you don&#x27;t need to grep logs. But how much more effort is required to use those tool-friendly loggings? Where is your god when the tool fails?<p>For me the main reason to access plaintext logs is they seldom fail, and they are simple. They are a bore to analyse, they CAN be analysed.<p>Anyway, this discussion only makes sense if the task at hand involves <i>heavy</i> log analysis, don&#x27;t complicate what is simple when it isn&#x27;t needed.<p>As for the razor analogy, you&#x27;re right, however I wouldn&#x27;t change my beard to be &quot;razor compatible only&quot;. In the software world I&#x27;d say it is still not uncommon to find yourself &quot;stranded in a desert island&quot;.

Oh jeez. Yes there are better and more performant tools for parsing optimised binary databases; nobody disputes that. And yes, tools like Splunk are more user friendly than grep; nobody disputes that either. But to advocate a binary only system for logs is short sighted because logs are the goto when everything else fails and thus need to be readable when every other tool dies. There&#x27;s quite a few scenarios that could cause this too:<p><pre><code> * log file corruption - text parsing would still work, * tooling gets deleted - there&#x27;s a million ways you can still render plain text even when you&#x27;ve lost half your POSIX&#x2F;GNU userland, * network connection problems, breaking push to a centralised database - local text copies would still be readable. </code></pre> In his previous blog post he commented that there&#x27;s no point running both a local text version and a binary version, but since the entirety of his rant is really about tooling rather than log file format, I&#x27;m yet to see a convincing argument against running the two paradigms in parallel.

This is a discussion for a sake of discussion. The way I see it is that author has a niche situation on his hands and therefore should use a product designed for that particular niche, instead of complaining how everyone&#x27;s wrong and trying to shove his perspective down peoples&#x27; throats.

Sounds like somebody in the systemd camp. I really dislike added complexity when it is totally unnecessary. If people want to transform their logs into a different storage format, that is up to them. Text files, however, are a fantastically simple way of storing... (drumroll please) text. Surprising &#x2F;s

&gt; For example: find all logs between 2013-12-24 and 2015-04-11, valid dates only.<p>That’s a straw man. If you’re grepping logs, you don’t need a regular expression that matches only valid dates because you can assume that the timestamps on the log records are valid dates. But I suppose<p><pre><code> 2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01])) </code></pre> doesn’t look so bad.<p>The whole thing is similarly exaggerated.

After reading the article I wonder if there are lots of tools that do all the binary advantages in indexes but leave the logs as text files, why that is not fine. To get the binary advantage the log does not have to be binary.<p>The example with the timestamps is also strange. No matter how you store the timestamps, parsing a humanly reasonable query like &quot;give me 10 hours starting from last Friday 2am&quot; to an actual filter is a complex problem. The problem is complex no matter how you store your timestamp. You can choose to do the complexity before and create complex index structures. You can choose to have complex algorithms to parse simple timestamps in binary or text form, you can build complex regexes. But something needs to be complex, because the problem space is. Just being binary doesn&#x27;t help you.<p>And that&#x27;s really the point here, isn&#x27;t it? Just being binary in itself is not an advantage. It doesn&#x27;t even mean by itself that it will save disk space. But text in itself is an advantage, always, because text can be read by humans without help (and in some instances without any training or IT education), binary not.<p>Yesterday I was thinking there might be something about binary logs. Now I&#x27;m convinced there isn&#x27;t. The only disadvantage seems to be that you also lose disk space if you store it in clear text. But disk space isn&#x27;t an issue in most situations (and in many situations where it is an issue you might have resources and tools at hand to handle that as well) It is added complexity for no real advantage. Thanks for clearing that up.

Grepping logs is terrible. Reverse engineering a binary format so you can diagnose why you are is worse. Logs should be handled as text until they reach their long term storage... then whatever helps analyze and query is fine...

My main problem with this is that ascii is not something that will ever change over time. The data format is wonderfully static. Forever. Introduce a binary format? You get versioning. It is a major downside.

What you lose when you move away from text logs is not any real benefit; what you lose is the illusion of control you have with text logs.<p>Text logs can be corrupted, text logs can be made unusable, you need a ton of domain-specific knowledge to even begin to make sense of text logs, etc.<p>But there&#x27;s always a sense that, if you had the time, you could still personally extract meaning from them. With binary logs, you couldn&#x27;t personally sit there and read them out line by line.<p>The issue is psychology, not pragmatism, and that&#x27;s why text logs have been so sticky for so long.

&gt; Does database store the data in text files? No? That&#x27;s my point.<p>This guy is a first class idiot who knows enough to reformulate a decided issue into yet another troll article. &quot;a database (which then goes and stores the data in a binary format)&quot;. How about a text file IS a database. It&#x27;s encoded 1s and 0s in a universal format instead of the binary DB format which can be corrupted with the slightest modification or hardware failure.

I think there are a number of issues that are getting mushed into one.<p>* Journal is just terrible.<p>* some text logs are perfectly fine.<p>* when you are in rescue mode, you want text logs<p>* some people use text logs as a way to compile metrics<p>I think the most annoying thing for me about journald is that it forces you to do something their way. However its optional, and in centos7 its turned off, or its beaten into such a way that I haven&#x27;t noticed its there.... (if that is the case, I&#x27;ve not really bothered to look, I poked about to see if logs still live in &#x2F;var&#x2F;log&#x2F; they did, and that was the end of it. Yes, I know that if this is the case, I&#x27;ve just undermined my case. Shhhhh.)<p>&#x2F;var&#x2F;log&#x2F;messages for kernel oopes, auth for login, and all the traditional systemy type things are good for text logs. Mainly because 99.9% of the time you get less than 10 lines a minute.<p>being able to sed, grep, tee and pipe text files are brilliant on a slow connection with limited time&#x2F;mental capacity. ie. a rescue situation. I&#x27;m sure there will be a multitude of stable tools that&#x27;ll popup to deal with a standardised binary log format, in about ten years.<p>The last point is the big kicker here. This is where, quite correctly its time to question the use of grep. Regex is terrible. Its a force&#x2F;problem amplfier. If you get it correct, well done. Wrong? you might not even know.<p>Unless you don&#x27;t have a choice, you need to make sure that your app kicks out metrics directly. Or as close to directly as possible. Failing that you need to use something like elastic search. However because you&#x27;re getting the metrics as an afterthought, you have to do much more work to make sure that they are correct. (although forcing metrics into an app is often non trivial)<p>If you&#x27;re starting from scratch, writing custom software, and think that log diving is a great way to collect metrics, you&#x27;ve failed.<p>if you are using off the shelf parts, its worth Spending the time and interrogating the API to gather stats directly. you never know, collectd might have already done the hard work for you.<p>The basic argument he puts forth is this: text logs are a terrible way to interchange and store metrics. And yes, he is correct.

Of course you need to log some data in textual format for emergencies, but if you had a tool that indexes events on timestamps, servers, monitorees, severity and event type, while severely reducing the storage required, you would be able to log much more data, and find problems faster. Arguing binary vs text logs is like arguing serial port vs USB on some industrial systems.

Great to see some effort in this area. I&#x27;ve been using New Relic and it&#x27;s pretty great for errors because we&#x27;ve setup Slack&#x2F;email notifications. However, there&#x27;s nothing for general log (e.g.: access log) parsing. I&#x27;m installing an ELK stack on my machine right now and hope that it&#x27;s enough

Doesn&#x27;t this just mean that we should have a more &quot;intelligent&quot; version of grep? For example, this &quot;supergrep&quot; could periodically index the files it is used on, so searching becomes faster.

*edit: I&#x27;m wrong, this was not the link posted yesterday. <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9496850" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9496850</a>

Isn&#x27;t everything will be solved by a some kind of grep that&#x27;s date&#x2F;timespan aware?

But then how will I watch the log files go by in real time?

It seems to me that most of the worry about a binary log file being &quot;opaque&quot; could be solved with a single utility:<p><pre><code> log-cat &lt;binary-log-file&gt; </code></pre> … that just outputs it in text. Then you can attack the problem with whatever text-based tools you want.<p>But to me, having a utility that I could do things like, get a range of log lines — in sorted order —, or, grep on just the message, would be amazing. These are all things that proponents of grep I&#x27;m sure will say &quot;you can!&quot; do with grep… but you can&#x27;t.<p>The dates example was a good one. I&#x27;d much rather:<p><pre><code> log-cat &lt;bin-log&gt; --from 2014-12-14 --to 2015-01-27 </code></pre> Also, my log files are not &quot;sorted&quot;. They are, but they&#x27;re sorted _per-process_, and I might have multiple instances of some daemon running (perhaps on this VM, perhaps across many VMs), and it&#x27;s really useful to see their logs merged together[2]. For this, you need to understand the notion of where a record starts and ends, because you need to re-order whole records. (And log records&#x27; messages are _going_ to contain newlines. I&#x27;m not logging a backtrace on one line.) grep doesn&#x27;t sort. |sort doesn&#x27;t know enough about a text log to adequately sort, but<p><pre><code> $ log-cat logs&#x2F;*.log --from 2014-12-14 --to 2015-01-27 &lt;sorted output!&gt; </code></pre> Binary files offer the opportunity for structured data. It&#x27;s really annoying to try to find all 5xx&#x27;s in a log, and your grep matches the process ID, the line number, the time of day…<p>I&#x27;ve seen some well-meaning attempts at trying to do JSON logs, s.t. each line is a JSON object[1]. (I&#x27;ve also seen it attempted were all that is available is a rudimentary format string, and the first &quot; breaks everything.)<p>Lastly, log files sometimes go into metrics (I don&#x27;t really think this is a good idea, personally, but we need better libraries here too…). Is your log format even parseable? I&#x27;ve yet to run across one that had an unambiguous grammar: a newline in the middle of a log message, with the right text on the second line, can easily get picked up as a date, and suddenly, it&#x27;s a new record. Every log file &quot;parser&quot; I&#x27;ve seen was a heuristic matcher, and I&#x27;ve seem most all of them make mistakes. With the simple &quot;log-cat&quot; above, you can instantly turn a binary log into a text one. The reverse — if possible — is likely to be a &quot;best-effort&quot; transformation.<p>[1]: the log writer is forbidden to output a newline inside the object. This doesn&#x27;t diminish what you can output in JSON, and allows newline to be the record separator.<p>[2]: I get requests from mobile developers tell me that the server isn&#x27;t acting correctly all the time. In order to debug the situation, I first need to _find_ their request in the log. I don&#x27;t know what process on what VM handled their request, but I often have a _very_ narrow time-range that it occurred in.

Logstash, Kibana, Splunk

Windows systems have had better log querying tools than grep for years now, with a well structured log file format to match. It&#x27;s good to see Linux distributions finally catching up in this regard.<p>Not that the log files on Linux are all entirely text-based anyway. The wtmpx and btmpx files are of a binary format, with specialised tools for querying. I don&#x27;t see anyone complaining about these and insisting that they be converted to a text-only format.