And yet syslog works to the point where anything sold as an syslog replacement ends up adding complexity(along with features) rather then an simplification of the core problem.<p>It's in general a trend for old unix tools to work better in reality then in theory something thats rare for more modern tools.<p>Sure it's nice been able to use more modern query tools and have graphing libraries available but syslog grep and awk does get the job done and dont require a lot of resources to set up and maintain.
Being a maintainer of a secure syslog implementation around
2001, I was asked to join the IETF syslog group. There were a couple of syslog-ng guys, a Cisco guy, and I think a Microsoft guy. Plus a couple of randoms that didn't participate. The Cisco/Microsoft guys there looked like people whose sole job was to sit on standard comitees. They were not developers, at least not as a day job.<p>I was trying to reach a compromise in a simple syslog standard so it would be easier to authenticate and analyze. And trying to make it good enough for non-*nix systems. Nobody else cared about this.<p>It was one of the worst time wasters in my life. It was all politics. The syslog-ng guys were adamant with their proposal which was a very, very over-complicated idea based on another standard (BEEP). And I strongly suspect the Cisco/Microsoft guys were intentionally trying to make the group not work in subtle ways. After months, I just left.<p>They eventually published RFC 3195. And it's barely used, of course.<p>It seems Cisco's implementation still uses DIGEST-MD5 for authentication.
I feel the pain. Everybody decides their appliance will emit "syslog", but they don't bother to look at either RFC (not even the very lax RFC3164) and just emit "log messages preceded by some sort of timestamp", as the article calls out:<p>> But what makes things hell is the fact that too many vendors decided not to care about what is in the RFCs, they decided that “hey, putting a year there is just fine” even though the RFC says “no”, that they don’t really need to set a host in the header, and that they didn’t really need to implement anything new after their initial legacy stuff was created.<p>It sounds like the author and I are doing similar work, so he knows my pain: if you make a product which can parse syslog, and somebody selects your product for parsing syslog, and they they feed it non-syslog logs from Company Y's product... it's now your problem, instead of Company Y's, even though you're perfectly capable of parsing <i>syslog</i>! Luckily, regular expressions and beer eventually get most things sorted out. :)
And that's why journald is such a cool thing.<p>* Want to parse stuff? journalctl -o json<p>* Lots of stuff going on, need more precise timestamps? -o short-precise<p>* Want metadata, like the pid? It's in there.<p>* Want to know where to continue parsing? It supports cursors.<p>* Want to save disk space? It uncompresses logs transparently and can trim logs to whatever size you want.
When I started playing with Linux back around 1996, I started learning the real value of having logs. It took me into my first career in IT. The Unix/Linux value behind logging everything and quickly getting to the bottom of any problem, was something that was missing from my MS Windows experience. Yet the Event Viewer existed. No, it was rarely all that useful. Somehow lines in a text file... rotated at a predetermined interval... is just so simple yet, completely effective. Through the years I've tried the other tools that are supposed to supplant it yet, I still think it was the best. I'm getting used to the SystemD/JournalD way but, I really did like having a directory full of text files (and gzipped friends from previous days)
A bit off topic, but we've been using Vector (1) for log ingestion, and I really like it. It's fast, low resources, actively developed and flexible.<p>(1) <a href="https://vector.dev/" rel="nofollow">https://vector.dev/</a>
Don't get me started on Windows Event Log. Sure it might be better structured than syslog on paper, but it's even more of a mess. Not to mention that it's really hundreds of logs that just share a similar format and same UI.
> Parsing all of that mess is extremely “hacky”, with tons of regexes trying to account for all vendor quirks.<p>Whilst that's possibly true, at least you/we have the possibility to do so.
Started using Grafana Loki recently, so far it seems very good at consuming all the various log formats you might encounter and you can parse them into metrics etc as needed using regular expressions. Much nicer than dumping everything on a central syslog. My only "gripe" is the high learning curve compared to say elk stack.
Did you try Pcapng specification? I believe it is used by new syslog, and this is the packet format for logging both network interfaces, syslog files, and many more things together in a single format. <a href="https://wiki.wireshark.org/SampleCaptures" rel="nofollow">https://wiki.wireshark.org/SampleCaptures</a> <a href="https://wiki.wireshark.org/Development/PcapNg" rel="nofollow">https://wiki.wireshark.org/Development/PcapNg</a>
I've thought about this a bit and ideally one could either use a compressor with a predefined dict populated with normal daemon syslog output and output based on bad compression ratio of syslog messages or use minwise hashing and have a learning mode to populate a 'good' table and similar to the compression close jaccard similarities get discarded.
Secure collection and transfer of logs, decorated with metadata (timestamp(s), logging host, mac address, pid etc.) is solved many times. Just pick a tool.<p>The hard part is the message field's content and format. It basically boils down to actions of thousands of individual developers. They will never agree on a format and logging style.
Syslog "Hell"? Ever looked at SNMP?!<p>When a "standard" sticks around this long and needs to support so many legacy devices things can get a bit messy. At least syslog is human readable, while things may not be as machine parsable as you'd like, the info you need is usually only a few greps away.