The Emacs Problem

Uuugh... XML... how I hate you. Everyone should need to learn Lisp, even if you never ever program it, simply to avoid developing some of these poisonous XML-flavored tools. He mentions Ant specifically. Let me tell you, as an unwilling Microsoft Build expert, XML should be NOWHERE NEAR your build system. He puts down Makefiles, but as someone who wrote a Makefile for the first time after years of Ant and MSBuild, it was like a breath of fresh air. The intentional lack of power is what makes it so great. If anything, Makefiles are too powerful and too close to turing complete.<p>Here is what a make file should look like:<p>1) vpath statements for various file extensions<p>2) Phony targets for logical build units, including "all"<p>3) Generic rules for mapping <i>.abc to </i>.xyz; these rules should have <i>exactly</i> one line of code which executes an external script/tool<p>4) A list of edges in the dependency graph<p>5) A few variables as necessary to eliminate redundant edges<p>If you put any logic in your Makefiles, you are doing it wrong.<p>If your builds are slow, add empty dummy files for larger culling by timestamps. If timestamps are insufficient, codify early out logic into tools.<p>Not having logic in my Makefiles enables parallel execution and strong support for incremental builds. If I were to use Lisp as a build system, I'd create a DSL that had these same properties; forbidding arbitrary logic in my dependency graph. It's about finding the right balance to inject expressiveness without losing desirable properties of the DSL. This is why every developer needs to understand more about programming language design. Anytime you create any type of file format, you are doing it. And anytime you are writing any type of file format, you are reverse engineering it. Understanding the intended separation of logic for Makefiles helps me write better Makefiles.

One solution that Steve didn't discuss is JSON. To be fair, JSON wasn't that popular in 2005, but it's still a great solution.<p>The way it works is that their are no mandatory newline characters in JSON. Whitespace between lexical elements is ignored, and any embedded newlines in strings can be escaped (i.e. as \n). So a log format that a few people are using today is like this:<p>{'kind': 'foo', 'id': 1, 'msg': 'hi'} {'kind': 'bar', 'id': 2, 'msg': 'there'}<p>Each log message takes up a single line in the file. You can trivially deserialize any line to a real data structure in your language of choice. You can (mostly) grep the lines, and they're human readable. I do this at work, and frequently have scripts like this:<p>scribereader foo | grep 'some expression' | python -c 'code here'<p>In this case we're storing logs in the format described above (a single JSON message per line), and scribereader is something that groks how scribe stores log files and outputs to stdout. The grep expression doesn't <i>really</i> understand JSON, but it catches all of the lines that I actually want to examine, and the false positive rate is very low (&#60;0.1% typically). The final part of the pipe is some more complex python expression that actually introspects the data it's getting to do more filtering. You can of course substitute ruby, perl, etc. in place of the python expression.<p>I feel like this is a pretty good compromise between greppability, human readability, and the ability to programatically manipulate log data.

<i>You could even trivially convert it to XML and use XSLT, if you were silly enough. But Lisp is directly executable, so you could simply make the tag names functions that automatically transform themselves. It'd be a lot easier than using XSLT, and less than a tenth the size.</i><p>And now anyone who can modify your log can execute arbitrary code in the reader process…

Did anyone else think of CL-PPCRE as they read through the "lisp does not have regular expression support" implications? That was answered by the comparison of elisp to modern Common Lisp, and I wonder if anyone has done any work to make CL-PPCRE work for elisp. In spite of being someone's library, it is much faster than perl's built-in regex support.<p>There is a point to be made for the idea that you are solving the wrong problem with the log parsing. On the other hand, if you are trying to interface with other developers' popular engines, you may not have a choice.

Just to point out LINQ to XML <a href="http://msdn.microsoft.com/en-us/library/bb387098.aspx" rel="nofollow">http://msdn.microsoft.com/en-us/library/bb387098.aspx</a>.<p>It takes a lot of pain out of XML processing. Don't have to remember the specifics of XPath/XQuery but you still have to deal with the pain of multiple namespace resolution inherit in XML.

It would be interesting to see an emacs implementation in language better suited for text manipulation like Ruby or Perl.

TextMate/Ruby = Emacs/Elisp