Using Lua as a Serialization Format

Fun fact: Lua started its life as a language for representing data like this. At first it only had lists and hash tables but over time it gained support for things like if statements and loops, to be able to describe more interesting data. Eventually it morphed into the Turing complete programming language that we know today :)<p><a href="https:&#x2F;&#x2F;www.lua.org&#x2F;history.html" rel="nofollow">https:&#x2F;&#x2F;www.lua.org&#x2F;history.html</a><p>The current version of Lua still has some traces of that origin. For example, to keep RAM use manageable when parsing very large files the parser directly emits bytecode in a single pass, without building an intermediate abstract syntax tree.

This isn&#x27;t an argument for putting a Turing-complete language in your config file parser. It&#x27;s an argument for documenting your config file format so your users can write scripts (in a language of their choice) to generate config files programmatically.<p>It seems cute to do it this way, but now you have to start talking about removing certain functions from the standard library, running the script in a sandbox, etc.

About 20 years ago, I used Tcl as object serialization format in a game engine, and then made the next mistake to allow injecting manually written &#x27;scriptlets&#x27; into this serialization format (after all, serialized objects were just Tcl scripts).<p>These two things combined were probably the biggest software design mistake I ever made. We still have to drag a stripped down Tcl interpreter along in our asset pipeline 20 years later because it&#x27;s nearly impossible to switch away (because of the manually written &#x27;scriptlets&#x27; lurking everywhere in asset source files - of course it <i>would</i> be possible, but it was never on the right side of the cost-benefit equation).<p>Never mix code and data, folks :)

Author is surprised by convenience of describing data in Lua. I think he&#x27;s in for another surprise if he re-implements simulation on Lua side. The LuaJIT is so fast, I can just code the most naive implementation and never go back to refining it. It&#x27;s the simplest fast language and to me optimal choice for fully utilizing a single CPU core.

Lisp programmers realized in the 70s that having an artificial division between code and data was frequently problematic, hopefully other languages continue to catch on as well :)

I tried this on 1 or 2 small projects, years ago.<p>It was okay. It&#x27;s cool that you can import data directly into Lua scripts, since Lua usually has poor library support for any other data format.<p>But I hit a wall with a limit on the number of tables in a file or something, that was specific to the Lua impl I was using. (Probably LuaJIT)<p>Sometimes I miss Lua. I think with only a few big breaking changes it could be a much better language.

My game, Shadow Physics used Lua as a serialization format as a side effect of the game initially having no level editor or level import capabilities, only Lua scripts to create levels procedurally. Copy and paste was also implemented using Lua, a fun side effect of this you could copy and paste arbitrary Lua code as a live coding technique.

My hobby project, a space game (Space Nerds in Space) allows for Lua scripting, and it will allow exporting (most of) the state of the game to a lua script which, when run, restores that state. Ironically, the state of potentially running Lua scripts is the main thing stopping me from being able to checkpoint the entire state of the game.

OP if you&#x27;re the author, FYI first link to previous post (&#x27;this is a follow-up to&#x27;) is broken:<p><a href="https:&#x2F;&#x2F;mkhan45.github.io&#x2F;2020&#x2F;06&#x2F;12&#x2F;Lua-integration.html" rel="nofollow">https:&#x2F;&#x2F;mkhan45.github.io&#x2F;2020&#x2F;06&#x2F;12&#x2F;Lua-integration.html</a><p>Edit: Ah, &#x27;Lua&#x27; should be lower-case.

This is cool. Props to the author for thinking in original ways. Most people would have reached for a static solution like templating markup or a serialization format like YAML. Using a real programming language is way better.

A warning: Don&#x27;t think that this is a good idea. It&#x27;s small, it&#x27;s fast, it&#x27;s elegant. But it&#x27;s totally insecure.<p>Everybody who had access to this format can take it over. It can do far too much, that&#x27;s why everybody uses JSON. It can only represent hashes, arrays, and some limited primitives. No links, no logic, no lib. But lua is far too powerful. Game over.<p>Lisp would be even better, btw. Much easier to parse, and much easier to avoid eval. But his idea is eval.

I used Tcl for that in the past. Allows for some pretty powerful configuration.<p>For pure &quot;serialization&quot;, this is a bit overkill, of course.

I feel like the ron format is just too verbose, and JSON would be comparable to your Lua example. As long as you don&#x27;t serialize stuff like polymorphic classes (and you don&#x27;t have those in Rust anyway), you don&#x27;t need to put the name of every object into the serialization file.