All of the keys in JSON must be strings, so they should not need tags for themselves. Instead why not put the tag of the value assigned to the key in the key:<p><pre><code> {
"s:string":"Hello, world!",
"b64:binary":"SGVsbG8sIHdvcmxk",
"i:integer":42,
"f:float":42.0,
"t:timestamp":"2016-11-02T02:07:30Z"
}
</code></pre>
This prevents having to mess with the values in general and integers don't need to be encoded as strings.<p>EDIT:<p>I see this constraint:<p><pre><code> Member names in TJSON must be distinct. The use of the same member name more than once in the same object is an error.
</code></pre>
which is still satisfied, however you could have `i:foo` and `s:foo` which would result in redundant keys in the resulting JSON document. This constraint could be clarified that, untagged key names must be unique.<p>Another question, is a mimetype planned for this? `application/tjson`?
When have you ever written a program that doesn't know ahead of time what type of data it's going to be operating on? Especially if you're using a statically typed language.<p>Whether you validate incoming payloads in JSONSchema or not, you will always have some understanding of what the shape of the incoming JSON is supposed to be, down to the most concrete types. You'll probably receive many JSON payloads that all conform to the same schema. So why bother redundantly describing that schema in every individual payload?<p>If you want strict types, write a JSONSchema. If you need to know specific sub-type information, start specifying what should go into the "format" field in JSONSchema. They did it in Swagger: <a href="http://swagger.io/specification/" rel="nofollow">http://swagger.io/specification/</a><p>Since the article complains about JSON parsers not knowing how to handle certain situations, perhaps people should start writing JSON parsers that allow you to pass in a JSONSchema document at parse time so they're sure to handle each field type correctly.
It's amazing how many people are trying to reinvent protocol buffers! Every time I see something like this I think the developer didn't do their research or maybe they wanted to make a hobby project anyway. Stuff like this is dangerous to use in production. Even JSON as simple as it looks had a lot of bugs that are now.<p>If you want typed data structure transfer, use protocol buffer.
> Its primary intended use is in cryptographic authentication contexts, particularly ones where JSON is used as a human-friendly alternative representation of data in a system which otherwise works natively in a binary format.<p>The author might care to take a look at canonical S-expressions, a format from the 90s which attempted to do the same thing for many of the same reasons, and has the advantage of being rather more elegant.<p>E.g:<p><pre><code> {
"s:string":"s:Hello, world!",
"s:binary":"b64:SGVsbG8sIHdvcmxk",
"s:integer":"i:42",
"s:float":42.0,
"s:timestamp":"t:2016-11-02T02:07:30Z"
}
</code></pre>
could be:<p><pre><code> (string "Hello, world!"
binary [b]|SGVsbG8sIHdvcmxk|
integer [i]"42"
float [f]"42.0"
timestamp [t]"2016-11-02T02:07:30Z")
</code></pre>
Which is a perfectly valid encoding, but can use the canonical encoding (useful for cryptographic hashes):<p><pre><code> (6:string13:Hello, world!6:binary[1:b]13:Hello, world!7:integer[1:i]2:425:float[f]4:42.09:timestamp[1:t]20:2016-11-02T02:07:30Z)
</code></pre>
Which can be encoded for transport as:<p><pre><code> {KDY6c3RyaW5nMTM6SGVsbG8sIHdvcmxkITY6YmluYXJ5WzE6Yl0xMzpIZWxsbywgd29ybGQhNzpp
bnRlZ2VyWzE6aV0yOjQyNTpmbG9hdFtmXTQ6NDIuMDk6dGltZXN0YW1wWzE6dF0yMDoyMDE2LTEx
LTAyVDAyOjA3OjMwWik=}
</code></pre>
Granted, 'elegance' is in the eye of the beholder, but I like it.<p>I also think that there's a deeper concern with any shallow notion of types. An application doesn't care so much about 'some integer' as it does about 'a valid integer for this domain,' and <i>that</i> concern is what leads to schemas and profiles and things like that. Just encoding the machine type of a value is insufficient: one has to encode the <i>domain</i> type, which means conveying the domain, which means assuming some sort of shared knowledge.
I feel like there's a missed opportunity in not calling it TySON or something like that.<p>That aside, wouldn't it make more sense to fix the JSON parsers instead? They are the ones having issues parsing e.g. 64 bit integers, JSON has no problem holding them.
wow, this looks awful and painful.<p>There's no reason to tag the type of a field when you have a typed syntax. The real problems with JSON aren't at all addressed by this:<p>keys have to be strings
lack of 'attributes' like xml, which means you have to make a document convoluted from the start.<p>For example, lets say I am storing product data, I might do it like:<p>{'title': "Billy goes to Buffalo",
'page_count': 193,
'author': "Ray Broadbunky"}<p>But later I might want to be able to store attributes or metadata, in xml this doesn't change the schema of the document:<p><product>
<title>Billy goes to Buffalo</title>
<page_count>193</page_count>
<author>Ray Broadbunky</author>
</product><p>Can be extended to:<p><product>
<title human_verified="false">Billy goes to Buffalo</title>
<page_count human_verified="true">193</page_count>
<author human_verified="true">Ray Broadbunky</author>
</product><p>It's not beautiful but anything using this data will not have to change at all to add any metadata like this.<p>However, with JSON you have to either add new data that can somehow be joined to the data originally, or more commonly you have to be very defensive and 'plan for' this stuff, greatly complicating the schema.<p>You end up starting with:
{'attributes': [
{'name':'title','value':"Billy goes to Buffalo"},
{'name':'page_count', 'value':193,
...<p>so that you can add unanticipated things later without breaking consumers of the data<p>but at least some are addressed:
no standard way to store bytestrings
lack of time type
isn't there a way to extend the types to specify our own and register constructors for them? like transit?<p>otherwise we will be in the same place of json in terms of extension where our own types are second class citizens.
Those labels in the example are confusing.
Instead of string, binary, integer, float,timestamp please use something like name, password, age, height, sessiontime.<p>Using string and binary is worse than using foo and bar.
Reminds me of Tyre – Typed regular expressions: <a href="https://news.ycombinator.com/item?id=12292389" rel="nofollow">https://news.ycombinator.com/item?id=12292389</a>
"underspecification has lead to a proliferation of interoperability problems and ambiguities."<p>So TJSON has a perfect spec and everyone, now and forever, will interpret it perfectly?
What about <a href="https://amznlabs.github.io/ion-docs/" rel="nofollow">https://amznlabs.github.io/ion-docs/</a> ?
Why muddy up the actual values where you will have to parse that value with "t:" where t is type?<p>Why stuff it in one key/val? why not separated where it looks to see if type is present, if so it converts to it/validates against it (you can also place other validations/constraints on it like min/max values, length etc -- that will fall apart if you are trying to stuff it all in one key/value).<p>Like this:<p><pre><code> {
"val":"Hello, world!",
"type":"string",
"validation": "[regex]"
}
</code></pre>
Instead of:<p><pre><code> {
"s:string":"s:Hello, world!"
}
</code></pre>
This is typically how we type fields in JSON when needed as there is no parsing needed on the value. If you need to check type and it is present you can act on it.
json became so popular in first place because of its simplicity, i.e. no schemas, namespaces, attributes, less bizarre notation than xml. let's keep it this way.
This looks similar to msgpack with saltpack for crypto parts. Right?<p><a href="http://msgpack.org/" rel="nofollow">http://msgpack.org/</a><p><a href="https://saltpack.org/" rel="nofollow">https://saltpack.org/</a>
Six things:<p>1) "Lack of full precision 64-bit integers" is bullshit. Numeric precision is not specified by JSON. If a parser can't deal with 64-bit integer values, it's a poor parser.<p>2) "s: UTF-8 string" What does this mean? JSON strings are strings of Unicode code points; JSON itself may be encoded as UTF-8, -16, or -32. So does this mean "encode the string as UTF-8, then represent as Unicode code points"? That makes no sense.<p>Does this mean "encode the string as UTF-8 and output directly regardless of the encoding of the rest of the JSON output"? That makes no sense either.<p>So I'm guessing the author just conflated "UTF-8" with "Unicode", which is concerning given that he is attempting to define an interchange protocol.<p>3) "i: signed integer (base 10, 64-bit range)" What does this mean? (-2^64,2^64)? (-2^63,2^63)? [-2^63,2^63)?<p>4) "t: timestamp (Z-normalized)" What does that mean? There are literally dozens of timestamp formats. Does he mean full ISO 8601, restricted to UTC?<p>5) What is the point of TJSON anyway? When you deserialize, you <i>still</i> have to check that the data is of the type you expect. At best this saves a bit of parsing, since the deserializer can do that automatically. Various JSON schema languages already exist, which give you this richer typechecking.<p>The only use case I can think of for this is exactly what the author mentions further down the article: canonicalization for content-aware hashing. But this only works if the only types you care about fall into the small handful he thought of. What about, say, IP addresses? Case-insensitive strings (such as e-mail addresses)?<p>6) If we're talking about canonicalization, TJSON does not say how to canonicalize decimal numbers. I suppose this stems from the author's mistaken belief that numbers in JSON are IEEE floats (they're not, regardless of what common broken parsers do).<p>I hate to be so negative, but this really comes off as half-baked.<p>EDIT: Looking at the spec [1] it seems to address <i>some</i> of these, but still indicates a strong confusion between data <i>types</i> (Unicode, rational numeric) and data <i>representations</i> (UTF-8, IEEE double).<p>[1] <a href="https://github.com/tjson/tjson-spec/blob/master/draft-tjson-spec.md" rel="nofollow">https://github.com/tjson/tjson-spec/blob/master/draft-tjson-...</a>
Why don't float types use a tagged string? It says "tagging is mandatory" in the initial document, but floating point types are then omitted in the official spec
I've been writing a JSON parser when I have a few minutes here and there. I was surprised by the lack of specificity in defining numbers, specifically floats. If floats are know to lose precision after a few decimal places...<p>iex> 1.5555555555555555<p>1.5555555555555556<p>...why not just specify a max precision? You can always say "if you need a more precise number, just store it as a string". If I wanted a room for interpretation, I'd use YAML!
This argumentation is complete bullshit and even dangerous.<p>> "Parsing JSON is a Minefield": From a strictly software engineering perspective these ambiguities can lead to annoying bugs and reliability problems, but in a security context such as JOSE they can be fodder for attackers to exploit. It really feels like JSON could use a well-defined “strict mode”.<p>Not at all.
This article just outlined the differences of the various implementations regarding the 2 specs. And then added a spec test suite, including all the undefined problems, with suggestions how to go forward.<p>JSON is already strict enough. The problem are people like op to make it even not-stricter. The latest JSON spec RFC 7159 adds ambiguity by allowing all scalar values on the top level, which leads to practical exploitability.
See e.g. <a href="https://metacpan.org/pod/Cpanel::JSON::XS#OLD-VS.-NEW-JSON-RFC-4627-VS.-RFC-7159" rel="nofollow">https://metacpan.org/pod/Cpanel::JSON::XS#OLD-VS.-NEW-JSON-R...</a><p>"For example, imagine you have two banks communicating, and on one side, the JSON coder gets upgraded. Two messages, such as 10 and 1000 might then be confused to mean 101000, something that couldn't happen in the original JSON, because neither of these messages would be valid JSON.<p>If one side accepts these messages, then an upgrade in the coder on either side could result in this becoming exploitable."<p>What the op now suggests is adding the insecurity-mistake YAML took by adding tags to all keys. Here types don't add security, they weaken security!<p>It is security nightmare as it is leading to exploits which are e.g. already added to metasploit (CVE-2015-1592). tagged decoders are always a problem, and currently JSON and msgpack are the only serializers safe from such exploits due to its strictness.<p>I would suggest that the remaining JSON libraries first fix their problems by
conforming to the specs. First the secure old variant (RFC 4627) as default, and then maybe the relaxed new RFC 7159 variant, but denoting the security problems with interop of scalar values.<p>Currently only my Cpanel::JSON::XS library pass all these tests from the Minefield article. E.g. the ruby one, which the author complains about, not.
The type problem is esp. problematic in dynamic languages like ruby, where classes are not finalized by default.
> All base64url strings in TJSON MUST NOT include any padding with the '=' character.<p>This seems like it makes a streaming parser's job (slightly) more of a headache, without any serious advantage. Which seems particularly odd to me given that this seems heavily focused on binary stuff.
I'm a bit confused that TJSON only allows UTF-8 strings. The only way to escape Unicode characters in JSON is \uXXXX. But to encode astral characters with this syntax, UTF-16 surrogate pairs must be used. How does TJSON handle this, if strings must be encoded with UTF-8 only?
and still no ability to have comments. one reason I strongly prefer JSON5 <a href="http://json5.org/" rel="nofollow">http://json5.org/</a>