De-identification of data sets (like cryptography) is a very difficult problem.<p>It is great that people are building tools for this. Even if I were skeptical of one or another in particular, the availability of tools popularizes the discussion of what is necessary and sufficient for de-identifying data.<p>The main use case I worked on was how to test an event driven (SOA at the time) pipeline without production data. Health information handling is very tightly regulated, so generating a test data set large enough that reflected the needs of the system was a significant challenge. Engineers couldn't just copy some production data and use it for testing. The regime I worked in that defined these rules (early PHIPA, PIPEDA in Ontario) is not unlike what people may encounter with GDPR.<p>When I was doing this sort of work, I found that it made more sense to find the structure of the data, then synthesize it from scratch. For a data format like HL7, this is non-trivial.<p>Synthesizing a few gigabytes of json/xml/text from a small training corpus provides incomplete test data. There are a few companies in the de-identification business, and I remember a few consulting services for it.<p>I can think of a few ways to do this, and they aren't simple.
How does this tool compare to other (libre) anonymization software programs, such as ARX [1]? From what I understand, there are only basic routines so to sample records and coarsen a few attributes (e.g. ZIP code, dates) implemented so far.<p>This might also not be sufficient to truly anonymize data, as a large body of research has shown so far [2,3,4]<p>[1] <a href="https://arx.deidentifier.org" rel="nofollow">https://arx.deidentifier.org</a><p>[2] <a href="https://www.uclalawreview.org/pdf/57-6-3.pdf" rel="nofollow">https://www.uclalawreview.org/pdf/57-6-3.pdf</a><p>[3] <a href="http://randomwalker.info/publications/no-silver-bullet-de-identification.pdf" rel="nofollow">http://randomwalker.info/publications/no-silver-bullet-de-id...</a><p>[4] <a href="http://arxiv.org/abs/1712.05627" rel="nofollow">http://arxiv.org/abs/1712.05627</a>
The intent behind this tool seems good, but I don't think it's a good idea. To actually anonymize data requires semantic understanding of that data and an understanding of what sort of data, harmless by itself, is transmuted into identifying data when provided in the context of other otherwise harmless data.<p>This tool doesn't help you with any of that. It seems to be a glorified awk script. My concern is that helping the user with the <i>easiest</i> part of anonymizing data stands to encourage the user to go full steam ahead without slowing down to stop and think very carefully about what they're doing.
> anonymising ... columns until the output is useful for applications where sensitive information cannot be exposed<p>This tool will not provide any significant amount of anonymity.<p>> rows to randomly sample ... hash (using ... 32 bits) the column ... mod the result by the [constant] value<p>This is not random. It deterministically selects the same very predictable fraction of rows.<p>> UK format postcode (eg. W1W 8BE) and just keeps the outcode (eg. W1W)<p>> Given a date, just keep the year<p>Partial postal codes and dates quantized to the year are still <i>very</i> revealing. Combined with other data (such as a hashed name), the partial postal code may allow a lot of people to be uniquely identified.<p>> Hash (SHA1) the input<p><i>Hashing does not provide anonymity. Substituting a candidate key with the hash of the key is usually a 1-to-1 map that is often trivial to reverse. It isn't hard to iterate through e.g. all possible names, postal codes, license plates, or other short-ish strings to find a matching SHA1.<p><a href="https://arstechnica.com/tech-policy/2014/06/poorly-anonymized-logs-reveal-nyc-cab-drivers-detailed-whereabouts/" rel="nofollow">https://arstechnica.com/tech-policy/2014/06/poorly-anonymize...</a><p>The salt </i>might* provide some resistance to per-computed tables, but a GeForce GTX 1080 Ti running hashcat can search for matching SHA1 at over 11 GH/s (giga-hashes per second). That means that a single 1080 Ti running for ~3-4 hours would not only discover <i>not only</i> that SHA1("hasselhof") == ffe3294fad149c2dd3579cb864a1aebb2201f38d; it would exhaustively search all 10 character or smaller lowercase strings.<p>> range<p>This is the only feature that could provide anonymity, if it is used correctly to group large numbers of individuals into the same bucket. This is probably more difficult that it first appears.
Slightly related: Metadata Anonymisation Toolkit <a href="https://mat.boum.org/" rel="nofollow">https://mat.boum.org/</a> (which seems to be in need of contributors)