Show HN: how I built the largest open database of Australian law

Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised there were no open databases of Australian law I could use to train an LLM on.<p>In this article, I run through the entire process of how I built my database, from months-long negotiations with governments to reverse engineering ancient web technologies to hacking together a multitude of different solutions for extracting text from documents.<p>My hope is that the next time someone like me is interested in training an LLM to solve legal problems, they won&#x27;t have to go down a year-long journey of trying to find the right data!<p>You can find my database on HuggingFace (<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;umarbutler&#x2F;open-australian-legal-corpus" rel="nofollow noreferrer">https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;umarbutler&#x2F;open-australian-l...</a>) and the code used to create it on GitHub (<a href="https:&#x2F;&#x2F;github.com&#x2F;umarbutler&#x2F;open-australian-legal-corpus-creator">https:&#x2F;&#x2F;github.com&#x2F;umarbutler&#x2F;open-australian-legal-corpus-c...</a>).

Australia has had free, searchable collections of Australian Law for 25+ years. Austlii is a prime example. There are Federal and State collections as well. The author is conscientious enough to read the scraping policy (or was blocked by anti-scraping tools) from feeding from one of these sites into his LLM.

This is cool and I&#x27;m a little surprised to see that Victoria is the one dragging the chain here. Is DataVic just talk, or does that not apply to law for some reason?

Great work and congratulations on your tenacity dealing with bureaucrats. Open access and machine readable formats should be widely available.

Good work reaching out to, and trying to get along with Australian government departments. As a fellow Australian, and one employed in government, I can very much say that many people in charge of operating these systems should not be.

I&#x27;m floored by what you accomplished in a year. Here in Canada, case law is under an iron grip by Canlii, LexisNexis and Thomson Reuters (Westlaw). As far as I&#x27;m aware, there is no truly open digital dataset that the public can use. This situation suits all the players involved and it isn&#x27;t changing - it&#x27;s refreshing to hear that one person can make a difference in a very similar setting (Australia)

Nice, but I have to ask how does it compare with <a href="https:&#x2F;&#x2F;austlii.edu.au&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;austlii.edu.au&#x2F;</a>, especially for completeness?<p>The Australasian Legal Information Institute is a great resource and yet seems strangely unknown (to the wider public at least.)<p>Trivia, the only reason I found out about it was when I did some work for an Aus govt agency and found out that they shared their web site with austlii! This was back in the early 2000s.

In my experience, one of the devil details is continuously keeping such a database updated. Without a set of common standards among the various governments, they can capriciously change URLs, formatting, and other details that may make it somebody&#x27;s fulltime job to keep it accurate and always up to date. Of course, not all use cases will require that, but many will.

What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well?<p>Is it overdue for innovation?

Would it we worth getting your corpus replicated into other venues as well, such at the Internet Archive or on GitHub itself?

I&#x27;m floored by what you accomplished in a year. Here in Canada, case law is under an iron grip by Canlii, LexisNexis and Thomson Reuters (Westlaw). As far as I&#x27;m aware, there is no truly open digital dataset that the public can use. This situation suits all the players involved and it isn&#x27;t changing - it&#x27;s refreshing to hear that one person can make a difference in a very similar environment (Australia)

It would be lovely I think if you used ML to help people ask questions so they can have more accessible law at their hands and understand what lots of things mean.<p>This can be applied to multiple countries around the world. The world of laws at your hands.<p>It’s an interesting concept

So good! Its crazy how legal information is such a spread out mess.<p>Whats worse is that git is such a perfect solution for legislation.

Could you explain how the majority of your corpus is under CC BY 4.0? I realise that&#x27;s the licence you have picked on HuggingFace, but if the source data was not already CC BY 4.0, how are you able to re-licence it as CC BY 4.0?

Incredible effort.<p>These types of projects have the potential to influence a nation.

I think in the law related subjects there is a huge potential for digitalisation. In Germany the law texts are online but the paragraphs not linked

Great work, thank you so much.<p>Fwiw and getting formatted text from html did you try<p>lynx —-dump url &gt;&gt; file.plaintext

That&#x27;s really neat. Such a shame VIC couldn&#x27;t be included though.

Insanely great. Amazing work.

Is there a U.S. equivalent?

Nice. Thanks for sharing.

Well done!