Interactive examples for learning jq

For me, bash and jq are, literally, the opposite of riding a bicycle. It doesn't matter the amount of time I spend on a given week working with them, a month later, I am gonna have to skim through my bookmarks and Kagi results (and now also chatGPT) for knowing how to do stuff I was easily doing a month ago.

I always struggled with understanding JQ. Each time I was just googling things. But actually it does make a lot of sense if you understanding the building blocks. I wrote it all down [1] but here is my summary:<p>jq lets you select elements like it&#x27;s a JavaScript object using dot notation and array indexing.<p><pre><code> jq &#x27;.key.subkey.subsubkey&#x27; jq &#x27;.key[].subkey[2]&#x27; </code></pre> You can turn wrap things in array constructors, or object constructors to create new objects and lists:<p><pre><code> jq &#x27;[ .[].key ]&#x27; jq &#x27;{key1: .key1, key2: .key2}&#x27; </code></pre> You can combine filters with pipes (|) to build complex transformations. Built-ins like map() and select() are useful for transforming arrays.<p>You put it all together into something like this:<p><pre><code> curl https:&#x2F;&#x2F;api.github.com&#x2F;repos&#x2F;stedolan&#x2F;jq&#x2F;issues | jq &#x27;map({title: .title, labels: .labels}) | map(select(.labels)) | map({issue: .title}) | sort_by(.issue) | [{issues: .[]}] </code></pre> This query fetches GitHub issues, transforms them into a simplified structure, filters out unlabeled issues, sorts them, and wraps the results in an array - demonstrating how you can chain together jq&#x27;s query language to wrangle JSON data.<p>[1]: <a href="https:&#x2F;&#x2F;earthly.dev&#x2F;blog&#x2F;jq-select&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;earthly.dev&#x2F;blog&#x2F;jq-select&#x2F;</a>

I was curious so looked up how it works before reading the summary at the end, and that led me to find another user of aioli.js jq implementation: <a href="https:&#x2F;&#x2F;jiehong.gitlab.io&#x2F;jq_offline&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;jiehong.gitlab.io&#x2F;jq_offline&#x2F;</a> (featured <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=28627172">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=28627172</a> two years ago); jqplay.org still sends all the data on every modification so they should learn from it...<p>Anyway, this article is neat! Good work!<p>If I were to nitpick one of the last examples with path has no explanation and flew over my head (would have to open the documentation), and a reset button for each example might be nice after messing with it a bit, but it was a nice play.

JQ is an insanely powerful language, just to put to rest any of your doubts about what it is capable of here is an implementation of JQ... in JQ itself:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;wader&#x2F;jqjq">https:&#x2F;&#x2F;github.com&#x2F;wader&#x2F;jqjq</a><p>It really is a super cool little, super expressive nearly (if not entirely) turing complete pure functional programming language.<p>You can:<p>* Define your own functions and libraries of functions<p>* Do light statistics<p>* Drastically reshape JSON data<p>* Create data indexes as part of you JQ scripts and summarize things<p>* Take JSON data, mangle it into TSV and pipe into SQLite<p><pre><code> cat data.json | jq &#x27;&lt;expr&gt;[]|@tsv&#x27; | sqlite3 -cmd &quot;.mode tabs&quot; -cmd &quot;.import &#x2F;dev&#x2F;stdin YourTable&quot; </code></pre> And also for prototyping you can also use it to tailor output of APIs to what you need in a pinch, using JQ as a library especially with something like python:<p><a href="https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;jq&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;jq&#x2F;</a><p>As a part of the library you can compile your expressions down to &quot;byte-code&quot; once and reuse them.<p>Saying JQ is a best kept secret is an understatement. JQ gets more amazing the deeper you dig into it. Also it is kind of crazy fast for what it is.<p>edit: Formatting fixes

For whatever reason jq is one tool that I simply can never remember the syntax for. It&#x27;s ChatGPT every time for me. I just can&#x27;t remember the specifics of how it differs from jsonpath vs jmespath (used by AWS) .... I wish there was a way for every tool to just use jsonpath instead.

One feature I found very useful when using jq in Bash is the &quot;alternative operator&quot; &#x27;&#x2F;&#x2F;&#x27;.<p><pre><code> result=$(echo &quot;$data&quot; | jq -r &#x27;.optional &#x2F;&#x2F; &quot;&quot;&#x27;) if [[ -n &quot;$result&quot; ]] ... </code></pre> feels more natural in Bash than<p><pre><code> result=$(echo &quot;$data&quot; | jq -r &#x27;.optional&#x27;) if [[ &quot;$result&quot; != null ]] ... </code></pre> Especially if an empty field should be handled the same way.<p>And when using the raw-output option it helps with the ambiguity between &quot;null&quot; and null.

Let me drop a link to my jq zsh plug-in: <a href="https:&#x2F;&#x2F;github.com&#x2F;reegnz&#x2F;jq-zsh-plugin">https:&#x2F;&#x2F;github.com&#x2F;reegnz&#x2F;jq-zsh-plugin</a><p>I find the biggest problem with jq is that the feedback loop is not tight enough. With this jq-repl the expression is evaluated at every keystroke.

Learning jq is great however you still need to know something about every new scrap of json you feed it.<p>To than end I wrote a line of jq to emit every structural path from any json as a list of jq arguments.<p>You can use it to make queries or keep track of a documents structure.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;TomConlin&#x2F;json_to_paths">https:&#x2F;&#x2F;github.com&#x2F;TomConlin&#x2F;json_to_paths</a>

For me jq is my epitome of &quot;When faced with a problem a programmer says &#x27;I know I can use X&#x27; and now they have two problems&quot;<p>I continually bounce off the &quot;language&#x2F;philospohy&quot; of jq in quite embarrassing ways. Every time I go &quot;Ah, I can use this as a reason to learn jq and half an hour lateI&#x27;ve written a python script to extract the data instead.

I started writing a book on jq, but realised it wasn&#x27;t really enough for a full book, so put it out as a series of blog posts:<p><a href="https:&#x2F;&#x2F;zwischenzugs.com&#x2F;2023&#x2F;06&#x2F;27&#x2F;learn-jq-the-hard-way-part-i-json&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;zwischenzugs.com&#x2F;2023&#x2F;06&#x2F;27&#x2F;learn-jq-the-hard-way-pa...</a><p>JQ really is the best kept secret in data.

Great article. Nice to have it interactive. How does it work? Do you have a terminal running somewhere or does it run in the browser?<p>One thing I noticed, and where I stopped continuing, is that the jump from Filtering Nested Arrays to Flattening Nested JSON Objects, is WAAAY too big. From a simple filter to triple nested filters with keywords that had no introduction in a simpler example, isn’t working for me

Exercism also has a jq track with interactive lessons : <a href="https:&#x2F;&#x2F;exercism.org&#x2F;tracks&#x2F;jq" rel="nofollow noreferrer">https:&#x2F;&#x2F;exercism.org&#x2F;tracks&#x2F;jq</a>

It seems like jq is getting a nice boost due to how useful it is getting JSON into and out of OpenAI and LLM environments that understand jq. The big new release&#x2F;relaunch shows the project is up and running again so maybe we see even more integration with Agent&#x2F;Function type use cases or some pydantic-ish guardrails. Thanks for the Bookmark !<p><a href="https:&#x2F;&#x2F;github.com&#x2F;jqlang&#x2F;jq&#x2F;releases&#x2F;tag&#x2F;jq-1.7">https:&#x2F;&#x2F;github.com&#x2F;jqlang&#x2F;jq&#x2F;releases&#x2F;tag&#x2F;jq-1.7</a>

If you have trouble remembering jq syntax (or any other weird CLIs) I&#x27;d reccomend increasing the number of lines of history stored in your shell and finding a way (I use FZF) to search through that history.<p>I do a quick ctrl+r, type jq, and I can find all of my JQ snippets I&#x27;ve used in the past couple of years. If I then type &quot;select&quot; I can find all of the times I&#x27;ve used that function, etc.<p>I also use it to find while loops, kubectl snippets, environment variables I exported to run a script, etc.

This made me think: If you wanted to make an &#x27;inverted bottom-up&#x27; introduction to the suite of Unix command line tools, you could go in the direction of more-to-less-structured text formats and the common tools we use with them quite easily.<p>1. JSON: `curl` to get interesting JSON APIs, `gron` and `grep` to explore what&#x27;s inside them, `jq` to process them into interesting formats.<p>2. CSV: Lots of good choices here. `xsv` is very popular but I think development ended a while back; I like the `csvkit` just because I like tabbing through the options you have here. `miller` I&#x27;ve heard good things about. Or go to the total opposite end of the direction, use Simon Willison&#x27;s excellent `csvs-to-sqlite` in conjunction with `datasette`, and then do a foray into the many interesting things you can do in SQL.<p>3. Bespoke text formats - `sed`, `awk`, and possibly even Vim macros reign supreme here, along with the rest of the &quot;standard&quot; Unix text kit. The big benefits of introducing these last is that these tools work as a superset of many of the previous ones for added flexibility.

Jq has one of the worst, non intuitive, non self evident syntax ever devised on planet Earth. Bash&#x27;s if constructs are a walk in the park compared to general jq syntax. And people try to sort out that mess... somehow people always want to climb a mountain when it&#x27;s in their way or someone say that would be an achievement of some sorts...

Reading through these comments here - some praise jq, others claim it is not possible to actually remember the syntax. There seems a consensus it reminds of complexity in awk, bash, sed... While I appreciate the magic behind jq, from intellectual point of view, and also as a tool, indeed - is impossible for me to remember reasonable part of it.<p>Interestingly I still remember most Perl5 syntax, even the crazy stuff, quite vividly, after some 6-7 years of not-writing Perl code. I wonder why - perhaps because Perl is not so complex (even the PCRE), and perhaps because one needs jq now and then, while Perl can be a primary tool for many things. Sadly, Perl is past its prime now, and there are no implications it&#x27;ll ever do a comeback.

One thing that has helped me write simple&#x2F;intermediate jq code is this: Imagine what <i>the context</i> is for your filter. Most importantly, update that context at each pipe character &#x27;|&#x27;.<p>On an empty command, the context is the top-level of your JSON. As you add filter stages, that context evolves.<p>(this really requires more explanation and diagrams than I have room for in this margin)

At some point in every declarative language’s life, so many features get bolted on to make it useful that it loses its declarative nature, at which point you might as well just use a more standard imperative language. In this case, just plain JS.<p>The same thing happened to HCL.

I thought it must be gojq at first, it&#x27;s easy to forget about Emscripten when golang- and Rust-based WASM are gaining popularity<p>I looked at aoili briefly. I didn&#x27;t see how to reproduce the build.

The one thing I would like to have the ChatGPT thing do for me.<p>(and regexes)

Looks nice! Is there something similar for yq?

pro tip: write in your favourite language and ask chatgpt&#x2F;github copilot to translate it to bash&#x2F;jq

I feel like learning something like jq is rendered almost obsolete with the onset of GPT. Spending time learning the syntax of a tool I will use one every so often seems like a waste of mental energy when I can just rely on GPT to spit out whatever I need on demand

That&#x27;s fantastic! Jq is amazingly powerful and just a little complex once you master it.