Back to the schedule
Previous: Analyze code quality through Emacs: a smart forensics approach and the story of a hack
Next: NonGNU ELPA

Traverse complex JSON structures with live feedback

Zen Monk Alain M. Lafon

Download video
Download captions

Download compressed .webm video (18.1M)
Download compressed .webm video (15.8M, highly compressed)
View transcript

If you are working with complex nested JSON structures, you are probably familiar with jq which is like sed for JSON data and great at what it does. However, being a command-line tool like sed, the feedback for writing queries and seeing their results is a discrete process and not live.

When working with Emacs, we are used to good auto-completion and live feedback. Formerly, this was mostly done with static input, but with modern completion frameworks like Ivy and Counsel, this can be done with dynamic inputs, as well.

counsel-jq is a package with which you can quickly test queries and traverse a complex JSON structure whilst having live feedback. Just call M-x counsel-jq in a buffer containing JSON, then start writing your jq query string and see the output appear live in the message area. Whenever you're happy, hit RET and the results will be displayed to you in the buffer *jq-json*.

In this lightning talk, I'll give a quick overview on how to use counsel-jq and how to build similar completion functionality.

Resources

https://200ok.ch/posts/2020-11-30_emacsconf_traverse_complex_json_structures_with_live_feedback_with_counseljq.html

Questions

Q4: Any plans for counsel-yq and/or -xq? ;-)

counsel-jq currently just shells out to jq. Adding tools build on top of jq (at least yq is afaik) would be very easy. We could employ a strategy pattern to find the right tool based on the current major-mode with a configurable fallback. Here's the place where the shellout happens: https://github.com/200ok-ch/counsel-jq/blob/master/counsel-jq.el#L23

Would you be interested in making a PR for that? (;

Q3: Why repository_url did not autocomplete in addition to the result (I know that it is ivy thing but possible to configure?)

There's no autocompletion for the search query, but that would be a great addition. That would theoretically be possibe by employing jq to look ahead in the current tree and providing options for autocomplete. I'm not certain if Ivy does have autocomplete for search queries, though.

If somebody has more knowledge on that and would like to ping me up or provide a (draft) PR, I'd be happy to help out in that endeavour!

Q2: Is it difficult to provide autocompletion for the JSON query in the minibuffer?

Good question. I'd be curious, too. It's the same question as Q2 where I went into a possibe scenario.

Q1: Is it possible to search in arbitrary deep objects? E.g., an AST represented in JSON.

counsel-jq uses jq under the hood, so all queries that are valid queries in jq should be valid in jq. Hence, I'm inclined to say 'yes' (;

Notes

Transcript

[00:00:00.799] Hello, everyone, and welcome to this short lightning talk: "Traverse Complex JSON Structures with Live Feedback." This is a pre-recorded talk and part of the EmacsConf 2020 schedule. This is what we're going to do. I'll make a quick introduction to the topic at hand. I'll give you a demonstration of some tools, and then we'll leave you with the links to said tools.

[00:00:29.199] Before that, just a little bit about me. I am the CEO and co-founder of a company based in the Swiss mountains called 200ok.ch. We are a product incubator and service consultancy, but we like to spend most or at least as much time as we can building free software. I'm also an ordained Zen monk and abbot of the Lambda Zen temple. You can reach me anytime on questions regarding Emacs, for example, at alain@200ok.ch.

[00:01:07.200] But back to the topic at hand. The proposition is as following: most work on the computer is based on either text processing or text consumption. And very often, the text which you need to process is in a structured format, for example, in JSON. That might even be if your job is not programming per se. Reading through such a bigger chunk of JSON can be non-trivial, however, while just reading and understanding it will be essential to getting your job done.

[00:01:40.320] So let's quickly check out an example JSON file. This is from the Github API, which is a request--sorry, the response to a request for a specific issue on the github API. So let's quickly check that one out. Okay. So here it is open, and we can already see that there is lots of stuff going on here. It's 200 lines. It's not going to be very easy just to find out what are the top level things in here, what are the top level attributes. Of course I can do this, and maybe do it by hand, but that doesn't scale. I can use cool Emacs facilities like the hideshow-mode and try to fold all the things that are top level, but that also doesn't really scale. There must be a better way. Of course there is. There is prior art.

[00:02:32.000] There is a tool called jq. I'm going to quote the USP (unique selling proposition) from their website: jq is like sed for JSON data. you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep, and friends let you play with text. Let me give you a quick demonstration of it. By the way, it's written in portable C. It has zero runtime dependency, so it's very easy to get started with it and use it on pretty much any UNIX-based computer. Sorry, no, Linux-based computer, apologies.

[00:03:14.000] Okay, so let's explore a JSON file with it. It's a command line tool, and it has a very simple command line syntax. So you call the binary and then you give it a query and a file, and then it will return its answer. So, for example, if I want the top level keys, I will just say jq keys the file and it will return the keys. Simple as that. So let's check this out in a real shell. Here I am in eshell. Let's run jq keys on the Github issue comment. We can see that we have actually received a list back here with the top-level things. So this issue... It looks very interesting. Let's ask it to give me more information on this issue. Then it's hairy again. That's a lot of stuff.

[00:04:11.360] I mean, lucky for us, we are in Emacs here, so we can use nice shortcuts. We can copy this. We can go in here, just select that, get that out or something like this. But still, this is not really the best way to do that, right? it gets kind of tedious. At this point the output can be humongous. The shell is not really the best place to read through such big output. I mean, eshell is probably one of the better shells for this, because it's just a regular Emacs buffer, but still, it's not really the best tool. I need to repeat the command all the time until I finally build the right query. And all the time, I lose my focus, I lose what I'm currently looking at. I'm seeing the new result.

[00:05:05.520] It would be so much nicer to have live feedback. When working with Emacs, we're quite used to that. So there should be an option. And of course there is. It's Emacs, right, so you can do anything.

[00:05:17.759] There is various good tools for completion in Emacs. I used ivy for this. I'm going to quote the USP for ivy. ivy is a generic completion mechanism for Emacs. While it operates similarly to other completion schemes such as icomplete mode, ivy aims to be more efficient, smaller, simpler, and smoother to use, yet highly customizable. And that's true.

[00:05:46.479] One of the cool things of ivy compared to other completion mechanisms in Emacs is that it can be used on dynamic data. So usually completion works on a static input. For example, you're in a buffer, a text buffer, and you use isearch maybe with ido-mode, and you find your results. That's all nice. However, if I want to search on dynamic data, that doesn't work. So whenever I type in my query for jq, I actually need to call the jq binary, and it will give a different result set back. So it's a really dynamic mechanism that we need here. It's much more like a search engine.

[00:06:38.240] ivy luckily has something built in, and it's called counsel. So I used counsel and jq and combined them, and built a new package with which we can use Emacs and jq to have live feedback. It's very easy to use. So you just call counsel-jq on a buffer containing JSON. For example, the one we have here. Let's call counsel-jq on it, and we already get a default query, the dot query, which just gives us the same file. But now we can change it. For example, find all the keys in here. And then we see I had this issue. This was the one that we were interested in. So let's find more information on the issue. What keys does it have actually have? It has assignees. That interests me. So let's check out the assignees in here. There's two of them, but I'm only interested in the first one. I'm making stuff up as I go here, of course. Whenever I hit enter, I get a new buffer which just shows me this particular result for the particular query that I entered.

[00:07:55.599] So let me do that again. We are in here. We are looking at a JSON file. This can be very, very big. Doesn't also need to be a file. Just needs to be a buffer. You call counsel-jq on it, and you can do any kind of query on it. For example, let's see if there is a URL here. Yes, there's a URL. Let's see if there's a repository here. Repository. No, there isn't. What was it called? Issue. Keys. Repository URL, it was called. So let's see issue repository URL, and then we see. So apparently this issue comment is for a repository called organice. I wonder what that might be.

[00:08:47.839] Okay. So that was a very short introduction to counsel-jq. You can see the timer here. I only have one minute left to go, so I'm going to leave with a very, very short introduction to the counsel-jq code. It's not even 60 lines of elisp, so building something like this is very, very easy. I would encourage you to go and read through the code in your own time, if you're interested in building something like this. If you're interested in just using jq or you're done, these are the links to all the tools. counsel-jq, of course, is readily available on MELPA. Also developed under the AGPL license on Github. And this organice thing, by the way, it's Org Mode for mobile and desktop browsers. Also a great free software tool maybe that interests you. Thank you for listening. Have a great time. 10 seconds left. I am going to stop this now. Enjoy EmacsConf. Have a great day.

Sunday, Nov 29 2020, ~11:12 AM - 11:22 AM EST
Sunday, Nov 29 2020, ~ 8:12 AM - 8:22 AM PST
Sunday, Nov 29 2020, ~ 4:12 PM - 4:22 PM UTC
Sunday, Nov 29 2020, ~ 5:12 PM - 5:22 PM CET
Monday, Nov 30 2020, ~12:12 AM - 12:22 AM +08

Back to the schedule
Previous: Analyze code quality through Emacs: a smart forensics approach and the story of a hack
Next: NonGNU ELPA