Emacs was async before async was cool
Michael Herstine (IRC: sp1ff)
Talk
00:00.000 Asynchronous programming 00:47.200 Automating my music player 01:42.600 Working with the API 03:22.080 make-network-process 05:05.200 The sequence of events 05:57.920 Queues 07:50.480 Callbacks 09:24.240 Client-side code 11:48.080 Demo 12:27.760 Logic 13:15.520 Callback hell 14:53.520 Lisp macros 16:46.400 Conclusion
Q&A
00:00.000 Introduction 00:21.600 How does this approach compare to using tq.el, Emacs' built-in library for transaction queues? 01:10.480 Have you considered using the aio.el library (written by Chris Wellons) that implements async/await for Emacs lisp using promises? 02:45.440 Are you aware that EMMS has an MPD client? There's also mpc.el built into Emacs. 05:20.360 Have you seen the Lonesome Pine Specials? 07:44.400 Would using dynamic/special vars add anything interesting / easier to async elisp in your opinion? 10:16.560 How does your project compare to some of the other MPD clients? 11:55.040 Can you share the code to the macro that creates the callback tree? 14:42.880 There's another package (chuntaro?) in addition to wellon's aio that also implements a coroutine trampoline on the emacs event loop. any thoughts on the async/await paradigm generally red/blue functions, etc? 15:03.440 Any thoughts on the async await paradigm generally, red-blue functions, etc.? 21:06.320 Do you think it's a viable future for Emacs to get out of callback hell? 24:39.320 Generators
Description
While async is all the rage in the JavaScript, Go and Rust communities, Emacs has had the ability to (certain) asynchronous processing all along. I learned this while writing an Emacs package to program to my music server's API.
Music Player Daemon is a project that does what it says: hangs out in the background & plays your music. It has an API that I wanted to call from Emacs. I had just learned async Rust, so I decided to see if I could do something similar here. It turns out I could: I wound-up writing a little async runtime that allows callers to "send" commands to the daemon by queueing them up and returning immediately. In the background, as output from previous commands comes in, new commands are dequeued and sent.
This required callers to provide callbacks in order to receive the results of their commands, which led to a situation all too familiar to Javascript programmers: callback hell. The coda to my story was (finally) coming to understand Lisp macros in order to extend Emacs Lisp with a more convenient construct for sending multiple commands.
This talk will dive into the details of the Emacs Lisp process API, specifically the Low-Level Network Access API, with illustrations as to how it can be used to build an asynchronous queue. It will also introduce Lisp macros and how powerful they can be.
Michael is a developer and long-time Emacs user from the San Francisco Bay area. He hacks in C++, Lisp & Rust and thinks a lot about writing provably correct code. You can find him at:
Discussion
- Q: (Referencing the first half of your talk): How does this approach
compare to using tq.el, Emacs' built-in library for transaction
queues?
- A: Great question; should have mentioned that... I took a look, but chose to just do it "by hand"; I wouldn't have used many of the features offerred by tq.
- Q: Have you considered using the aio.el library (written by Chris
Wellons) that implements async/await for Emacs lisp using promises?
It's implemented using Elisp's record data structure, and turns
the nested callback structure into regular-looking Elisp code
(without extra keywords etc). +1
- A: I wasn't aware, but thanks for the pointer-- will definitely take a look
- Q: not to take away from your excellent work, but are you aware that
EMMS has an MPD client? There's also mpc.el built into Emacs.
- A: Another great point; I am, along with mpdel (another MPD client for Emacs). They are all full-fledge applications-- I just wanted a small, tight toolkit
- Q:Have you seen the Lonesome Pine Specials? I saw your music library
and figured you would be interested. My favorite is the one with
Edgar Meyer, Sam Bush, Jerry Douglas, and I think Bela Fleck and
Mark O'Connor?
- A: LOL I haven't, but I I think I will be!
- Q: can you share the code to the macro that creates the callback tree?
- Q: would using dynamic/special vars add anything interesting /
easier to async elisp in your opinion? i noticed you using
leta lot, but if you defined a variable hmm... not sure if i can
i
was just wondering if having dynamic binding in Elisp opposed to
something like JS adds some power to async programming
- A: lexical binding is easier to reason about
- A: lexical binding is easier to reason about
- Q: There's another package (chuntaro?) in addition to wellon's aio
that also implements a coroutine trampoline on the emacs event loop.
any thoughts on the async/await paradigm generally red/blue
functions, etc?
- A: Longer discussion in the chat room, but I think it's a promising if over-used approach to concurrency.
- Q: How does your project compare to some of the other MPD clients?
- Q: Any thoughts on the async await paradigm generally, red-blue functions, etc.?
- Q: Do you think it's a viable future for Emacs to get out of callback hell?
Comments from YouTube:
- Thank you for the informative video Michael! Now I know a way to avoid callback hell in Emacs lisp!
- Nice talk, thank you Michael!
Transcript
[00:00:00.000] Hey everyone. I'm Michael, and I'm going to be talking to you today about asynchronous programming in Emacs Lisp. I'm located in the San Francisco Bay Area, where I'm a developer as well as a long time Emacs user. You may have heard of async or asynchronous programming. The idea has been around for decades, but it first gained widespread attention in JavaScript back in the aughts. And in the teens it gained tremendous popularity in the DevOps world with Go lang. And just in the last few years, support for async programming has landed in Rust. Well, it can be done in Emacs as well, and this talk will demonstrate that by walking you through a little problem that I actually solved for myself.
[00:00:47.200] Like a lot of these stories, it begins with scratching a personal itch. In my case, automating my music server. I use something called the Music Player Daemon locally, and as the name suggests, it just kind of hangs out in the background. Reads music files, and talks to assorted sound drivers. In fact, it is so focused on that mission that it doesn't even offer a user interface. Instead, it serves an API and invites application developers to build clients on top of that API. Okay, so let's hop into a vterm, and I'd like to show you the MPD client I use for my daily driver, something called NCMP CPP. Doesn't exactly roll off the tongue, but I've got a playlist, I can browse the file system, looks like I can search my music library, yada yada yada. It's got all the basic features. The point that I want to make is that
[00:01:44.854] NCMP CPP is a completely independent project of MPD, separate and distinct. It does all of its work by simply communicating with the Music Player Demon over the API. Well, I wanted to program to that API only from within Emacs. Now, there are already Emacs MPD clients out there, but I didn't really want a full-blown client. I just wanted a few small tweaks over my current configuration. A command to skip to the next song. Maybe shove the current track into the mode line. Things like this. I needed an Elisp API that would let me do this. Okay, well, let's get out of ncmpcpp, and let's get into a netcat session with my local MPD server. As you can see, we get a welcome string. So, it is a server goes first protocol. But after that it's a very familiar text based request response oriented protocol. I can ask for the volume. I can ask for the status. But in particular I wanted an asynchronous API. If I issue a command like find every track in my library, that's going to produce a lot of data. That's a human perceptible pause as Emacs processes all the input.
[00:03:22.080] What I wanted was a style of programming
where I could fire off my command,
have the Emacs command loop keep working,
and only invoke some callback
when there was data available.
Well, Emacs is famously single threaded,
so it shouldn't come as a surprise that it offers
a rich set of primitives that enable the sort of
network programming that I wanted to do.
In particular, it offers
a function called make-network-process.
Now this method offers
a bewildering variety of options,
but at the heart of the matter
it opens a network connection
to some endpoint out there,
and we can configure it to be non-blocking.
It returns a handle that you can use to refer
to this network connection with other methods.
Other methods such as process-send-string,
which as the name suggests,
allows you to send textual data
to the remote endpoint of your network connection.
You can also use it with set-process-filter,
which allows you to associate a callback
with your network connection.
That callback will be invoked
when there is data available
in the process's read buffer.
In other words,
in a request response oriented protocol
like that of MPD, you open your socket
with make-network-process,
send your request via process-send-string
and life will just continue in Emacs
until some data shows up
in the process's read buffer,
at which point your callback will be invoked.
[00:05:05.200] It turns out this was enough
for a purpose built async runtime.
Let's work through the sequence of events
when opening a connection
and firing off a few commands in this style.
So, let's imagine a library
that offers a connection object of some sort,
a caller, and an MPD server out on the network.
The caller will presumably get themselves
a connection object by invoking some sort
of connect method on our library.
We can handle this through make-network-process,
but we're going to invoke make-network-process
with nowait equal to true,
in other words asynchronously.
That means the method is going to return immediately.
We won't even know if the connection is up,
let alone what the response would be.
[00:05:59.247] This has some implications.
At this point we've returned control to the caller,
the Emacs event loop is proceeding
quite happily, and so the caller is free
to start using our connection object.
They might say issue a status command.
Okay, well, in our library
we don't have a connection yet.
How on earth are we going to service this?
Well, we can simply give ourselves a queue
and note down the fact that
we owe a status command.
That's pretty quick.
We've now returned control back to our caller,
and they are again free to issue more commands.
Maybe they issue a play command.
Okay, well, we're going to go deeper into depth,
and note that we also owe a play command.
At some point in the indeterminate future,
the connection will get up,
MPD will allocate resources to track a new client.
They will write the welcome string into the socket,
and those bytes are going to show up
in the Emacs process's read buffer,
at which point our callback will be invoked.
We can parse the welcome string, maybe
note the version of that connection object
that might come in handy.
But the key point is our callback needs to
take a look at the queue and notice:
"Oh. We owe a status command,"
and so we'll invoke process-send-string,
and send the status command down the pipe.
Again, at some indeterminate time in the future
some bytes are going to show up
in our process's read buffer
and our callback will again be invoked.
We've got volume is 75 plus a lot of other stuff,
and here we come to the next problem.
[00:07:50.480] If our caller invoked status, they probably wanted to know about the status, so how shall we get them to them? Well, there's really not a lot of options at this point except the callback. Okay, so change of plan. Our queue is no longer a queue of commands. It's going to be a queue of commands with associated callbacks. We read the response off the socket, invoke our caller supplied callback, and then pop the queue. At this point our callback (the library callback) needs to know that we still have a pending command, we fire that off down the pipe, at some indeterminate time in the future we get a response, our callback is invoked, we invoke the caller supplied callback, and we pop the queue. The structure of such a program is best viewed as a finite state machine. And this is typically where you end up in asynchronous programming, at least when you don't have a runtime grafted onto your program the way you do with Go lang, or when you don't have sort of extensive library support the way you do with Rust. Your data structure exists in one of these states at any given time, and when input shows up on your file descriptor, you transition along one of these edges to a new state.
[00:09:24.240] Cool. So, let's take a look at some of the code
that flows from this.
Okay, let's hop over to an Emacs
and take a look at how we might code this up.
If you recall the sequence diagrams I shared,
we're going to be scribbling down the command
and the callback that we'll be invoking
upon its completion.
The first thing I did was
give myself a little command struct.
With that, I was able to define
the connection object.
We're going to be storing
the handle to the connection.
We're going to write down the protocol version
that we harvest from the welcome message,
and of course we'll be recording
the command queue as well.
And so I gave myself a little connection object,
connection struct with those three attributes.
With the data model squared away,
it was really pretty easy
to code up the connect implementation.
I'm eliding some details for exposition purposes,
but in the event it's really not that more complex
than what you see here.
We're going to unpack the arguments,
figure out where the MPD server is
to which you would like us to connect.
We'll connect via make-network-process.
We'll associate a library defined callback
with that connection via set-process-filter.
Then we'll instantiate the connection object
and return it to the caller.
Once the caller has a connection object,
they're free to send commands down that connection.
So, what we're doing here is simply
instantiating a command object
on the basis of the caller supplied arguments
and appending it to the queue.
Then the last thing we do,
and I've just indicated this with a comment,
is we kick the queue.
This kind of goes back
to the state transition diagram
I laid out earlier.
What this means is the logic for saying,
well, if we're awaiting the completion
of a previously sent command,
there's really not much more to be done.
We're just going to push this command
onto the queue and return.
On the other hand, if the queue is empty
on entry to elmpd-send,
there's no reason not to just
immediately send the command.
This is an example of the sort of client side code
that results from this API.
[00:11:48.120] So, you can see here, we are giving ourselves a connection to the MPD server on the localhost. We're going to send the get volume command down that connection. And if that command completes and all is well, we'll just send a message to Emacs. Unfortunately, you can't see my minibuffer, so I'll hop over to the Messages buffer. And there's our result. The volume is 43. Great, I thought. Simple, clean, responsive, easy to code to. That is unfortunately not the end of the story.
[00:12:27.800] Let's continue this example a little bit.
Let's imagine that
if the volume comes back from the server
and it is less than 50,
we would like to set it to 50.
So, this is interesting
because we have two commands
and whether or not we send the second command
it is going to depend on
the response we get from the first.
Okay. I thought, well, that's fine.
I can simply put that logic in the callback
that I specified for the get volume command.
So, here we are we check the return code,
we parse the volume, we compare it to 50,
and if it's less we just invoke elmpd-send again
from the first command's callback.
Okay, I could live with that
it's not the worst thing I've ever seen.
[00:13:15.520] Let's extend this example a little further, and this is contrived, but bear with me. Let us suppose that if we do set the volume to 50, we'd like to get the volume one more time just to make sure that our change took on the server. We can play the same game. We will put that logic in the callback that we specified for the set volume command. And here we are, we check the return code, we send a message to Emacs, we send the get volume command again along with its own callback. And at this point I hope it's clear (the problem that is emerging), and if it's not yet, let me note that so far we're only handling the happy path in each of these callbacks. We really ought to do something about the error path. For purposes of illustration, let's just say, we send a message to Emacs that means it would look like this. At this point that I really think it's impossible to deny that this API is actually not that easy to program to. If there are any JavaScript devs watching, you're probably chuckling right now because I have discovered for myself what they call "callback hell". If you are returning the results of asynchronous function invocations to their caller via callbacks, you pretty much inevitably end up in this sort of deeply nested sequence of callbacks that is difficult to write, difficult to read, and difficult to reason about.
[00:14:53.520] And yet when I was stuck in this situation
it just seemed like it really shouldn't be this bad.
If I give myself this sort of tabular data structure,
I felt that this expressed precisely the same logic
just in a much easier to read manner.
I could, in my mind's eye,
see the code for transforming this data structure,
which is really just a list,
into the code that you just
saw in the previous slide,
and really if Lisp is good at anything
it is list processing right.
And it was really at this point
that a little bit of enlightenment dawned.
I learned that Lisp is homo iconic,
which is just means that the language itself
is a data structure in that language.
Lisp code is, after all, just a list.
And the power of Lisp macros
is taking that data structure,
some data structure that you've defined,
and doing exactly what I wanted to do.
Transforming it from one list into another,
the destination list being Lisp code.
So, I got busy, and I coded up my first Lisp macro,
which I called elmpd-chain.
And that lengthy list of, you know,
three or four nested callbacks
gets turned into this,
which I hope you'll agree is much simpler
much easier to read, much easier to reason about.
And if you're morbidly curious,
you can expand your macros,
and this invocation of elmpd-chain expands to this.
So, that's my story.
[00:16:46.400] In all fairness, I should note that the MPD protocol has some subtleties and complexities that I didn't really get into, both due to time constraints and because they're not terribly relevant to the points I wanted to touch on. I should also note that there's a fair amount of work in the library itself around accumulating partial responses as they show up in the buffer and dispatching them piecemeal to the caller. That was really too complex to get into here. If you would like to see the code, it's available on GitHub as well as MELPA. I'll be putting a version of this talk on my personal site, and you can always reach out to me personally, I hang out on IRC as sp1ff, or you can just email me as sp1ff@pobox.com. Thank you very much.
Captioner: bhavin192
Questions or comments? Please e-mail emacsconf-org-private@gnu.org