Back to the schedule
Previous: Sharing blogs (and more) with org-webring
Next: Day 1 closing remarks

OMG Macros

Corwin Brust

Download compressed .webm video (45.8M)
Download compressed .webm video (24.3M, highly compressed)
View transcript

Macros are a powerful tool. In the context of Emacs Lisp programming they can also provide us with a "foot-gun" of immense proportions. Join the dungeon-mode project as we trip over our own macros, so to speak, in the context of building a GPLv3+ turn-based role-playing game engine and game design features for Emacs.

In this 20m talk I'll briefly introduce some rationales leading to storing all game source and play state information within org-mode documents (spoiler: it's about freedom), then go into some detail around the "ETL" process design that currently accomplishes this. Finally, we'll look closely at one especially problematic macro deep within this solution, and invite people to throw fruit^11dhelp draw conclusions, ask questions, and discuss.

Additional Materials

Kind	Target	Size	Description
org, svg	tar.gz	25k	Notes for all talks
demo	webm	26m	Character Sheet
demo	webm	19m	"Sketch" Map and Tile editor
demo	webm	16m	Battleboard, damage tracking
demo	webm	9m	Game Maps, controlling fog-of-war
demo	gif	724m	All demos, no overlays
demo	json	274K	OBS scenes
elisp	www		Corwin's init files

• Actual start and end time (EST): Start: 2020-11-28T16.17.32; Q&A: 2020-11-28T16.34; End: 2020-11-28T16.38.32

Questions

How is your background work?

See 06: Trivial Emacs Kits's Q&A: Corwin uses Wallpaper Engine.

• [Corwin] Wallpaper Engine on Steam is probably the thing that's grabbing attention. I haven't tried it under GNU/Linux. My family are (mostly) Windows users right now heavy sigh I don't want to get into my tool chain a huge amount, but I will talk about it some as/durning the Welcome to the Dungeon talk tomorrow. For now I will say I'm using a mix of free (free and not-free but too easy to avoid tools on my one pretty good computer). I would love to have the time to invest to use more (only) free stuff but sometimes we we can't afford the freedom, in terms of the learning curve. I think this is the most important problem space in free software, FWIW.

What was the key message you wanted to share with your talk?

Macros are powerful and necessary. Consider how you use them?

Do you mind if I use your macro code as inspiration for an elisp uglifier?

Have At! It's GPLv3 and you are welcome; let me know if you have any trouble finding fruit to throw.

Transcript

Good evening again. I think I have a little time here to talk about macros. Is there still room in our schedule for that, or should I just jump to some of my thoughts on the day? (Amin: Pretty sure we have some time.) Corwin: All right. Great. (Amin: Yeah, go for it.) Well, I'll just dive into my prepared thing here then. (Amin: yeah, actually, you're right on time, so...) Corwin: oh what an amazing thing. I just... You know, I have been trying to do what I... I've got a big thank you planned at the end, but let me just say, it's been really cool to watch the way that people work together. (Amin: Absolutely. It's... This whole event today has been nothing but awesome, and no little part thanks to all of the help from all of you guys and everyone. Yeah, it's awesome. With that, I'll just shut up for now. Take it away, Corwin.) Corwin: Who knows how to make make that the default in good old smex?

All right. So I'm gonna try to continue my theme from the previous talk. I'm a longtime Emacs user, but I'm a pretty new person to trying to really understand what's going on within Emacs and make my customizations to it--simple for what I tend to just think of will work. And maybe that's a nice bow to put on that earlier talk. Let's see here now. It's C-x M-i. That's right. And let's try that again. Okay, good. So demoing is fun, but I will save most of that for tomorrow where my dear friend and co-collaborator in bringing you the dungeon-mode project, which is sort of the exciting thing that we hope you'll be interested in, gets a little more of a reveal.

Tonight, I'll just close saying a few things about the process of making it and continuing my theme of community. First of all, a specific and upfront shout out to tv's wasamasa who absolutely shaped and guided this this program. I may have taken out a slide with your name on it, but thank you.

So when we think about Emacs macros and the power that they give us, I think about them as a really deep rabbit hole. They confuse people a lot. And so, to try to center myself on that, I remember first that they're going to be talking to us about code. Excuse me, I realize I hadn't set my timer. Here we are.

So a simple macro syntax is going to generate something that is implicitly confusing to somebody that knows the syntax of Emacs Lisp well. We see something like this and a veteran eye says "That x isn't quoted. What's going on?" but it can be hard to miss. A lot of the functions (as we'll talk about in a moment) that are built into Emacs really are macros, so a lot of Emacs features work this way. It might be scary, but we have to look at it closely if we really want to get friendly with Emacs.

Let's just jump right into defmacro, which is our key entry point. The notes from this talk include the link to that, which... Definitely read through a couple of times. That may take you through into the cl-def macro, which adds the Common Lisp extensions. Definitely challenging. I've struggled there, as we'll take a look at in a moment. So I haven't played too much with cl-maclet. Perhaps success in in that keyword space and figuring out what the right balance is there will give me the confidence to try some more lexical macros.

Let me also briefly introduce the comma (,) and back quote (`). If you have allowed your eyes to cross when you see these, that's not a shameful, shameful thing. It's confusing, and we should be alerting each other when we stick macros in, often by putting them in different library spaces for complicated projects, or otherwise warning people that this is not an interactive function, even if you get away with using it like one. Watch your back.

The manual itself talks about macros as being a way of evaluating, as being an evaluator that will take our Emacs Lisp expression and the set of forms that will feed to it our code, but it also provides us with this concept of an environment. That's really where the power comes in. Through that, we can have lexical variables and think about--bring in some of the capabilities that can be harder to reach with a pure declarative statement that doesn't allow for top level asynchronous... Asynchronicity... I'm gonna basically ignore the byte-compilation phase for this talk in order to have any prayer of getting through it in the remaining 9 or 11 minutes or whatever. But suffice to say, that's a scary space, and that's really the thing that you want to start learning about as you think about taking macros on in earnest.

Coming back to the comma syntax, then, having given ourselves a working definition for the Emacs Lisp runtime environment, then we can say that macros are going to inject code back into that stream, whereas backquote (`) is going to give code back. to the stream--or interject, sorry, it's going to interject back into the stream. Sort of an exclamatory "Excuse me, I'd like to have a value here." We can take that value from the environment as it exists when our macro is evaluated. Backquote, on the other hand, takes the result from that and returns it back to the stream for evaluation at the processing level that invoked us. So in other words, perhaps back up to a top-level eval expression where our macro is invoked.

I'm going to briefly bring you back to the game for just a moment. I won't linger on this slide, but briefly: this is a role-playing, pen and pencil, physical dice tradition that dates back a long time from a technology perspective. It's old in the same way that other tools that I like are old. It's simple to understand. I can communicate a lot with it with a simple amount of typing or scribbling something on a piece of paper. It has a complicated problem space of its own.

Again, I don't want to get too much into the game here, but in this talk, for the last five minutes, I'll focus on the process that we took to automate getting data out of the Org Mode tables which eventually (as we'll talk about more tomorrow) are used to draw game maps and other things. Here I talk about why we did that. I'm going to skip briefly past that, and say instead that at a high level, it's symbolic informatics. We're giving a symbolic name to a tile set, and then assigning that tile set some some characteristics like physical speeds, screen space (a variable that we might want to swap in), and so forth. You know, our project rests heavily on Org Mode and its fundamental capabilities. The code I'm going to show here is focused around a sticky problem space in information technology. I'm a professional software engineer turned technology architect. I support the websites for a recognizable financial services brand that I don't identify just so I don't accidentally end up inadvertently misrepresenting my firm in some financial perspective if I let some other companies' name slip, or my own. It's certainly no representation of an opinion other than my own. So ETL has to do with moving data around. We have the idea of a pipeline where we'll be able to verify certain assumptions, nominally about data quality, but it could be about anything. Before the pipeline starts, okay, we've got a state where we think it should work if we run it. We have some extraction where we'll get our sources, and we may have the opportunity to make some assertions there. In the transform stage, as well as the load, things get a little dicer, to the point where we come out of the load stage and we should have some really solid assertions again that we can even go back and compare to the extract stage. From this, we have the rudimentaries of a data quality practice. In this case, we have a number of Org Mode files that will all be distributed across a number of players' computers, so we might not want to update every part of every buffer. I think it's a complicated problem space. So we tried to take a long-term view of the solution that we needed. So I'll go ahead and open up the function that... Let's actually start with the one that's pretty easy to read. I'm gonna go ahead and just crank it up huge, in case anybody's watching in 480.

This program is not a work of art. It's a simple implementation of the idea that an alist of functions that return maybe some data, maybe some data and an entry back into that alist, can be done quite extensively with very few lines of code. Neither is it an especially tight or thrifty implementation. It's just trying to get the job done with a doc statement for everything. At the heart, we see a call to this macro called dm-coalesce-hash, and that's what I'd like to focus in on. You can see... I think that something unpleasant is happening here. I've got an eval in what is...

I will share a fairly central function that those implementing this ETL pattern are welcome to derive from. That is, this is a default transform that you can get when loading certain kinds of org-mode tables that have been properly adorned. Again, we'll get into that all tomorrow. So, keeping an eye on time. Couple minutes left.

Let's look at the macro itself. I have a slide on this, but let's go ahead and risk getting off page. Oh boy. Here we go. So this is my utilities bucket. It has such basic features as "give me a hash table with some defaults, I'll think about that later," and "add to list," a special version that enables us to be a little cavalier in experimenting with alist versus hashes versus plists. We've made a right mess for ourselves in the proof of concept area, and it's ripe for someone to write a whitepaper about when to prefer these things. The merge alist... Same work here. Let's get down to business.

This function has quite a... This macro has quite a doc string. I think I mentioned earlier that I got myself into trouble with the keyword properties. You can see that we have not only quite a number of them, but a lot of default values, many of which may be relying on the values that are passed in here. This is complicated. As it turns out, I wasn't brave enough in most cases to try to write a lambda that could understand and replace its own local variable. I just didn't... It didn't save me enough time. This was really easy to read and write and understand as I thought through my problem, but now, as I use it, I've lost a little ground with this. I'm not even sure I like what I got from the many keyword properties when it... And we can look, perhaps if we have the time, at what that looks like in Oh, all right, I have to separately dismiss and restart that.

So that's just about my time. Being respectful of that, I want to invite presenters to just jump in at any of the many large pauses I leave. I'll just leave up the doc string for a moment and maybe split the screen and pull open an item. (Amin: Thank you very much for your talk, Corwin. I think you still have maybe three or four more minutes, if you want to quickly wrap up.) Corwin: Okay, so three or four more minutes I can easily spend on thank yous. I might switch to that if there aren't questions on the pad. (Amin: Would you like me to pull up the pad, or are you looking at it?) Corwin: I am. I bookmarked it. I am pulling the tab and I'll bring it in. Okay. All right. This is the wrong Etherpad. Thanks for the link. All right. So I think I'm looking for macros.

Okay. Key message. Sure. So, the key message is that it's a jungle out there. Macros, along with any other design, can leave you in a position where you have a nice API. I can show you other examples (you can find them in the dungeon-mode source) of many, many other places where I use this exact same formula, quickly sketching out how a character sheet or another big data set needs to figure out what tables are going to be interesting from the collection of files, and then load up the tile set, and the layout file from that. And I mean, it works. The project is moving forward with this. I have the flexibility that I need. But here I am evaling my own code to make darn sure even if I get byte-compiled, this macro does get evaluated in the user's real run time. Clearly a design fail. So that would be...

The key point of my talk is to present this design fail and thank the community, but especially wasamasa for some patience. Let me add at this moment that he was so frustrated with me. They were sort of frustrated with me (I think I didn't qualify pronouns) with doing this. The first... This was one of our first interactions, and the feedback was, "Why is this a macro. Full stop." And that's a great message, actually. I hope that maybe this can encourage further talks across the subject about, you know, "Hey, wait a minute, macros are really fantastic," as I hope I made clear. You can do a tremendous amount with them, and we rely on them for almost all the fun goodies, from defun, setq... I want to get to my thank yous.

Let me just peek back at the pad. Well, that was actually a scratch buffer, so I'll have to read it cold off my notes. But I'll switch to... I'll also... I'll say a couple of thank-yous if you don't mind, Amin. In addition to the big thank you that I hope was implied by my shout out to wasamasa, I also want to thank you, Amin, for your kindness in extending to the project as well as to me, the the chance to present here. You've also done a lot of great stuff for our project. Thank you very much for that. Sacha Chua (I'll get there), thank you so much for the inspiration that you are to our whole community. I also want to thank the presenters for just being so flexible and nagging back through the whole thing, and especially to Leo who has done so much to drive the show today. This is a fractious tent at times, and sometimes it is indeed a little bit of a circus, but I am learning so much so fast. I'm just inspired by how much Emacs can teach us. (Amin: thank you, Corwin, for your kind words about me, of course, about all of us and the conference... Indeed, thanks to everyone who's helped, including the speakers, of course, without whom EmacsConf really wouldn't have been a EmacsConf. It's been a pleasure knowing you and working with you, from afar for the most part on dungeon-mode, helping with small things here and there but yeah, it's been my pleasure, and it's great to have you and everyone else part of the community, and for me to be part of the community. It's been a lot of fun. Thank you.) Corwin: It's an honor. I don't use that word an awful lot because I sort of smirk at it. It gets us in a lot of trouble, honor does, but this will be a sure time to use it. (Amin: Thank you.) Corwin: Likewise.

Saturday, Nov 28 2020, ~ 4:16 PM - 4:36 PM EST
Saturday, Nov 28 2020, ~ 1:16 PM - 1:36 PM PST
Saturday, Nov 28 2020, ~ 9:16 PM - 9:36 PM UTC
Saturday, Nov 28 2020, ~10:16 PM - 10:36 PM CET
Sunday, Nov 29 2020, ~ 5:16 AM - 5:36 AM +08

Back to the schedule
Previous: Sharing blogs (and more) with org-webring
Next: Day 1 closing remarks