orgvm: a simple HTTP server for org

Corwin Brust (he/him/any, IRC: corwin)

The following image shows where the talk is in the schedule for Sun 2022-12-04. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 47-min talk followed by included in main video Q&A (#emacsconf-gen)
Etherpad: https://pad.emacsconf.org/2022-orgvm
Discuss on IRC: #emacsconf-gen
Status: TO_INDEX_QA

00:36.000 Introduction
01:31.080 What is orgvm?
02:01.880 Nodejs
02:49.560 The itch I was trying to scratch
03:38.320 Demo
05:29.760 Needs a relatively recent version of Emacs
06:24.920 Usage patterns
08:13.520 Emacs Lisp
09:09.160 Variables
10:38.720 Replace
11:19.120 Getting into the code some more
13:06.480 Generating Elisp
13:37.320 Org blocks
14:32.400 Building some Lisp
16:43.000 How Elisp gets encoded
19:25.040 How the export works
22:09.860 Walking through the code
26:07.440 Executing the source block
32:39.760 Conclusion
33:58.880 Questions and answers
35:48.940 Why am I not running the web server in Emacs?
36:22.140 Is this using org-info-js?
37:35.460 EmacsConf
38:15.580 How I'm using this at work
42:04.340 Volunteering for EmacsConf
43:08.380 It's easy to build a program that uses Emacs in the pipeline

Description

OrgVM is, so far, a very simple nodejs wrapper, invoking Emacs in batch mode. I'll talk about how I use it to make project notes available on demand to my colleagues and how it works, especially the generation of elisp from javascript/JSON.

Discussion

Questions and answers

• Q: Why not build upon Nic Ferrier's elnode web server written in Elisp?  As Nic describes it: "Elnode is a non-blocking IO webserver written entirely in EmacsLisp. It's like an EmacsLisp version of node.js."  Your implementation will likely get much simpler if you keep it all in Lisp.
  • A: I chose to build with node.js because that was easy for me; but an Emacs based version sounds awesome!
• Q: Is this using org-info-js? 
  • A: nope; I need to learn more about this - seems very exciting and like it could influence this project
• Q: Why did you make yet another web server?
  • I read it was to make your org files portable.
  • He's auto-generating .html files to match his org files in the background so that he can browse them via web browser right?
• Q: How is that portable. You still need a web browser. Org files need the very basic..... cat
• Q: Have you heard about https://codeberg.org/gopiandcode/emacs-viewer, by any chance?

Transcript

[00:00:36.000] And so this little application-- well, I'll skip that and just kind of jump right into my thesis for those of you that might be planning to duck out for the RMS talk, starting in a little bit. So essentially, my thesis here is really that the Emacs toolchain can easily be combined with other skills and used in kind of a Unix paradigm of having sort of different tools to do different steps. We might actually use the same tool to implement a couple of steps. But with that paradigm, each step is an individual item that can be sort of dropped in and replaced. So over the course of the talk, hopefully I'll come back to that thesis.

[00:01:31.080] But I'll now jump back and start walking through what is orgvm? So this is a very simple proof of concept program. We'll just jump over to perhaps a prettier view of the source code for it. This is implemented-- oops. There's some cruft, I think, in my local. All right, so this config block at the top... And we'll be jumping back and forth between the code and the documentation.

[00:02:01.880] So the first thing I want to point out is that this is written in Node.js, but I think you'll find it'd be pretty trivial to implement in any language. Certainly, you're more than welcome to use this. I'd be happy to accept your patches or feature requests and things like that. Of course, bug reports. But I'd also encourage others to roll their own. You might well come up with a different version of this that's even cooler. And we can learn from each other. If you heard one of my talks before, you probably recognize a common theme. I'm a big fan of head-first development as a way to get invested in both the tool chain and a culture. All right, so let's come back to orgvm.

[00:02:49.560] First of all, we'll start with the itch I was trying to scratch. I wanted to be able to quickly use a web browser to browse my Org documents. It's particularly handy when the documents are full of cross links to each other. That meant I wanted to automatically export, particularly to HTML. But it made sense for me to include Markdown, PDF, or whatever format I want. Because many times, I'm going to look at that file and then pop it into an email or upload it somewhere. And then finally, it should be, therefore, pretty easy to download the document rather than view it, once I'm done.

[00:03:38.320] So let's just run a quick demo. You'll see I'm still a Windows user. Yeah, I'm working on it. So all right, first thing that we're going to do is fire up the program. Actually, for simplicity, let's just admit we live in a DOS world. And as you can see, there's not much to it to get the application running. So with that done, then, I can run out to my localhost. And we'll just start by plugging in the name of an Org file. So I've got a little Org file that I prepared that just kind of provides a proof of concept to this. And you can see, as imagined, we're automatically turning that Org file... Let's just take a quick look at it. And here's that file now. You can see, nothing up my sleeve. This is a very basic Org file that I use for testing this program. Images work. We've got some nicely syntax-highlighted code blocks in a couple different languages, and not really that much going on there.

[00:05:29.760] All right, let's come back to the documentation. I pretty well covered this, I think. But you'll need a relatively recent version of Emacs. I haven't taken any pains to make this backward compatible. To be fair, I haven't tested it extensively. It may well work on Emacs 26 or older versions. I'm personally running 27.1 and 28, as well as recent builds of 29. There's some quick start instructions here, which I'm going to take as read. You probably saw the key element of this, which involves starting the program. You do-- I will call out Yale. If you're trying to play with this yourself, don't forget to run the npm install command. That'll bring in express.js, which the JavaScript we're about to look at is built on.

[00:06:24.920] So let's just take a look at the usage patterns real quick. To use this, we're simply giving the document name without the .org extension in whatever file path-- or I'm sorry, whatever we've configured the server to run on, in this case, port 3000. I also want to call attention to the fact that nothing in this program protects you from damaging yourself. This isn't meant as a production capability. This is something that's used to publish your own note files and roll them out to yourself. That's something I'll definitely look at adding, but I want people to be careful of it while this is in an alpha state. So the default response is HTML, and we saw that here. But we also can modify the response format. We're currently supporting HTML, Markdown, and PDF. And that's really enough to select a different format. That's really nothing more than adding type. Okay. Not sure what's going on there. Okay, well, there goes my demo. Shows me for doing my talk live. But this, fortunately, this error message is a nice segue to the part of the talk that I'd really like to focus on, hopefully bringing me back to that thesis.

[00:08:13.520] So as we start to look at code, what we're looking for is really this Emacs Lisp that's getting generated here. And you'll notice that's the stuff I thought was important to produce as diagnostics for the programs running as well. So, spoiler, this Elisp is dynamically generated by the program. And that's really the core of the way orgvm or my orgvm works. So this should look pretty similar to the view of the code we had a moment ago. You can see I've got some bases. This is all hard-coded into the program, nothing fancy going on here. The debug is simply controlling that diagnostic output that we looked at. There's some other, hopefully fairly self-explanatory programs or properties, where to find Emacs and so forth.

[00:09:09.160] And then finally, we come in to the meat of it, the variables that are used to control what Elisp we can generate dynamically. So here, we're controlling the extension that it should look for Org files. Hopefully not too many people out there with a weird extension for the Org files, but this should support that. I'm afraid that is something I've been known to do. Then we define a list of additional export types. Here's one that ought to work. Let's take a look at type=org. And, aha, it's giving us the file. So I'm not going to open that up, but now we can see that that's definitely working, for certain versions of working. So this list of type parameters is controlling the supported types. Hopefully it should be fairly easy to add in different ones. The fancy footwork here is just a list of the types that we're going to be using. The fancy footwork here involves, first of all, there's the extension and the MIME type. That's, as you might guess, used to control the response content type.

[00:10:38.720] We also have this replace variable. This prevents-- there's an optimization to send an existing PDF or HTML file if that's already there, but only if the original source Org file hasn't been modified since. This replace effectively can turn that off. If I remove the replace: true attribute, then I'll be prevented from overwriting that. In other words, I'll always send a cached version. That might be helpful if, for example, you've got hand-tuned PDFs and you don't want to accidentally overwrite them.

[00:11:19.120] All right, let's get into the code a little bit more. I'm going to skip past the really good stuff and jump into the boring parts so that we have them as context. Here's the default path. And it is going to send me the readme from the project-- from the project repo if I don't specify a path. And then we have a couple of different endpoints that we support. We'll come back to this first one. For now, let's start with the more normal one, which is just giving us a file name. So we can see we start by figuring out what the physical file name should be called. And assuming that that exists-- sorry, I've confused myself. So this is the caching or the optimization that I mentioned, sending the existing file. This file exists is where the optimization is that regenerates the file if the source or document for the HTML generator has changed. Again, this is a short talk, so I'm not going to go into all the nuances of this JavaScript code. It's pretty far from an Emacs-related thing. So with that said, then, the rest of this program is really mostly just handling the different errors: "I didn't understand that type." "I don't know the document." "I failed." Otherwise, there's the caching.

[00:13:06.480] And here's really where things get interesting, where we've generated some Elisp, and then we're calling Emacs with that Elisp. If everything works, we'll send the file. If it doesn't, we'll send the 500. And we've already seen the 500, so we know that works. All right, let's get to the interesting part. Sorry, one more footnote.

[00:13:37.320] There is a capability built in that will allow us to execute an Org block. Let's see if that's working in our local. I'll remind myself how to do it. It's run. I think it's called test. And that's returning a 500. I'm suspecting that's running because I'm running in command instead of bash. Oh, yeah, so the failure is happening after I generate the Elisp. I'm pretty confident that is what the actual problem is. If we have time, I'll jump back over there and relaunch it in mingw bash. And we can see it actually work. But this works pretty well for me on my work laptop. I didn't have to make any changes to it. So I have a fairly high amount of confidence, at least in trivial cases, this works pretty well.

[00:14:32.400] All right, so what I actually wanted to talk about today-- and I'm going to be kind of hand-waving around this ES5 class that I've got and kind of the way that works. Hopefully, this will be pretty familiar to you if you are a JavaScript programmer. The interesting stuff comes when we want to build some Lisp. Here, you can see that I really don't have a whole lot of code around formatting LISP. You can see that I've special-cased whether the arguments that were passed happen to be a function. If they are, I'm going to call that function. And then the result will be formatted as Lisp. So this would be a recursive call here. Otherwise, I'm just going to return the arguments. Sorry, otherwise, I will slap a pair of parentheses around the result of walking that list if I get... formatting each element of the list of arguments that this formatLisp process calls and separating them with spaces. So in short form, this program walks through a list. If the list it receives is a function, it calls that function. Once that's handled or otherwise, we simply walk the list, taking the arguments, concatenating them on strings, and finally, wrap the results in parentheses. So what I didn't mention there but might be obvious is if I have a nested list, the inner list will be subjected to the same treatment. So this is a recursive sort of algorithm.

[00:16:43.000] All right, so now when I go to export, actually, in the interest of time, I'm going to avoid walking through that piece of code and let's focus instead on the more interesting part of how that Lisp gets encoded. So coming back to the PDF is a good example here, because it's got a special case. You can see I've specified this exportFun or export function. That's a property none of these other types have. And you can see it contains some Elisp telling us how to call the export for it. Let's go see how that's used. At the very end of what I just skipped over, the detailed "how the Org export process works," you'll see that I am ending with a step to call the export function. Here, I look to see whether I have an export function property. If I do, I call that function. And if I don't, I build this list with the default org-export-to-file function using the filename and an output filename. So this, hopefully, is pretty familiar to anybody that's manually messed around with calling org-export-to-file. If it isn't, you can pretty well trust me for it. There's nothing very special going on. This looks rather like... Poor example there. Let's go back to our markdown. And there, we can see-- Can you hear me? Oh, I'm not showing my face. Damn it. OK, I'll make the announcement. You won't see my face quite yet. We are about to get started. Well, we actually just got started on dev with the talk by RMS. So if you want to hop over to watch the talk by RMS, feel free to do so. Otherwise, we will be continuing on Gen with Corwin to finish his talk and have a Q&A. Corwin, you can feel free to go now. And for those sticking around, I'm just going to keep pressing on with this.

[00:19:25.040] In fact, I'm going to dive back into the part that I skipped here, which is the rest of how this export functionality works. So just to make sure the dot is tied together, the core of how this program works is generating some Elisp and then passing it to Emacs in batch mode. So if that wasn't perfectly clear, that's really what's going on with this program. The rest of the implementation is just a way to do that or certain features that are supported in that generated Elisp, if you will. So this is, you could say, the minimum implementation I could come up with to create a web server for my local Org documents. And I will also interrupt myself to just pull up the Etherpad real quick. Actually, if somebody is listening and can share a link to that, I closed my browser window with my links in it. But sure, I'm happy to take questions at any point, Leo, if there are any questions for me. Are you hanging out with me, instead of watching RMS? You can go. I'm teasing. have both streams open. It's fine. And right now, it's not the Q&A with RMS. It's just the presentation. So feel free to hang out a little longer if you just want the live stuff. Don't worry about it. You're fine. if you were hoping for a quick, succinct talk. I happen to know I was going to be opposite RMS, so I awarded myself the liberty of rambling. So if you do have a question, something that I alluded to and haven't come back to yet, you should, by all means, prompt me. I'm just giving you a little heads up. I might need to go help at some point of dev. So if I need to do so, I will let you know right now inside the BBB room, and you'll be on your own to manage the chat. And you can just talk backstage to us to manage what we do with the stream, OK? I've got my pad up now. Thank you, ??. And I'm sorry about butchering your name there. And yep, I've got my chat open. And I think I'm pretty well set to self-manage. Oh, I don't have a camera on. So you can't see me giving you the thumbs up.

[00:22:09.860] All right, so let's just walk through, because it's sort of an interesting code. Let's just take a look real quick at how we generated our Elisp here, because it is-- there we go. It is a little bit interesting. So here is the method. So I didn't get into detail on this. But there's an ES5 class that represents an Org mode document. It has the static debug property that, as you might imagine, can be overridden by that debug setting we looked at in the defaults. We also have a static variable that-- a static property that does nothing more than getting the path to Emacs out of those defaults. Similarly, we have a class method to spawn out an Emacs, as I mentioned, in batch mode, eval-ing some arbitrary Lisp that's passed in. All right, so the type-- this is where things start to get interesting. So this is an implementation detail, but-- that it's written as a static method. But essentially, what's going on here is looking up from that type list to try to find a type that's passed in, and that's returning one of these blocks. Let's say I requested HTML, which would be the default. Then I'm going to get this set of properties back. All right. Essentially, this program generates a program or a little block of executable elisp. However, in some cases, where if the load-path has been customized in that type block, or I think that's the only case I supported. There was another complexity I removed. So in that case, then I can simply replace that program with a let. Either way, I'm going to have everything I generate be encapsulated in a single block. The-- then I'm calling that formatLisp process that we talked about, appending to that-- or inserting into, you could say, the outer scope. And we start by finding the file. We then load any libraries that might be needed. In some cases, the type might not have any external libraries. So we just-- so that's a no-op. And then finally, we're going to execute that logic I mentioned before about selecting either the default org-export-to-file, or else whatever Elisp we've staged for exporting that particular file type. And again, in the case of PDF, there's a special function that's used to trigger that export. Or you may be aware that that's a little more complicated. There's intermediate forms there. All right. So just reminding myself if there's anything else I have to cover on background. And I think that pretty well covers the basics.

[00:26:07.440] All right, let's look at that source block execute. This is the other use of the format list function. So here, rather than looking at the type and passing that through our Org export method, and then that type is used to get the list that we want to create. In the case of source block execute, we're kind of rolling it a lot more by hand. So this gives us a good chance to sort of unwind how that list looks when it's staged as JavaScript data. So here again, I wrap everything in a progn. I start by preventing an interactive prompt for the Babel execution. And then we load languages. This relates to another piece of our configuration where we've specified a set of languages that it's OK to execute. So if that type isn't in this list, then we won't be able to execute it in line through our trivial little web server. All right. With that done, then, loading the selected language, we then once again open the file. And we're-- whoops. Let-bind a return value, which is calculated by using Org source block execute [org-sbe] on the name of the block that's given. And then we use a temp buffer to write that out to a temporary file. This is actually a little clumsy, but I haven't put the effort in to have this written out to the standard output cleanly instead of using a temp file. So under-- this is another example of where it may not be production-- well, it definitely is not production-worthy code in that under heavy load, this would certainly break with collisions on the Babel file, the name of the Babel file. In any case, once we've staged up our Elisp, which is-- this is basically variable interpolation, then we just call Emacs on that. And if we look down to where that's called, you can see that the Org Babel filename calculated here. I'm just lost in my code. Uh-oh means, oh, I need to intervene. What is going on? Carry on, please. Thank you. All right, so then-- so you can see we get-- we send the Babel file here, which is calculated manually. A bit sloppy there, since I have essentially the same-- I have two different places where I'm calculating the Org doc file in two different ways. Have I encouraged you to write your own yet? Or send patches. All right, so that's pretty much the nuts and bolts of this program. Let's go back to just seeing if we can't make it run. All right. All right, well, I apologize for not having taken the time to stage my demo this morning. I'm going to try to make it better for you. But apparently, it's going to be non-trivial to make the program work. Let's just-- before I completely give up, let's go ahead and try our Babel execute. And that, too, is failing. So there's something unhappy in my local world. There it goes. But in any case, let's go ahead and just take a look at that. Let's see. Control Enter. Let's take a look at that generated .el and compare it to-- whoa-- and compare it to-- I'm just going to format this manually, because I've forgotten my key bindings to auto-format it. There we go. All right. So now we can see, as promised, there's really nothing going on here other than the interpolation of the variables in. We're inserting-- we're using an insert and write file method, which is, again, rather sloppy, to generate the text file. All right. Let's come back to our documentation and see if we can put a bow on the project.

[00:32:39.760] So I hope I've convinced you that this was actually rather easy to do. The entirety of my index.js file is 262 lines, and that includes a good 40 of whitespace and configuration. It has only one dependency, the Express, which really builds the web server. Any language you'd rather implement this in will have a similar capability for building some type of trivial web server. And I think you may find-- I certainly found that a large portion of the code base is really making the errors meaningful, in that, in some cases, sending an appropriate HTTP status based on what happened. In other cases-- let's see if I've got an explicit throw left in here-- in other cases, just trapping different types of failure conditions. I'm going to look at my pad, and I do see a question here. So let me jump in here.

[00:33:58.880] Are you finished with your presentation? But yes, let's say yes to that. you are in charge of the room. We are going to open up the room so that if people have questions watching right now on Gen, feel free to come in. And there was something else I needed to say. Yes, Corwin, if there's any problem, whisper to us on Mumble. So you might want to unmute Mumble and be able to listen to us over there. If I unmute, Mumble is going to bleed through. Well, if you have any problem, type in #emacsconf-org channel, and we'll be with you, OK? But I don't anticipate having any problems. I'll put something in -org when I run out of steam here. How's that? So I will have to leave the room, though. I'm leaving the recording going so that we have your Q&A. And whenever you're available-- Good luck, Corwin. All right, and if you're still with me, well, thanks. I appreciate that. I did offer to be opposite RMS. And I'm in no way offended if people do want to jump over, especially as that starts to shift over to Q&A. I'm taking Leo's leaving as a pretty good indication that that's happening now-ish. So I totally understand if folks are more excited to do that. Meanwhile, let me just jump over to the question that I received. I'll show the pad here so that I save myself reading the question out. But I'll paraphrase it.

[00:35:48.940] Why am I not running the web server in Emacs? That would be a great way to do it. I chose to build it in Node.js because that was trivially easy for me.

[00:36:22.140] And then finally, am I using org-info-js? No, I learned about this essentially at this conference. So that's something I'll be learning more about. And it could well influence this project. All right, and thanks for the questions. All right, I'm going to slow my roll just a little bit here because I think I kind of have all the time in the world. I will be wrapping up within about 15 or 20 minutes at the latest just to avoid stressing out my fellow organizers, especially Leo and Sacha that have the bulk of the heavy lifting this year, and Amin, and really, thanks all to everybody. God, the nicest part of doing my own talk is that I get to say that. It's just so much fun to contribute to EmacsConf.

[00:37:35.460] And if you're at all interested, there's plenty of completely backstage, behind the curtain role. Behind the curtain roles doesn't mean you have to be somebody that likes talking or being on webcam. Sorry that my camera isn't working this year. I spent quite a while fussing with that and lost all my time to get my prerec working. All right, so trying to think where I can take us without my demo working. I was really hoping to show the Org Babel piece. That's really fun.

[00:38:15.580] So let me just mention briefly how I'm using this at work. So at work, I'll have some type of Org document. And usually, it's a project. So the title of the document is My Project. And then I'll have a requirements section. And I'll have a meeting notes section. That's probably the key thing. And then as the project goes on, I'll start having-- I'm a solutions architect. So my job is formalizing design in large part. So then I'll have a design documents section. And this is where I'll be doing a lot of my work. So I'll start out saying-- And maybe Bob is a subject matter expert whose buy-in I need to have on how we're going to do the high-level design. Maybe a lead engineer or a dev manager or something like that. All right, as my work goes on, then this will start getting into more detail. And things of this nature. As things get further and further, I'll actually have documentation that I'm adding in here. Oh, I see. It's a big mess. All right, well, we'll just reuse this. So I can insert those all in line. And now for the fun part, let's see if the most trivial case is working here. No. All right, completely broken. Let me drag. All right, well, apologies again for the poor quality of my demo today. And let me just look real quick at my Etherpad once more. And I'll glance at BBB to see if there's anybody jumping in with questions. And then I'll go back to IRC and look for questions there. OK, and I don't see any additional questions on the pad. I'm just going to scan IRC real quick. I suspect that the TreeSitter comment isn't for me. All right, and I'm not seeing a lot of questions there. So I'm just going to vamp for just a minute or two.

[00:42:04.340] As I mentioned, I'm a conference volunteer. This is my third year volunteering with the conference. And probably if you take one thing away from my talk, it should be I really like volunteering for the conference. It's fun. It makes me feel sort of close to the pulse. And it gives me a chance to just interact with people that have very different perspectives on Emacs, which is something that I really value a lot. Emacs, like anything else sort of in the internet world, has a real echo chamber factor. If you do or don't like use-package, you probably interact with a lot of people that feel the same way about that. And so I really recommend volunteering for EmacsConf as a way to sort of mix it up and get to know people that may not use Emacs the same way that you do.

[00:43:08.380] Or perhaps more on topic, though, the log line for this talk is it's really quite easy to build a program that uses Emacs in a pipeline capability. I think there's a ton of opportunity in this space. This particular example is just a trivial web server written using Node.js. But as was pointed out, we could have used elnode as a web server and done the entire thing within Emacs Lisp. Or really, almost any technology would get us this capability. From an implementation standpoint, I had a lot of fun building this trivial little Elisp parser, and I'm rather pleased with the fact that the entirety of that-- the entire algorithm for turning JavaScript or JSON data, we could say, into Elisp is really a one-liner, albeit a nasty one-liner. That was pretty cool to discover how simple that was. So in my mind, that opens up a lot of possibility. If it's this easy in JavaScript, I wouldn't expect it to be hard, any more difficult in your favorite language. Glance one more time to see if there happen to be any other questions. And not seeing any, I'm going to go ahead and start wrapping up my chat now. It will take me a couple of minutes to do that. So if you do have any other questions that you want to drop into the pad or any comments, you're more than welcome to hit me with those as I coordinate closing this chat, this talk, with the organizer team.

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

CategoryOrgMode