Org workflows for developers

George Mauer (he/him/they/ze, IRC: gmauer, gmauer+emacsconf@gmail.com)

In this talk, George Mauer demonstrates several ways that Org Mode can help people explore and develop code. Afterwards, he will handle questions via BigBlueButton.

The following image shows where the talk is in the schedule for Sun 2022-12-04. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 21-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-workflows
Discuss on IRC: #emacsconf-gen
Status: TO_INDEX_QA

Talk

00:00.000 Introduction
00:53.960 The future
02:30.200 Org development workflows
03:15.680 Taking notes
04:54.600 org-capture templates
06:10.680 Building up a dashboard
06:49.160 org-store-links
07:45.680 Formatting
08:21.480 Pasting code
08:52.200 Git
10:04.960 async-shell-command
11:29.040 Literate programming and tangling
13:47.840 Noweb
14:36.400 Running commands
16:04.480 Buttons
16:43.600 Workspaces
18:04.800 dash
18:36.000 Header arguments
19:29.920 Conclusion

Q&A

00:00.000 Introduction
01:23.774 Q1 - Does it become unwieldy due to the interaction of the edit org-source to use org-mode and the virtual linear programming as the project becomes larger?
02:38.982 Q2 - I want to take a look at the files used in your demo, are they somewhere online?
03:16.080 Digression - some explanations about the background dinosaur 
04:54.960 Information about org-entry-get
05:49.640 Are workflows as they are in your life closely tied to particular projects or are they general workflows? - Long discussion about the workflow!
13:32.960 Wrapping up

Description

Table of Contents

We all know org-mode is great but much of the discussion often focuses on the agendas, todo lists, and project planning. These are all valuable. yet rarely do we talk about workflows that do work, not just plan it. Inspired by literate programming ideas, this talk will demonstrate a grab-bag of workflows developed over the years that are of use not only for planning, tracking, note keeping, and ops work, but in actual day-to-day enterprise software development.

Discussion

Notes

• Start with all the things from howardism literate devops articles - its all great and is what got me started http://howardism.org/Technical/Emacs/literate-devops.html (he also just did a talk on eshell)
• Learning request.el, dash.el, cl-loop facility, and s.el reasonably well is a pretty big 

• there is also https://gitlab.com/mtekman/org-tanglesync.el

• Q: are arduino blocks included in org-mode?

  • A: I did add arduino-mode I think thats the only thing I did so maybe its from there
• A: This is the bit that I'm particularly proud of btw - the ability to do dynamic scoping of variables that is driven by properties in the org outline
• Q: how did you insert the texts (not caption)?
  • A: in kdenlive

Feedback

• liked it a lot
• awesome talk gmauer`
• It was fun captioning it.

Transcript

[00:00:00.000] Hello. Welcome to my first ever EmacsConf talk. This is really exciting for me. I've done lots of conferences, but rarely ones this technical and this nerdy. I also feel like I have something interesting to share. I come to Emacs relatively late in my career, only about six years ago, but I've been absolutely amazed at the innovation and commitment of the community to do things their own way. Oftentimes, these become things that are not readily available anywhere else. So, as I've been using Emacs (and Org mode specifically) a great deal in my day-to-day workflows, I've been leaning more and more into some of these tips and tricks. I find that there is almost every day that I discover some useful tweak that can make my development better. I want to share them with you now.

[00:00:53.960] Who are you? Oh, nice. How good. No, I'm you from, like, a month from now. Look, you know how these talks are pre-recorded, and you know how you've spent the last two years criticizing conference speakers for trying to do the same old thing and not creatively adapting to the online conference medium? Well, you are recording this back in November. I'm in December when everyone is watching this for the first time. That is something we can do now. Cool! And I see you still haven't figured out how to remove backgrounds with OBS. I have to get a plugin or something. So, yes, it's kind of a gimmick, but I also have a cool point. You know, how you just said that you discover something new every day? Well, your talk isn't that long, and I found a bunch of cool new workflow synths. I'm starting a new job in the intervening time. So, I have more stuff I want to add. once we set the ground rules, the audience might have some of their own suggestions. That is a good idea. Okay, go away now. aren't you gonna explain the dino? You think a dinosaur built out of boxes and old dishwasher parts is the weirdest background thing we'll see? [Both making dinosaur roaring sound: ROAAAAR!] Okay, bye now.

[00:02:30.200] Hey everyone, you heard the idea. This is going to be a thinly-veiled attempt to show you stuff about Emacs and Org mode, specifically, that I think is super cool and immediately useful while you're doing development. Let's define the scope of Org development workflow as something specific you do with Org mode that helps in certain common development related activities. Now, tie-dye me from the future said he's got some more ideas beyond what I'm presenting here. I'm sure many of you have ideas as well. So, we're going to share a collaborative document, and let's all as we're listening to this be talking, and chatting, and entering our own ideas and workflows, so that we can learn and improve together.

[00:03:15.680] And now with that, let's begin. I've got a ton of ground to cover, and I want to start by talking about note-taking. Shrink down! [transition] Note-taking is incredibly important. We can't keep all this stuff in our heads. So, for example, I find myself with the need to learn about the solid project. This right here is the solid project, and I want to play around with it. I am going to start by creating a note for it. Now, one of the things that I want to do is explore one of their tutorials. That's the site I just saw. I can go ahead and create a note for myself. Right, "Solid React Example", and maybe a set of stuff ending on there. I'm going to clone this project, which I've already done, and I can pull it up right here. So, I can pull it up right here, and I can now start to explore it. So, for example, this code base sounds…, it seems interesting. I'm going to want to store a link to this in my code. I'm going to run org-store-link, and I can come in here and say, let's explore structure. Local link, and here I'm going to put that right there. Now, at any given time I can come into this note and be thrown right into the structure.

[00:04:54.600] I want to go and now start investigating the code, but before doing that I'm going to take an extra step and customize the Org capture system. I'm going to create a playground node here, where I can do whatever. Now, what does this template do? Well, it's just going to create a new template. And whenever I hit the s key, it is going to go ahead and add a new heading to which I will enter, and it's going to grab a link to wherever I'm pointing at, and any highlighted code will also be inserted into a source block, and eventually, drop my cursor where I can work on it. So, we can grab our template, and the one thing I'm going to need to add it here is to say what file this goes to. I'm going to copy the name of this file, and put it right in there. I'm going to go ahead now run this template.

[00:06:10.680] Now, we can explore our code. For example, I can look in the server, and say, "Oh yeah, this slide looks interesting. Go ahead and capture that." There you see our template. You see, yeah, this is Next.js app, and you can see it got added right in here right next to my other code. So, that's interesting. I can always go ahead and click that link, and get thrown directly to where in the code I was. I'm kind of building up my own dashboard as I explore this project of interesting points within the project.

[00:06:49.160] One of the things I noticed here by looking at the file structure is that there is an area for certificates. That's a little unusual, so we'll make a note of that by again running org-store-link. This comes with certificates, so we'll put that there. One of the good standbys is, just to use our regular shell commands. So, we will go ahead and say, the default directory for this is our project. And we can go ahead and say, cat certificates/localhost.key, and then we'll output the first five lines of it, just to make sure it's a regular certificate.

[00:07:45.848] Now, notice this got broken up a little bit. This is due to Emacs auto formatting. We can come in here, and tell it to format it as code, which will be the same as this block right here. Now, there are other options available. If, for example, we don't want [it] to be a shell block, we wanted a Python block for some reason, we do :wrap src python, and execute that, and it's now wrapped as a Python block, but I like it as a shell.

[00:08:21.480] Let's, for example, go down into pages here and look at this document file. We're saying, "Okay. Well, this looks interesting maybe highlight that," and we'll go ahead and capture that template and say, grab all this code and paste it in here. Now, there is a bug at the moment, where if you highlight more than one lines of code, the link will not work, and that honestly might be something I look into fixing.

[00:08:54.540] One of the things that might be useful here would be to check out how this file has evolved over time. To do that, I'm going to use Magit. I'll pull up a log. Look, there's only a single change. I'm going to run a command called orgit-store-link, and now I can come in here and say, "It's only changed once." Go ahead and insert that link. Now, this file… the arguments here are kind of weird, and in fact, if I click this, it will actually go to the full log of that branch. However, we can fix that pretty easily. Grab the path of our file, and this right here is really just the arguments that are passed into the log command. So, here we go, we put that in there, and there we go. We get the full file history.

[00:10:04.960] Now, I want to actually build the program. So, "Build the app." Now, I could of course run it as a shell, right. npm ci. The problem with that is that Emacs is single-threaded. So, if I were to do that, the entire time while it was running, it would be locking out my Emacs. Additionally, I might not actually want all that scroll--npm ci produces a lot of it-- actually in my document. So instead, what we could do is use an Emacs Lisp function, and it's called async-shell-command. And when you run something in async-shell-command, it's going to a comint buffer with a process attached to it, and run it in there. I will need to set the directory here first, and since, again, this is going to be opening up in a new buffer. I don't need to see that. I'm going to run it. And what's going to happen is this is actually not going to work. And it doesn't work, Not for any particular reason I can control. It's unfortunately that the repo is broken, but that is a totally valid result of our investigation.

[00:11:29.040] One of the things that I really love to do with Org mode is to actually use it for literate programming, because Org mode has a pretty capable code generation facility built into it. It's called tangling. So, if I go ahead and take my document... This is for a little Arduino project, where I was figuring out to spin things around using an old Roomba motor. I can go ahead and write a script like this, and then notice, I use the tangle variable that is just going to determine where that file gets written when we call the command org-babel-tangle. So, if I go ahead and run this, you can see down in the minibuffer, it's going to write to /tmp/go-batsy-playground/go-batsy-playground.ino. That's where this right here would write, and then I could run commands on it. Then I want to start being able to use this to build out a program. I'm going ahead and writing in prose and interspersing it with code. So, it's the inverse of code, in which you intersperse comments, [here] you write prose, and then you intersperse code where as needed. Tangle is implicitly defined up at the higher level in this property block right here, which I will talk about in a little bit. But if you want to see what properties are available at any given time, you can hit org-babel-view-src-block-info right there, and you can see that tangle is enabled. All of these blocks have the exact same tangle. If I run and see what it is, it's just going to write to this directory to go-batsy.ino. org-babel-tangle is going to go ahead and tangle all these source code blocks, and I can go ahead and look at my file and here it is. This is the full Arduino file that was generated from there. I start writing code here, and I'm basically doing it in a prose way. As I'm thinking about it, I write down what I'm going to do. Now these braces, we haven't seen these before.

[00:13:49.240] This is an aspect of Org called noweb, which again is not too much of a templating system too. But it does one thing, which is insert code, which turns out to be enough. So, this right here basically says, take that block with that exact name and just insert here. If you want to see exactly what a block expands to, you're going to come in here. You're going to run org-babel-expand-src-block, and there we go. That's what this block expands to. That's what all those places of this little bits and pieces expand to. So, that becomes really useful, and notice basically we just take these little blocks that are not going to be tangled directly but it will be in this other block. And we turn off their tangling.

[00:14:38.074] Now that you have some sort of tangling, you want to be able to interact with those files that are written to that directory. So, right here I have an area where I can do things like run a compiler. Now what does that compiler do? Well, this right here references a source code block that appears in another Org file. And I find that when doing these sort of things it can be useful to have a little utility Org directory. So, here it is, org/ci.org. This is just part of my repo. We open up, and here we go. I have a compile function. Basically, it's doing some stuff to clean up things correctly. But then using that same async-shell-command to open things up and a new buffer. In this case, named after whatever heading it was under. And then we're going to go ahead and inside of it run Arduino CLI command to compile, and pass that into watchexec. which is a little Rust program that watches inode[??] files for any changes, and when they detect them, we will run this. If I were to, for example, add a line here, and now run org-babel-tangle, you can see watchexec immediately picks it up and restarts it.

[00:16:04.480] Now it's kind of a pain to remember to run org-babel-tangle all the time. So, I can come here and click this button. It asks me to execute it there. And what does that do? Here you go. It's just a very simple hyperlink, but to the Elisp protocol. The Elisp protocol just adds a hook that says, whenever a document is saved, run org-babel-tangle. And now that I've run that, I can go ahead, come in here, and delete that. And look at that. It tangles automatically for me. Because I don't want to actually have this

[00:16:45.040] playground script tangle to my real file, I need this concept of some sort of workspace directory. And a workspace directory what I really want is a variable that is tied to where in my document hierarchy this appears. I want a dynamically scoped variable that's scoped to my document. And you can do that. For example, in this case, I have in my properties a key value declared workspace-directory, [it] goes into a temp directory. And here, by running org-get-entry starting at the current point, find workspace-directory with a second parameter 1. You can see down in the minibuffer, goes to /tmp/go-batsy-playground/. This right here is going to override the workspace-directory at the top level, which is dot. Dot means here. That's what makes sure that the rest of these tangle to that go-batsy file right relevant to here. And that does mean that we need a little bit more complex thing here. So, we're saying go ahead and org-entry-get the workspace-directory.

[00:18:04.800] If anyone hasn't seen this syntax, this dash arrow is from the dash.el library, which is basically a big library of all the utility functions that you wish Emacs Lisp had. They're well-named. I highly, highly recommend it. This is the threading operator. So, we're just basically taking it, getting the workspace-directory, if it happens to be dot, then we're just going to return the current directory, otherwise whatever directory said.

[00:18:36.000] And then I want to just take a moment and look at the rest of this structure. So, workspace-directory we talked about. header-args if you noticed, none of my code blocks for the most part have any header arguments. You can drop the header-args property, which is going to be header arguments that are added automatically to all source code blocks under this heading. header-args+. Well, sometimes you don't want to type… you have a bunch of args, you don't want to type them out in this one big line. So, you basically are adding a new header-arg to the existing list of header. And then you can have header-args that are specific to certain languages, like, for example, this default-directory var is going to be set for all Emacs Lisps. And for all Arduinos, evaluation will be disabled, and tangling will be automatically enabled.

[00:19:32.440] These are just some of the workflows that become useful when you're actually doing the coding. Me from the six months from now. thought the pacing was all over the place. going to be filling in those gaps in the Etherpad. as I worked on it, didn't I? Kdenlive is pretty cool. But yeah, I wanted to take a shot at something different, and I figured if anyone can appreciate trying something different, it's EmacsConf, right? I hope people found it useful. Oh, I should tell you about the coming Orca war.

Captioner: bhavin192

Questions or comments? Please e-mail gmauer+emacsconf@gmail.com

CategoryOrgMode CategoryCoding