Back to the schedule
Previous: Creating technical API documentation and presentations using org-babel, restclient, and org-treeslide
Next: The use of Org mode syntax outside of GNU/Emacs
Org as an executable format
Tom Gillespie
Q&A: live Q&A or IRC
Duration: 7:09
If you have questions and the speaker has not indicated public contact information on this page, please feel free to e-mail us at emacsconf-submit@gnu.org and we'll forward your question to the speaker.
Talk
Q&A
Description
Org mode is known for its flexibility, power, and staggeringly diverse number of workflows, users, and use cases.
This talk explores just how far we can push the boundaries of the sane and sensible with regard to Org workflows.
In particular it will discuss shebang blocks, and elvs: two parts of a complete system for creating executable Org files.
Org syntax does not support shebang lines. However, it turns out that Org syntax enables something even better — shebang blocks.
Org is also (supposedly) not an executable file format. However, by combining a shebang block with a Org babel source block, and eval local variables (elvs) Org becomes a multi-language executable format.
In this talk we introduce shebang blocks and elvs as a two part system that transforms Org files into executable documents that can run on any recent version of Emacs.
These ideas are implemented in https://github.com/tgbugs/orgstrap/blob/master/README.org and https://github.com/tgbugs/orgstrap/blob/master/shebang.org, and orgstrap.el is available as a package on MELPA and can be installed via M-x install-package orgstrap.
The talk will open with a demo of how to create an executable Org file using the orgstrap machinery.
We then discuss security considerations, and show example use cases.
Finally the talk will cover the details and development of the portable shebang block for Org mode that works on a wide variety of systems and shells, and on the development of a formal specification and a reference implementation for using Org source blocks to transform Org files from plain text documents with a bit of markup into self describing computational documents, or interactive applications.
Discussion
IRC nick: tgbugs
- what prompted you to create orgstrap?
- Tom Gillespie: https://github.com/tgbugs/orgstrap/blob/master/README.org#background-file-local-variables-and-checksums
- yeah, usually this kind of stuff starts out as a minor annoyance
- is there a way to choose which blocks are going to be executed?
- What are some practical specific use cases that you had in mind?
- Tom Gillespie: https://github.com/SciCrunch/sparc-curation/blob/master/docs/sckan/welcome.org
- Tom Gillespie: https://github.com/tgbugs/orgstrap/blob/master/README.org#use-cases
- i'll for sure incorporate it in my company and research, so both corporate and academic environments will benefit from your work. i already use Org for documenting a lot of stuff so its just a natural next step
- I have experimented with dblocks in an org-mode buffer to do the equivalent of javascript in a browser. Does that make sense?
From IRC:
- This is supercool !!
- That is absolutely wild
- is the hash not just security theater?
- tgbugs: you need it to enable non-theater workflows
- anyone who could write the org-file could update the hash as well, no?
- tgbugs: you need it to enable non-theater workflows
- I don't understand why he needs to talk about powershell, more than the other shell.
- tgbugs: very late response, but the reason I talked about powershell more than the others is because it is the most different and required some explainiation, I also was
about that
- that sounds that is a good reason to not include it xD . you can add fish in the talk and it will be bettter.
- tgbugs: very late response, but the reason I talked about powershell more than the others is because it is the most different and required some explainiation, I also was
Outline
- 5-10 minutes:
A demo of adding the orgstrap block and elvs, adding a shebang block, and then running an org file.
Transcript
Hello all! Welcome to EmacsConf 2021. I'm Tom Gillespie. Thank you to the organizers for all your hard work, and for inviting me to give this short talk on "Org as an executable format". The links to the talk page, the GitHub page for the project, and the package on MELPA are listed on the right.
[00:00:20.160] Let's start with one of the motivating use cases for executable Org files. Many users keep global configuration for Org in an init.el file, which works for many workflows. However, for reproducible research, this is a challenge because if an Org file is dissociated from the init.el file, then often it will no longer function as expected.
[00:00:44.640] One potential solution to this problem is to be able to include all of the global configuration for Emacs and the environment in the Org file itself, in which case when you go to reuse the Org file, it will work as expected.
[00:00:58.640] What does an executable Org file look like in action? Here's a demo of an executable Org file running in Bash, Dash, Zsh, and PowerShell. So, we are currently in Bash, and we can run our demo, and it will print some stuff, and wait for input. We can also run it in Dash, which is the default for Debian and derivatives. Same program works as expected. Zsh also. And lastly PowerShell, if we try to run demo.org itself, we see (that) we get an error because PowerShell cares about file extensions, so, if we symlink to ps1, then it works as expected, and there are ways to alias this, so that you can run it as a program without the ps1 extension.
[00:01:58.640] So, how does this work? There are three components to an executable Org file that all need to be present in order for this to work. Starting from the top of the file, we have a shebang block. Next we have an Org Babel block written in Emacs Lisp, which is what we actually saw executing, and then there are some eval local variables or Elvs that are involved in making this actually executable.
[00:02:26.800] Let's start with the shebang block. Org syntax does not have support for shebang lines. However, it supports the shebang block. This is because Org comments, blocks, keywords, etc. that start with the sharp sign have the same syntax as comments in POSIX and PowerShell. This block is in fact valid Bash, Dash, Zsh, PowerShell, and maybe some other shells as well. In essence what it does is, perform some setup to avoid polluting standard output, and then it runs Emacs to load the file itself.
[00:03:08.959] The Elisp that is passed on the command line is explicated over here on the right, and in essence what it does is, to keep the startup time minimal and as low as possible, it loads the absolute bare minimum needed for Babel, and then it calls hack-local-variables triggering the eval-local-variables. What do the eval-local-variables do, and how those work? The essence of the approach is to use org-confirm-babel-evaluate to allow Babel execution. We can't set it to nil because that is an arbitrary code execution vector, which we don't want if we're sharing files.
[00:03:48.000] Instead what we do is, we use the fact that it can be a function, and we normalize the block of code, we checksum it, and then we check that it matches this checksum up here at the top of the file, and then sort of inside there, inside of org-sbe, which is Org source block evaluate, we call the block.
[00:04:08.959] The actual implementation of this is somewhat more complicated. However, it's small enough to fit in the local variables at the end of the file. One thing to note is that if you are using PowerShell, PowerShell parses the whole file which means that for any normal Org content, you need to put it in a multi-line PowerShell comment, and close it.
[00:04:31.199] So, once we hit hack-local-variables at the end, we run the eval-local-variables block, then we enter this Elisp block, and you can write whatever you want. All the power of Org Babel is now at your fingertips in order to do what you need for this file.
[00:04:48.800] Finally, let's do a quick demo of how to use this to make your own Org files executable. Orgstrap is available on MELPA as mentioned, and it can be installed using package.el by calling package-install orgstrap. It will download, and it will install. Then you can open an existing file or a new file. In this case, let me open a file called example.org. And then orgstrap provides command called orgstrap-init. What orgstrap-init does is, it populates a file with the machinery needed to run an orgstrap block. We're just going to do a message "hello orgstrap!". If you look up at the top, you will see that the orgstrap-block-checksum will change when I save the file. This makes it much easier to author files with orgstrap blocks. And then we need one last piece of machinery, which is the shebang block. I am just going to steal the shebang block from this other file over here since it is available. You can also get it from shebang.org, and I have plans to add a command to insert this into the file directly, which may actually be done by the time this video is actually posted and visible.
[00:06:19.520] There's one last step, which is that we need to run dired in order to… There we go. So, we use Shift m, capital m, in order to make our example file executable, and then if we come back to here, we see that example.org is now executable, and we can run it. "hello orgstrap!", and we're done.
[00:06:50.560] So, that's the basic workflow for getting orgstrap files to be executable. I will be around to answer questions live, and I will be also available in the #emacsconf IRC channel all day. I hope you have found this useful, and thank you very much for watching. captions by bhavin192 (Bhavin Gandhi)
Back to the schedule
Previous: Creating technical API documentation and presentations using org-babel, restclient, and org-treeslide
Next: The use of Org mode syntax outside of GNU/Emacs