This Year in Org

Timothy (he/him, IRC: tecosaur)

00:00.000 Introduction
00:32.080 Project housekeeping
01:08.800 Continuous integration
02:04.680 Funding contributors
03:32.560 New features
03:58.640 An assortment of export improvements
04:36.520 A collection of babel improvements
05:12.280 A multitude of general org-mode improvements
06:04.160 Citations
07:31.600 Quality of life improvements
07:48.320 Org fold
09:02.480 Org element cache
10:07.360 Org persist
11:02.720 More careful resource downloading
11:40.200 Bug fixes
12:15.800 Asynchronous session evaluation
12:42.800 Nicer tangle mode syntax
13:18.280 A flourishing ecosystem
13:52.600 Org-modern
14:10.120 citeproc-org
15:44.040 Continuing work on the Org format
17:06.320 Mailing list management
17:41.920 Further engraving

Description

You've all been avid readers of the (somewhat irregular) "This Month in Org", now you can be an avid listener of a special edition exclusive to EmacsConf: "This Year in Org", a quick rundown of major developments in Org over the past year, and perhaps a hint of some things lying around the corner.

Discussion

Questions and answers

• Q: Not a question, but just a great thanks for "This month in org" which helps us get awareness about the greatness of Org!
  • A:Thanks
• Q: Does the project need other kinds of support (infrastructre, etc.) which can't be covered by donations to devs? (without detriment to supporting devs!!)
  • A: There isn't much in terms of ongoing costs (just hosting https://orgmode.org really), but donations are great for dignifying the work done, indicating the value it has to the community, motivating developers, and also helping justify/enable more time to be spent working on Org.
• Q: What is the use of parsers in other languages? Is it to make org available in other applications?
  • A:Org is being used outside Emacs (e.g. Hugo, Logseq, rendering on GitHub/GitLab/Gitea), and so it's worth trying to make sure they treat the syntax in a consistent manner. Similarly, if people build nice tools for Org outside Emacs, that's nice for us
  • Voit: Shameless plug: https://gitlab.com/publicvoit/orgdown/-/blob/master/README.org is also an idea to promote the syntax of Org mode in tools outside of Emacs. After all, everybody is getting advantages when Orgdown (syntax of Org mode; often named "org" but it's frequently mixed up with the Elisp implementation) is a rather popular syntax.
• Q: citar package is a really pleasant addition with support for org-style citations. What's your take?
  • A: Citar is great, IMO
• Q: How many hours a week do you spend contributing to org?
  • A: It varies a lot. It's also a bit difficult to say, because there have been a fair few patch sets which have been "incubated" in my config before brining them to Org mode, and so I need to detangle "time spent tinkering on Org in my config" and "time spent working on patches for Org mode". Some weeks it's ~0h, others it might be as much as ~30h. The average might sit around ~5h, but that's just a wild guess.
• Q: As a fan of emacs org-mode and Julia myself too, do you see any possiblities/wishes/plans to somehow connect org and julia evenmore (apart from ob-julia). Perhaps a julia parser of org-mode, or something like that?  Just wanna personally thank you for all the effort in org, emacs and julia you are putting throughout! I have learned quite a bit about doom emacs config from your blogs too. I feel like our setup/interest overlaps quite a bit, and its always helpful too see your work out there. Thanks again, keep up the great work!
  • A: Wait a few slides 😉 (https://github.com/tecosaur/Org.jl)
• Q: "Org", "Org-mode", "org-mode", "Org/Org-mode"? Which one for the format/notation and which one for the software proper, and then the whole thing (with org-contrib and third-party packages) vs just the repo and major mode per se?
  • A: "Org mode" for the project, "org-mode" for the major mode, "Org" for the format
• Q: How much time/week do you spend editing your doom config?
  • A: Err, too much 😆 
  • I feel that
• Q:In your doom configs, you like using variables fonts and stuff, basically a lot visuals everywhere. I feel that it makes everything sluggish. Am I doing something wrong? Maybe there are less visible tricks in your config?
  • A: It doesn't make things slow enough that I care, basically. Doom does some nice performance stuff, and I try to go for deferred loading and look for text-properties over overlays for performance in visual packages. As for varaible pitch fonts look, that's handled by Harfbuzz not elisp AFAIK (and so doesn't really affect performance).
• Q: Do you use linux or mac system? What kinds if not a secret?Q
  • Linux, OpenSUSE Tumbleweed specifically
• Q: I wonder why the export to HTML has not been modernized. Any particular reason?
  • A: It's going to take a whole lot of time an effort. I take that as being more flexible and better able to suit "modern HTML/CSS" usage. Incidentally, the HTML and Markdown backends are two things I'd like to have a look at next year (if I end up having the time).
• Q: haven't looked at citation support at all yet, any great intro articles out there?
  • I may be (am) biased, but try https://blog.tecosaur.com/tmio/2021-07-31-citations.html 🙂
  • More discussion about citations:
    • and I have found the citar package a really pleasant addition with support for org-style citations
    • Citar is great, IMO
    • Citar is indeed great. I have been meaning to switch to Citar from ivy-bibtex, but honestly I am just being lazy
    • I am not sure as I haven't used ivy-bibtex. I am half-way migrating away from org-ref but will likely keep that around for a while longer (mostly for old links and its doi-utils import functions)
    • I found citar really easy to set up and get started with. Very clear and clean entry points
• Org mode outside of Emacs: https://gitlab.com/publicvoit/orgdown/-/blob/master/doc/Tool-Support.org
  • A: Also https://orgmode.org/tools.html 🙂
• your visuals have always been charachteristic, where did you learn that? in the sense that I can reasonably guess you made something when I see the result. I was also thinking about your config or the survey website, or are they inspired by that theme?
  • A: I think that's just "themed Beamer metropolis". I do also naturally go to light themed content with a pale yellow-y background.
• Q: "Org", "Org-mode", "org-mode", "Org/Org-mode"? Which one for the format/notation and which one for the software proper, and then the whole thing (with org-contrib and third-party packages) vs just the repo and major mode per se?
  • A: "Org mode" for the project, "org-mode" for the major mode, "Org" for the format
• Q: Thank you Any plans to use tree-sitter with org? How would it relate to org-element? But if I remember correctly org syntax can not be fully expressed as tree-sitter grammer... So maybe tree-sitter is not for org?
  • i don't grasp the recent infatuation with external parsers.
    • A: They're happening, and syntax divergence is bad. and if they are used to make neat things, it's nice if we can make use of them too

Other discussions from IRC

• yeah org-modern ?
• lots of progress on the syntax doc, i just checked it out on worg
• engraved-faces is excellent.
• Again, an excellent talk.
• Both talks were great! Thanks
• good pace, slides, and clearly delivered
• "aspirational rather than descriptive" -- I will remember this for future use
• thanks tecosaur for the very nice overview and reminder of all the recent changes in org!

Transcript

[00:00:00.000] Hello. If you're listening to this talk, then you should be at least a bit interested in Org mode, which is fantastic because there have been quite a few interesting developments over the past year or so. I'm Timothy, as you may have gathered from the last talk, and I'm also quite involved with the Org project, so I'd like to go through a few of those developments over the past year or so and give you a few hints as well as to what potentially lies around the corner with Org mode.

[00:00:32.080] The starters, slightly on the more boring side but rather significant change to the project, occurred with the housekeeping or organisation. The codebase for the Org project has actually shifted over from a self-hosted Gogs instance over to Savannah, which means it's now living right alongside the Emacs codebase. This has been accompanied by the creation of a whole bunch of Org-related repos under Bastien's (Org's maintainer) personal sourceHut account. We've got the source of the website, the Org wiki Worg, as well as Org contrib.

[00:01:08.800] Another recent addition to this list of Org-related repos is the new Org mode tests--continuous integration. Now, this is rather important, because while we do recommend that all contributors actually run make tests before submitting patches to the Org project, this doesn't always happen. It can also actually be a bit harder than you expect to run the tests because there are a lot of trans-dependencies you get with Org; for instance, with all of the various Babel libraries which actually require other packages or programming language to be installed on the system. Having a single self-contained test system to actually make sure that Org can be regularly and thoroughly tested should be a great help for actually ensuring the quality of the contributions.

[00:02:04.680] The funding structure for Org has also undergone a bit of a shift. Historically, we've just directed everybody who's interested in financially supporting the Org project to the maintainer Bastien's personal GitHub sponsors and LibrePay accounts. Now, early this year, Bastion has created the Librepay Org mode team account, which means that you can actually now support the Org project as opposed to the person leading the Org project. Currently, this just distributes donations between Bastien, Ihor, and myself. However, the idea is that as the active contributors for the Org project come and go over time, the list of people on this team can be changed as seems sensible. The hope here is that it will simplify both how easy it is to actually financially support the Org project as well as how easily people contributing to the Org project can be supported. If you're interested in supporting the Org project, there's never been a better time than now to have a look at this and let anybody who might also might be interested know. Hopefully, this leads to a healthier funding structure that will scale better into the long term and thus better support the work that happens with Org.

[00:03:32.560] Now, the project itself has of course also seen quite a bit of development over the past year. We've had about 800 comments from 80 contributors. Within these comments, there's been a lot of polishing quality-of-life improvements, and also quite a few new features. Now, I haven't got nearly enough time to go through this exhaustively, so we're just going to go through a quick highlight reel.

[00:03:58.640] There's a collection of export improvements from things which affect all export backends, like including remote content and adding new things like DOI links and support for encrypted Org files, as well as a whole lot of export-backend-specific changes. For example, quite a few backends-- I've mentioned the LaTeX one here, but also others such as Texinfo--have now got rich support for various types of attributes and objects. The HTML backend has had a few things boosted up and well, if you want to see the full list, just take a look at the release notes.

[00:04:36.520] We've also seen a similar collection of improvements with the Babel backends. Once again, this is scattered-- or well, it can be split into two sets of changes. There's some which affect all of Babel, essentially. For instance, the new syntax of parsing the raw content of code blocks, or the changes with Noweb. For example, :noweb-prefix is a new option that can be used. And then there's also a collection of backend-specific changes. So ASCII graphics with PlantUML or enhanced return capabilities with the ob-python library.

[00:05:12.280] And then of course, as before, a whole collection of more changes which you can find in the release notes. Last but by no means least, there have been quite a few changes within the rest of Org. So this is, once again, far too many things to list, but it's things like improved refiling, capture templates, image preview sizing, clocktable settings, agenda tweaks, and well, a whole lot more. Yes, basically, the essence of what's here is a lot of little changes which just address particular use cases in ways that I don't think anybody's going to be seeing the impact of all of them, but I think most people should at least find one or two things which actually improve their own usage.

[00:06:04.160] Now these are the sort of assorted relatively minor improvements, but there are also some major ones. And one in particular, citations. So I think this has been, at this point, over a decade in the making, but Org finally has first-class support for citations. And I have to say, it is marvellous. You'd hope so, after the labour. I think it is. It can be said that it's actually worth the wait. I think out of the various options you've got now, (for example, the way that Pandoc and Markdown otherwise[??] do it) Org has a fantastically succinct and flexible syntax for citations, which scales really well for all sorts of different use cases. Additionally, on the backend side of things, we've now got a generalised way for handling citations which has been quite helpful for the--I think I could say rather rapid development of multiple citation backends for Org. And I think it's just fantastic, really, seeing how quickly Org has gone from having no support for citations at the start of this year to what can be described as a wonderfully rich and flexible support with, well, multiple backends for citations. I think that's something that we can really be proud of. And it's been a fantastic contribution for everybody involved in this process.

[00:07:31.600] Okay, so we've had features. There have also been a whole lot of quality of life improvements. Once again, many more than I can reasonably mention here. So I'm just going to flick it through a few of them. A few big ones though,

[00:07:48.320] Ihor is responsible for three lovely developments with Org, one of which is Org fold. So this is a generalisation of the way that content is folded in Org. And I think quite a few of you will actually underestimate how much can be folded in Org. It's not just a matter of headlines. It's headlines, code blocks, lists, environments, all sorts of things can actually be folded in Org. And the introduction of Org fold is important for two reasons. One is that it has allowed for text-property-based folding, which in Emacs versions less than 29 has a huge difference in performance, which is particularly apparent with larger files. The second significant thing about this is that it now actually provides a more general way to actually describe changes to the folding structure. So before there was direct modification of messing with overlays scattered around the Org code base. Now we have a much more well organised system where we use Org fold to say what is and isn't folded and to manage the state of all of that, which is, I think, just from a sort of design, sort of project design approach, a much better system.

[00:09:02.480] We've also got the Org element cache by Ihor. This is actually something which was discussed quite a while ago, but has somewhat stalled due to the difficulty of cache invalidation. Ihor has sunk a tremendous amount of effort into this and has improved it to the point where we've now actually been able to enable this by default. So what this basically does is it records lots of information about the structure of the Org document and allows for, well, with the appropriate modifications that Ihor has also made throughout the Org element library to use this information to speed up various operations based on the Org document syntax tree. And so this has been quite-- the improvements have been scattered all over the place, but for a good example for libraries or anybody who's wanting to quickly map over Org elements is `org-element-cache-map', which now provides a much, much faster way to map over all of the Org elements in a document.

[00:10:07.360] This also ties into the third major feature from Ihor which I'd like to mention, which is Org persist. So this provides a method of persisting values across Emacs sessions, basically saving them to a file somewhere and loading them. Now this works for Elisp values and it also works for files, which we made use of with the improved capabilities for remote files and exports. This has also been used with the `org-element-cache' data. So now, if you've got a massive Org file and you open it once, that data can be saved to with the Org element cache to Org persist, and the next time you load this file in another Emacs session, we can just start with the cached data instead of having to construct everything from scratch, which is quite nice. Once again, a change which much like the other ones, we will see more of an impact in larger files, but a very welcome one everywhere.

[00:11:02.720] Now with remote files, there's also been the beginnings of a bit of an effort with Org to improve the approach we have to safety. So in this case previously, Org would unconditionally download all the remotes of files that's all referenced. And now, it's actually going to maintain a list of sort of safe resources and prompt you when it's surprised by something, to work out whether it should just download this one resource, mark the whole domain as safe, and a few other options. We're also going to probably see a similar approach extend to, for instance, bits of Babel execution in the future.

[00:11:40.200] Okay bug fixes. It will be remiss of me not to mention that along with all of the features and quality of life improvements, there has been a huge pile of bug fixes. I think the best way to actually get a look at this would be to look at the release notes or maybe even the actual commit log, but you could also just take my word and say that there have been a whole load of them over the past year. So just yes, the code base, I think, is just gradually getting into better and better shape.

[00:12:15.800] Asynchronous session evaluation is I think possibly the final quality-of-life improvement I want to mention. This came early in the year, just with ob-python, and it's been delayed because in order to actually make it work, they've required some fundamental changes to the way that ob-comint works. Now that's been implemented, we've since seen support extended to ob-R, and hopefully, we'll see more languages join this list in the not-too-distant future.

[00:12:42.800] Now I guess one bonus which I tacked on just for fun is it's now more convenient than ever to actually specify the permissions for tangled files. Previously you had to give a list expression which should be evaluated. Now you can give it directly in octal form instead of being a list expression that produces an integer representation of the octal permissions. Or you can use ls style: rwx and dashes. Or even just chmod. I want to be able to execute this as a user, which will basically modify the default permission to add that capability.

[00:13:18.280] Alrighty. So that's the Org project itself, but there's also a whole ecosystem. So what have we got here? Well a whole bunch of Zettelkasten or personal-knowledge-base-type projects. One of which is logseq, so that's an online open source Zettelkasten which supports both Markdown and also Org mode as a first-class format. Then of course we have Org Roam, which provides a Zettelkasten built directly on top of Org within Emacs. Both of these are seen considerably interesting over the past year.

[00:13:52.600] Moving on to visuals, minad has produced another lovely minad package in the form of org-modern which just spruces up the visuals of all documents and seems to have been quite well received recently since its release.

[00:14:10.120] Building on top of the citations from earlier, Andras Simonyi has produced the wonderful citeproc-org library, which, if you're not familiar, allows the capabilities of the citation style language which has now become something which is quite widely supported to be used for Org citation exports. This means that you've got access to I think at this point is it thousands or tens of thousands of different bibliography and citation formats which is obviously a huge boon to org citations. Lastly, just to be slightly critical, I'm actually going to mention the Neovim Org mode project, because I think this really shows the interest in Org as the format, beyond just Emacs. I think I haven't gone into it much here, but there's been quite a lot of development with external tools making use of the Org format. Clearly, we've done quite a few things right, and so I think it's interesting to see the interest that exists outside of Emacs, even without all the lovely tooling we've built, just out of appreciation of the formatting, its potential. Speaking of the format, though, we've also seen three new parsers on the scene this year. We've got one in Julia, Haskell and another one for Tree sitter. The last one, I think, is currently the least capable, but also potentially the most interesting in terms of what possibilities it allows for. Okay. So that's a quick speed run through some of the developments over the past year.

[00:15:44.040] What's coming next? So there's been quite a lot of work done with the Org syntax document. In fact, I've completely written it, and we've now taken it up to spec to actually accurately describe the way that the Org format is, as of Org 9.6. Now, I think this is quite an important document for the growth in parsing tools to help actually ensure that the way that external tools process Org is actually in sync with the way that Org mode does. I think it's worth doing everything we can, really, to avoid the sort of implementation drift that we've seen with Markdown. I don't want anything near that. This is also quite good for the Org format itself because, in the process of going through this sort of effort, it brings attention to irregularities in the syntax which you might want to resolve, and as well as helping the robustness of org mode itself. So ultimately, this is to everybody's benefit, really. It's also my personal hope that this might actually get to the point where we consider submitting this as a text format to the Internet Engineering Task Force as a new text format. So that would be quite nice.

[00:17:06.320] The Org project itself has a layer on top of the mailing list called Woof developed by Bastien, and that's about to have another major release. So what this is going to do is improve the ease of which we can actually monitor what's going on with the mailing list. So this is when people have patches, bug reports, or other types of things raised on the mailing list. It's a nice way to collect the status of those and put them all in one place. So improvements to this improve the ease of which the Org mode project can be managed, which is always quite nice to see.

[00:17:41.920] There's also been--jumping back to the export which is mentioned right at the start of this presentation-- we've got the introduction of engraved faces to LaTeX export. Now what's quite interesting about this is that it's actually used as Emacs' native font lock and allows for processing that in a generalized way to different output formats. So at the moment, this is just integrated with ox-latex, but it contains the functionality needed for HTML and ASCII, and it could also be extended to other formats like ODT. So we could potentially have full syntax highlighting based on Emacs exported to, well, really all of the Org mode backends, except, I suppose, the plain text ones like Markdown. And I think from both the capabilities perspective-- because I think, really, font lock in Emacs from Emacs major modes tends to blow basically everything else vaguely used out of the water, whether it be listings, minted or other efforts--and also from a consistency point of view, this could be quite a nice development. Alrighty. Now this talk is "This Year in Org," and I think you all may have guessed this is very much tied into my work with This Month in Org which started, I think, a bit over a year ago. And so, as you're all avid readers, I'm sure you've noticed that there haven't been as many posts as of late. Now this isn't because my interest in This Month in Org has been diminishing. Simply, it's the consequence of an evaporation of my free time. However, This Month in Org is still going to stick around. The only change really is that the title is going to be-- probably continue to be a bit more aspirational than descriptive in the near future. We'll see how this goes. Well, thanks for listening to this overview of the state of Org at the moment, and hopefully, I'll see you next year.

Captioner: sachac

Questions or comments? Please e-mail emacsconf-org-private@gnu.org