The Future of Org

Ihor Radchenko

The following image shows where the talk is in the schedule for Sat 2024-12-07. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 40-min talk ; Q&A: BigBlueButton conference room https://media.emacsconf.org/2024/current/bbb-org-update.html Etherpad: https://pad.emacsconf.org/2024-org-update
Etherpad: https://pad.emacsconf.org/2024-org-update
Discuss on IRC: #emacsconf-gen
Status: Q&A open for participation

00:00.000 Introduction
01:14.280 Message from Bastien Guerry
03:15.920 My step-by-step journey to Org maintenance
05:09.241 Priorities for Org maintenance
08:11.767 Modular Org
08:41.590 Slim down large Org libraries
10:00.000 Upstream generic Org libraries
11:25.400 Use modern Emacs APIs and libraries
13:13.257 Improve Org parser APIs
14:45.731 Improve Org babel APIs
15:57.380 Beyond Org code and Emacs: third-party packages, apps, parsers
16:31.200 org-contrib
17:37.820 Org orphanage
18:25.840 Mobile apps and parsers
20:23.869 Long-standing syntax problems
21:56.240 New syntax features
23:30.503 New features I hope to see in Org
25:54.073 Org community
26:01.358 Org community forums - Org mailing list
27:17.160 Org mailing list - world
30:05.580 Contribute ideas!
31:01.520 How much can a single person do?
31:35.000 Contribute code!
33:02.080 Why contribute?
35:40.240 Benefits for code contributors
37:41.420 Contributing as non-programmer
38:30.440 Got no free time, but still want to help?
39:12.997 Thank you

Description

Transcript

[00:00:00.000] Introduction

Hello, everyone. My name is Ihor Radchenko, and I'm the new official Org Mode maintainer. Today, I'll briefly introduce myself and then share my ideas about the future of Org Mode development. I will start by passing the word from Bastien, the previous maintainer. Then I will tell you a little bit about my story, starting from ordinary Org Mode user all the way to the maintainer. Then I will detail the new directions of development and specific features which I want to see in Org Mode in the coming years. And I will conclude by asking you, Org Mode users, to contribute to Org Mode because all the features which are too many can only be implemented with the help of the community, with the help of more contributors. And for that, of course, I will also talk about Org community and how I see it evolve so that we have a good communication between the development of Org Mode and the community ideas. Let me first pass the word to Bastien.

[00:01:14.280] Message from Bastien Guerry

Hello, everyone. Ihor Radchenko is the new Org Mode maintainer. He's been acting as such for a couple of years now, and I'm really glad he's finally agreed to take on the role officially. As a maintainer, I've probably done a few things right and certainly made many mistakes. Here are 4 lessons I've learned in 14 years. The first is that maintaining Org Mode isn't just about code, it's mostly about users. Of course, some of them will never learn how to report a bug, some of them will behave like spoiled children, and most of them will expect you to work for free forever. Nevertheless, the time is as valuable as yours. Whatever they request, there is always something that can lead to a positive outcome for Org Mode or its community. The second lesson is that maintenance isn't just about technical choices. It's also about predictability. Be very clear and very loud about what users and contributors can expect of your time, skills, and motivations. Stick to the robustness principle by being liberal in what you accept and strict in what you produce. Thirdly, it's all about learning. Let's build a culture together where it's okay to ask stupid questions. No one is born knowing how to write in English, how to report a bug, or how to maintain a large piece of code. Remember that old-timers were newbies and that newbies could become maintainers. We all have a lot to learn, even if it's just how to respect each other. Finally, as a maintainer, think about the next one. Who will be in your shoes next? What kind of maintainer do you want for a software you will be using for decades? It's also a responsibility of the Org Mode community. How can we collectively attract maintainers that want to help us use and enhance this wonderful little tool? How can we, as Org Mode users, help Ihor pave the way for the next maintainers? Enough said. I'm confident Org Mode is in good hands and I'm a very happy user. Thanks everyone for all these years of fun and learning.

[00:03:15.920] My step-by-step journey to Org maintenance

Now back to my talk. Let me start by briefly introducing myself. I'm actually not a programmer. By training, I'm a material scientist. and I only started using Emacs and Org Mode and naturally doing programming (because that's Emacs) when I was doing my PhD and I wanted to use Org Mode to tame my research work. A couple of years I was just an ordinary user, until I learned enough and got enough courage to report my first bug. Then it all evolved over the years. I started participating in the mailing list, I started learning more about Elisp, I reported more complex bugs, I eventually got around to go and fix the tricky bugs. Then I started participating more in Org mailing list in helping fixing bugs in selected areas of Org mode and eventually switching to all parts of Org. At some point I ended up doing the de facto maintenance job together with Bastien and got an actual maintenance offer which I accepted recently. The key takeaway I want you to get from here is that you don't have to be a programmer, you don't have to know Elisp to contribute, and you don't even have to be like that to become a maintainer. All it takes is slow, methodical, persistent learning over the years, participating in the community, and eventually submitting your patches upstream. And eventually you can become a maintainer, or not a maintainer as you wish. Enough about me.

[00:05:09.241] Priorities for Org maintenance

Let's talk about what I think should happen with Org Mode and what my goals on Org Mode is. The first and top priority for me is the basics. The basics, the code-based stability. Basically, all the foundations, all the APIs in Org Mode, all the basic libraries, which everything builds upon so that we have fewer bugs and we have more understandable code so that others who want to contribute have easier time understanding what is going on in the code base and contribute without much of a problem or confusion. Second equally important direction is the Org community, because a single person, even a couple of people, cannot really develop such a big project as Org Mode. And we always want new contributors, which are not coming from nowhere. We need to have the community of users. We have a community of people who participate in discussions, who later submit patches and code, and that's where we get most of the new features. So I would like to improve the communication between Org community and the development. The third direction I want to pursue is making life easier for third-party packages. Because we don't only have org-mode as it's distributed with Emacs. We have a lot of ELPA packages. We have a lot of MELPA packages. People who want some specific features on top of org-mode do develop these packages which are widely used or not so widely used. It's not only about Emacs. We have a lot of mobile apps that can work with Org files, have a lot of parsers which enable this mobile apps, or in general, programs outside Elisp to understand Org files. Fourth direction is the Org markup as a foundation of org-mode as a major mode, because we have a lot of functionality in org-mode itself inside Emacs, but it's all based on the underlying markup, and markup should have enough features to support the functions we want to see there. Last direction, which is somewhat less important, mostly because I don't have that much time to focus on everything, is the new features. I do want to see certain important features in Org mode, but I usually cannot spend too much time on them because of the previous more important parts. I do rely on the org community and the contributors to implement these new features. My idea is that I want to direct which features and how they should be implemented, but the actual people who implement them should come from the community. Now let's go into the details.

[00:08:11.767] Modular Org

The code base. As a bit of motivation, I would like to share this email from Richard Stallman, who a couple of years ago asked about improving Org mode by making it more modular. That's directly asking about improving the foundations of the code. That's what I think is an important direction as well. Here's an example.

[00:08:41.590] Slim down large Org libraries

We have a lot of really, really large org libraries, like org.el, which is like one megabyte or something large, org-agenda, org-table, org-list, a number of files which are really, really, really large. What is worse is they are hard to understand sometimes. I even have an example, alphapapa complained that part of his motivation to write org-ql and specifically his agenda part, agenda-like part, was because org-agenda is so hard to understand that it's easier to start from scratch. I don't want such situations to happen in future, if possible at all. One of the projects I'm currently working on, it's work in progress now, is splitting Org libraries into smaller parts, into more documented parts, into the APIs which are actually documented and explained in the code at least, so that people who just open org-mode code cannot be scared away and go and read the comments and understand what is happening there easier. At this point, I have almost doubled the number of libraries. It's still work in progress, so there's a lot of room for improvement in this area.

[00:10:00.000] Upstream generic Org libraries

Another direction which is somewhat reasonable in relation to splitting things down is that some libraries are really generic in Org Mode, because Org Mode often has a functionality which is really new, and for that, it had to implement some very generic functionality that doesn't have to be used just for Org Mode. This can be generalized for Emacs in general. There is a number of libraries which we may or may not upstream to Emacs, depending on what Emacs maintain, I think. As one example is org-capture, because it's a very obvious example. org-capture started as support from remember.el, which is still a part of Emacs. It has more features than remember.el, which we required for Org. But, you know, these features can be backported. Why not? And then not only Org-mode, but other Emacs libraries can benefit from these features we have in Org-mode only. Similarly, I have a long list of different libraries that can be shared. Yeah, for some I'm not sure, but in general, there is a lot of work that may be done and may be discussed in the future.

[00:11:25.400] Use modern Emacs APIs and libraries

The third part about the basics is making use of the new Emacs libraries. Org Mode in general is quite well written in terms of Emacs integration. We do support many of the Emacs features and libraries which are generic. However, in more recent Emacs versions, we started getting some new features, and we do want to make use of them in Org. For example, recently we contributed yank-media support for clipboard pasting and drag and drop. Now it is supported in Org mode already in the released version. Eventually we want to support transient.el, because now Org uses ad hoc system. It's of course much better to use existing and more powerful menus, which are implemented in transient. It even has some initial work-in-progress implementation. I hope it can be eventually extended to the whole Org Mode There are other things like compatibility, which there is an excellent Emacs library, compat.el, that provides backwards compatibility and Org also has something like this in org-compat. We don't have to write it ourself again. We can make use of the existing library. Similarly, there's a very, very new library track-changes for tracking changes in real time. Eventually, if you want to support context-menu mode, maybe touchscreen, Android support, I don't know, but I hope it can be done by someone. And some more generic library: select thingatpt. That's about using external APIs.

[00:13:13.257] Improve Org parser APIs

Now about the internal Org APIs. One important, probably one of the most important parts of Org is the parser, how Org itself understands the Org files. The situation is that we have two parsers in Org mode. One is the Org element, the proper parser, which we use as a reference, but many parts of Org still use regular expressions, which are approximate. These two parsers are not exactly consistent, which is really bad, and I hope to solve this. I already started doing some work by factoring out some part of abstract syntax tree and working on real-time parser, incremental parser, which is enabled by default in Org 9.6, but there are still parts which I need to work on. Eventually I want to get rid of regular expression-based parser completely, so that we don't have any inconsistencies inside Org Mode. One of the examples of these parts that are still using regular expression is fontification, which is often simply wrong, especially in some edge cases, and we really want to use the proper parser in this area. Maybe even editing org files using the parser syntax tree, but that might be tricky, although there is an existing library that implements some ideas for this. The key point is that org-element-api, the parser, should eventually be used everywhere so that everything is consistent.

[00:14:45.731] Improve Org babel APIs

The second important API is the Org babel. Currently, Org babel does have some API, but first, it's not well documented. Second, it's sometimes awkwardly designed, especially compared with the exporter. I do want Org Babel APIs to be more consistent. Another thing about Org Babel, it's not exactly API, but you know that documentation for most of the Babel backends are not even in the Org manual, even though the backends are built-in. They are on Org Wiki, and we do want to move them to the manual eventually. That's the important part, and it should be done. Those are some obstacles, like not all the features are properly implemented, and that's a bit of an extra job that should be done. Another small thing which thanks to Bruno Barbier is being done, in progress: we should have a more robust asynchronous API for babel. I hope it can progress further. For now, it already progressed quite far.

[00:15:57.380] Beyond Org code and Emacs: third-party packages, apps, parsers

That's all about the basics, the underlying backbone of the Org codebase. Let's move to the second important direction which is the third-party packages and basically the parsers for mobile apps. I will postpone the community to the end because I want to have a call for contribution at the end. For third-party packages, I would like to remind you that

[00:16:31.200] org-contrib

Org mode used to have something called org-contrib as a part of Org mode, which is a collection of small libraries, small packages that didn't have a proper copyright assignment basically, but more or less a part of Org mode. This is no longer the case. Now what we did is we moved a number of very rarely used libraries from Org core itself to org-contrib, and now we treat org-contrib as basically the libraries that we really want someone to take responsibility for. We want to maintain this for everything that is in org-contrib, and from me and other Org team, we do not spend too much time maintaining this package, just do some most basic bug fixing, and that's all. If you know, if you see some libraries from org-contrib and you use them, and you know Elisp, please volunteer to be the maintainer, because otherwise there will be not much progress in these libraries.

[00:17:37.820] Org orphanage

As a natural extension of this and inspired by Tarsius's Emacs Orphanage idea, we also maintain a small page basically listing the libraries, some others like packages, Emacs packages that are not really maintained. If you are a maintainer of a library and you don't have time to do it, you can write to Org mailing list and we can add the library to this page so that we can search for new maintainers in a more centralized way. If you are an Elisp hacker and you want to help something for Org Mode, you can check that page and see where you can help.

[00:18:25.840] Mobile apps and parsers

Now away from Emacs, or mobile apps. We have quite a lot of mobile apps at this point. Unfortunately, it's very hard for me and many other Elisp contributors to contribute to these apps because they are not using Elisp naturally. But these applications heavily rely on Org markup. I do hope that we can keep Org markup consistent enough and rich enough so that people don't have to invent extensions to Org like what happened to Markdown. I really want to emphasize that I want to see more Org parsers in different languages so that they can be used by developers. For people who are writing these parsers, I want to share this link. It is the org-syntax reference. It is the official Org syntax, which is what we think it should be. It's described in plain human language. It's not a code. All details should be listed there. Please use it as a reference if you are writing a parser. Eventually, this document will be submitted to IETF, I hope. In the future I hope to write a set of tests which will work as benchmarks. basically we have some existing tests for our internal parser and I want to factor out these tests so they can be used by any parser, so that we can compare the performance and which parts of Org mode are parsed and which parts are not. I mentioned that we want to submit to IETF, which means that Org markup will become the actual registered format.

[00:20:23.869] Long-standing syntax problems

But before we do that and thus fix Org markup in stone, because it's very hard to change things in the IETF, it's important to address important problems, existing problems with Org syntax. There are some problems, like I mentioned the inconsistencies between the two existing parsers in Org mode. There are also some parts, there are some examples, like there are problems with numeric priorities, for example, which are not treated consistently. There are problems, more general problems with syntax when people request some edge cases which should be addressed. Like, it's very hard to do interword markup. We have zero-width space workaround, but many people dislike it, so maybe we want to do something about it. We have some edge cases when we combine emphasis with links. We have some edge cases when we have double blank lines inside some source blocks, for example, and combination in the list. I hope we can somehow address it. It's not impossible to do it, it just requires time. One annoying part is the inline task syntax. It's annoying both from the programming perspective, internally as implementation, and from the UI perspective, because there are too many stars. We probably should redesign it eventually, maybe in backwards-compatible way, but we will see how it goes.

[00:21:56.240] New syntax features

Another part is not just fixing the edge cases or problems, it's the completely new syntax features. That's probably done after we submit to IETF. But there are important things that people often request, like time zone support in timestamps. Better repeaters, like more flexible repeaters, that's really a frequent request also. Another idea is some custom markup, which is coming to various requests, like, for example, people often ask to highlight some words with a color, for example, or with some other special way and then export it in a special way, just as we do with special blocks, basically. What I want to introduce is the ability to do it on a macro level or inline. Of course, a new syntax feature which I wish we could have is the multi-line cells in tables. It's very frequently requested as well, but I really have no clue how to do it. We had a discussion about this in previous discussions, but there was no conclusion. We don't see a good way how to implement it syntax-wise. Unlike time zones where we decided exactly the syntax, how it could be and we just need patches to be submitted, here even the idea of syntax is not clear. Please do participate in these discussions if you have ideas.

[00:23:30.503] New features I hope to see in Org

The last direction is the new features. In general, I welcome all kinds of good features, but there are certain things which I explicitly want to see and I hope to see submitted. If you are interested, please do submit patches. One, and probably many people are aware about it, is the asynchronous LaTeX preview, developed by Timothy and Karthik. I hope it can be finalized eventually and upstreamed. It's pretty much in ready state, but on the technical level it should be discussed further and revised. The second is org-ql by Adam. I hope it can be upstreamed. It's also a work in progress. It's just a question of free time for Adam mostly, I think, and me. That's another important part, new feature. The third is the so-called multi-page export. The idea is the same as many packages for blog posts, so that you have a single Org file and then you can export multiple HTML pages, for example, or PDF pages, anything like that. This work in progress by Orm, thanks to him, although it was a little bit stuck because I am not exactly sure how to best integrate it into the existing APIs. If you are a developer of one of the blogging packages, I would appreciate if you can chime in and probably share some ideas here. Next are just some wishes I wish we could have, but it's not very detailed. One is the multi-language support, so that we can have Org documents in multiple languages, or maybe even translations. The collaborative editing that many people would wish to have, I think. Things like tracking changes, adding comments, importing from some other Org formats with the comments and changes so that we can actually participate with all those Microsoft Word users and stuff like that. But that's really too much for me alone to handle. If you want to see one of these features, please consider contributing. Just write the mailing list about your interest and we can start from there.

[00:25:54.073] Org community

Now, the important part is, you see, I keep asking people like, please contribute, please contribute, but who should contribute?

[00:26:01.358] Org community forums - Org mailing list

So I want to improve communication between the community and the mailing list. Now, people often discuss new features or ideas on all kinds of places like Reddit, Mastodon, like all kinds of Matrix/IRC chats, even on meetups, some non-English language. That's very nice that we have this community, but not all the ideas are visible to the developers. I do wish that the most important things that people want to see should end up on the mailing list, one way or another. I'll later talk about some ideas how I think it can be done. Another part is we have org-wiki and I hope that we can make it more centralized space for interesting Org mode articles, for tutorials, for blog posts, at least linked to blog posts. If you have some idea about good blog posts, it would be nice if you submit a patch to work or at least email about this link to the mailing list.

[00:27:17.160] Org mailing list - world

Of course, not everyone likes to use mailing lists or don't have a good setup to do it, or even don't want to read everything on Org mailing list, because there are things like bug reports, people don't really want to see that. So Bastien actually developed a tool that can help with this. If you want to monitor Org mailing list, but want to see only the most important discussions, and maybe participate if you decide to, you don't have to register. We have Woof, which is basically a web page that monitors our main list, but not every email. That's the most important announcements, some blog-like posts, or feature requests, or some discussions. Then on this webpage, you can see it as HTML, or you can subscribe as RSS, or even download in Org or MD format. Thanks to Sacha Chua, also weekly news about Emacs in general, but Org Mode as well. It also includes the new features in Org Mode on the development branch and the interesting new blog posts and discussions on various Reddit forums or mailing lists everywhere. I did this little bit of experimental integration so that many lists can also be read, kind of announced on the chats. For example, in #org-mode Matrix room, we have a bot that connects to Woof RSS so that all the news and discussions are notified in the chat so that people can see if they are interested and maybe, hopefully, participate. I wish we could also have similar kind of both for Reddit, Mastodon and maybe IRC. That way we have mailing list connected to more active and more modern forums and chats. More people exposed to what is happening. Another part is that we actually have a web interface to Org mailing list and you can even reply from there, but it's not always obvious, unfortunately. We have this public inbox software to transform the mailing list into HTML pages. That's a decent interface, but it could be improved to look more forum-like, so that people can easily find the reply button or basically participate without too much effort, even if they are not subscribed or they are just casually reading. That would be nice if someone knowledgeable of CSS could help with this.

[00:30:05.580] Contribute ideas!

Again, I cannot emphasize more that most of the Org ideas of the new features are coming from people, but often they are known either by someone submitting a patch to the mailing list or submitting an idea to the mailing list. Rarely, it happens when someone is reading posts from Reddit that don't share to the mailing list. If you think that there is some important discussion happening on the forum, it would be nice that you can go ahead and share it with Org mailing list. We don't care about on-topic, off-topic, because unlike emacs-devel, we don't focus on development. We can discuss some related to Org mode topics in open-end list, just like on an ordinary forum.

[00:31:01.520] How much can a single person do?

I would like to end my talk with the call for contributions. Let me explain a little bit, because you saw now I shared many many ideas and I do spend a lot of time on Org Mode. In fact, for the last year, I was spending like 30 hours per week or something. Until recently, I can spend a bit less, but still even with that much of commitment, there is no way I can handle everything by myself.

[00:31:35.000] Contribute code!

We really, really need more people to contribute so that Org mode keeps moving forward. Again, I just shared ideas, but I have so many more. I have more than a thousand ideas noted down. Yeah, there's no way I can do it myself. I do need help for new contributions, for new features. If someone can help fixing bugs, it would be really great. If someone knows specialized things, like for example Open Document Format, which I don't know very well, it would be really nice, because things like ODT export requires knowing that, and if someone already has the knowledge, it would be much easier if that person can help with such things. Although in the coming years, it's very important to have regular contributors because my life is my life and things may happen, so it would be nice to have some kind of backup so that bugs keep being fixed and things like patches being accepted and stuff like that, so that some person can at least temporarily take on my job. I would like to emphasize that the code contributors are the most important contributions for Org Mode. All other types are less important, really.

[00:33:02.080] Why contribute?

I want to spend a couple of slides trying to motivate you to contribute. If you ever considered contributing or if you ever wished to have some feature in Org Mode, really don't wait, because most new features are contributed by people. They are not contributed by me. I contribute a few things, but I am alone. I cannot contribute many things. Most of the things are contributed by users who go ahead and submit patches. And even in very commonly requested features, it's usually someone who steps up. If you want something, don't wait. Just go ahead and write to mailing list. You don't have to submit a patch immediately. You can just say, I am interested. We will start from there. We'll start because that's my job and I will guide you through. If you have problems with Org in general, I'll explain, because that's what I know. I will explain how to implement things better, but I need someone to actually do the job and write the code. Even if you don't have 10 years experience with Elisp, it doesn't matter. You can learn on the way. There are many examples of this in the mailing list when we start slowly, fix things one by one, and eventually arrive to a good quality when the person is not experienced. You just need to be ready to learn things, Spend your free time (that's as usual, right?), and have the interest in specific thing you are contributing. Don't be afraid to be wrong. I will be there to assist. If you don't like emails, if you don't like patches, it doesn't matter. Again, you can share GitHub link. It's okay. You can go and modify some workflow directly. You submit the modified version. It's also okay. It's easy for me to create patches if I need to. If you don't like emails at all, you don't want to participate in the mailing list, it's not ideal, but I can still work with this. I am on IRC, I am on Matrix, you can ping me, it's yantar2. We also have monthly meetup, so you can go and ask by voice, we can just talk in person, discuss your ideas, it's also fine. The key point is that we always welcome new contributors. The more contributions to Org Mode is better.

[00:35:40.240] Benefits for code contributors

To increase the motivation, I'll just try to show some benefits of contributing to Org Mode and free software in general. It can be actually useful for your CV if you're a programmer. It doesn't matter if it's Lisp, because you can have a pet project that demonstrates your skills, that you can finish something to a usable state. A pet project usually demonstrates that you can work alone, but it doesn't demonstrate anything about you working in a team, in the production team. When you contribute to Libre software, look, you will work with a number of people who contribute and comment on your work. You'll have to learn a new code base. You will have to follow certain standards. All these things, by having a public record of contribution, will be a valid point that proves your knowledge in your CV. I put a small quote (which I'm not going to read in the interest of time) from Rudolf Adamkovič, who is describing these three points in probably a more expressive way. Another benefit is you can actually get money from this. Thanks to a number of Org Mode users who kindly contribute to Org development, we have some amount of money coming in, and we don't hold on this money. If we get another person who contributes to Org regularly, we are happy to share this money, because we do know that getting some extra money, even a little bit, does improve motivation. We are really ready to share this. If you are serious about contributing, you can just request this and we can share a part of the donations to you.

[00:37:41.420] Contributing as non-programmer

For non-programmers, we also have a lot of work to do. There's a lot of stuff we can do on the Org wiki, especially with CSS styles, with updating articles, with adding links to tutorials, or even writing tutorials about Org Mode. It would be nice to have more screencasts. It would be nice to improve Org manual, because that's the most difficult part for me because I'm too familiar with the code. Writing the manual when you know things internally is very hard. I just know too many things. I can assume that people know too much, that new users have no idea about certain things. I can just omit those without being aware.

[00:38:30.440] Got no free time, but still want to help?

The final slide I would go to come back is about donations. Again, most important is contributing code, but I do hope that donations can increase the number of contributors. I don't know. I know for sure because some people like Timothy, who is participating in this, he did find the donations helpful, especially for more boring tasks like bug fixing and to move things over the long time. If you cannot contribute by other means, it would be appreciated to contribute money.

[00:39:12.997] Thank you

We came to the end of my talk. Thank you for your attention. If you have any questions, please feel free to ask. I think we can even discuss further during the next Org Meetup in the coming week. There, if you have more detailed questions, we can continue discussing apart from what happens after this presentation.

Questions or comments? Please e-mail emacsconf-org-private@gnu.org