Back to the schedule
Previous: Org-mode and Org-Roam for Scholars and Researchers
Next: Sharing blogs (and more) with org-webring

Org-roam: Technical Presentation

Leo Vivier

Download compressed .webm video (23.5M)

Org-roam is a Roam replica built on top of the all-powerful Org-mode.

Org-roam is a solution for effortless non-hierarchical note-taking with Org-mode. With Org-roam, notes flow naturally, making note-taking fun and easy. Org-roam should also work as a plug-and-play solution for anyone already using Org-mode for their personal wiki.

Org-roam aims to implement the core features of Roam, leveraging the mature ecosystem around Org-mode where possible. Eventually, we hope to further introduce features enabled by the Emacs ecosystem.

The purpose of the talk is to present some technical aspects of Org-roam. From the very beginning, we wanted Org-roam to scale with your notes, and this meant that we had to keep a close eye on our performances. As we iterated, optimisation remained a top-priority, leading us to constantly peek under Org-mode's hood. Not only has this made us better developers, but it has also uncovered paths of optimisation for Org-mode itself.

The talk is targeted at software engineers willing to peek under Org-mode's hood. A rudimentary understanding of Elisp will be required.

Points to be covered

  • SQL database via emacsql
  • Elisp libraries
  • Parsing of Org-mode files
  • org-elements.e
  • Parsing with a background-process
  • Ensuring consistency via hooks
  • Actual start and end time (EST): Start: 2020-11-28T15.39.41; Q&A: 2020-11-28T15.56.29; End: 2020-11-28T16.01.03

Questions

Why not to run a background Emacs for parsing instead of implement a new parser?

Running a background Emacs progress sounds great, but is still limited. Forwarding all queries to a background Emacs (like org-mode's exporter does) is only feasible with a (??? zaeph can probably fix the answer).

How often does the DB index get updated in order to contain changes within Org files?

Either on save, or on idle-timer.

Did you ever think of opening up (or designing) the SQL DB as a general Org speedup-tool outside of org-roam so that other libraries that do execute complex queries are able to re-use the summarized data?

FYI, see John Kitchin's work, he uses a SQLite database to index his Org files. https://kitchingroup.cheme.cmu.edu/blog/2017/01/03/Find-stuff-in-org-mode-anywhere/.

  • John's DB approach is great. However, we should not end up using several DB-index in parallel. ;-)+1

Obviously with the 'global backlinks' agenda, it would be interesting to combine with the eev stuff from before :-) (https://github.com/edrx/eev)

About the external program, you could just talk to the PANDOC guys (or Firn [Parses org-files into data structures with Orgize https://github.com/PoiScript/orgize], Logseq [OCaml & Angstrom, for the document parser https://github.com/mldoc/mldoc]), they're very helpful and have already a good org-mode parser

Is it feasible to have this process of parsing org-roam following the LSP protocol? that would allow to be editor agnostic, and it would save the work to define the communication protocol and any other technical details.

Notes

  • "org-roam just wants to create backlinks".
  • org-mode has many many files (377 lines in dired… including .elc files).
  • If you want to create an index of all the org files using the native format, it would be very slow. So org-roam uses a sqlite database.
  • ripgrep (written in Rust) is more capable than grep; used by some Zettelkasten implementations.
  • "Is there something we could do to import backlinks into Org mode?"
  • "We've always tried to have an experimental ground where we can track backlinks"

Saturday, Nov 28 2020, ~ 3:40 PM - 4:00 PM EST
Saturday, Nov 28 2020, ~12:40 PM - 1:00 PM PST
Saturday, Nov 28 2020, ~ 8:40 PM - 9:00 PM UTC
Saturday, Nov 28 2020, ~ 9:40 PM - 10:00 PM CET
Sunday, Nov 29 2020, ~ 4:40 AM - 5:00 AM +08

Back to the schedule
Previous: Org-mode and Org-Roam for Scholars and Researchers
Next: Sharing blogs (and more) with org-webring