Back to the talks Previous by track: PGmacs: browsing and editing PostgreSQL databases from Emacs Next by track: An example of a cohesive student workflow in Emacs Track: General

Literate Programming for the 21st Century

Howard Abrams (he/him) - @howard@emacs.ch , https://www.howardism.org

Format: 16-min talk ; Q&A: BigBlueButton conference room
Status: TO_CAPTION_QA

Talk

00:00.000 Introduction
01:35.253 Do I still literate?
03:06.332 Advantages
04:28.720 Disadvantages
05:24.133 Ease of typing
06:24.720 Keep tangled code sync'd
07:22.501 Code evaluation
08:19.960 Has that block been eval'd?
09:05.239 Evaluating code in a subtree
09:26.872 Evaluating code from a distance
10:26.020 Navigating by headers
11:26.794 Navigating by function names
13:40.480 Why literate programming?
14:23.166 LP prose isn't comments
14:55.800 Summary
Duration: 15:51 minutes

Q&A

02:07.400 Q: What's the largest code base you've ever tackled with the literate approach (esp. Emacs + Org-mode)?
03:58.080 Q: Have you ever used org-transclusion?
04:08.440 Q: What is your usage of dynamic blocks in such workflows?
04:48.840 Q: Is the minibuffer being deliberately hidden in this video?
05:17.341 Q: What's your take on Emacs+Org vs. Jupyter notebooks (for interactive programming)?
07:07.800 Q: Do you think any programming language is more suited to literate programming than another?
08:21.560 Q: Do you use inline org function calls and org babel library and such?
09:36.970 Q: How do you handle the cases where org markup may sometimes interfere with some of the code?
11:06.220 Q: You said at the start that literate didn't catch on in corporate DevOps - why not?
11:29.421 Q: Why not that full stack on Markdown?
12:22.120 Corwin's aside on orgvm
14:49.520 Org and Markdown fragmentation
16:17.920 Q: How does your management of "TODOs" (projects/tasks) interact with this literate mindset, any insightful things you do on that front?
17:30.630 Q: Do you LP also on larger projects?
18:38.936 Q: Have you used Cucumber/Gherkin/BDD and do you think it has a strong overlap to what you talked about here?
19:54.600 Q: What granularity are you looking for re your org files and contents, with respect to a codebase that it tangles to, or in non-coding contexts?
Listen to just the audio:
Duration: 22:48 minutes

Description

Donald Knuth’s idea of literate programming in the 80’s and 90’s was interesting, but he didn’t realize what Emacs and Org can do in this century. In this talk, I would like to go back your initial dabblings with Org src blocks to show how you can program literately as quickly as you can in any other mode.

Some of the tips and tricks include:

  • Automatically keeping your lit code sync’d
  • Easier code generation
  • Jumping to Org headers to help organize code
  • Jumping to code definitions with the xref interface

At the end of this talk, I hope to inspire you to try it again, as my personal “go to” is programming literately.

I will be following the format and outline in my essay: https://howardism.org/Technical/Emacs/literate-writ-large.html

About the speaker:

About ten years ago, I gave a talk I called literate devops and people still ask me if I still use those techniques. For all my personal projects, I do. Even projects that I share with others, I often start programming with an Org file.

I will admit, programming within Org blocks has some burrs, but over the years, I’ve filed them off with helper functions, snippets and other features. Thought I would share these.

Discussion

Questions and answers

  • Q: You touched on it briefly, but how do you handle things like "C-h f" helpful info not being tied back to the defuns in src block code when you "C-c C-c" them in the org buffers instead of re-tangling it to the files, and other such things? Did you create wrappers for jumping  back and forth atop org's built-in mechanisms to go back and forth between org/tangled files?
    • A: (not yet answered)
  • Q: Apropos large literate programs: what's the largest code base you've ever tackled with the literate approach (esp. Emacs + Org-mode)?
    • A: The largest is the one I mentioned in the talk ... about 8000 lines of "code" and another "10000" lines of prose. I think I came to 15,000 max (in code blocks only).
  • Q: Have you ever used org-transclusion (https://github.com/nobiot/org-transclusion)?
  • Q: What is your usage of dynamic blocks in such workflows? Any interesting use cases and custom ones?
    • A: Not yet, will report back
  • Q:Is the minibuffer being deliberately hidden in this video? (first noticed this in the section previous to "Navigating by Function Names")
    • A: Not intentionally :)  You may notice the minibuffer comes and goes, sorry about that; not intentional (didn't quite "fix" all of them) (Thanks for the answer, no worries.)
  • Q:What's your take on Emacs+Org vs. Jupyter notebooks (for interactive programming)?
    • A: Not something I use right now.  Tend to include things from jupyter/python (e.g. numpy) that has been the biggest challenge (not knowing that stuff all that well), things like matrix multiplications are easy in jupyter not such much in org. May make sense to stay where you are comfortable.  Curious what the community can do to make this transition easier
    • You can't work with Jupyter in > 1 language either (I think). It's Py + SQL or R + SQL etc. Org allows 45+ languages in one document (I often mix languages).
  • Q: Do you think any programming language is more suited to literate programming than another?
    • A: R, C are my favorites (for literate programs). C (and C++) have got great support. There are some great books implemented in literate programming I think. The two that come to mind are Physically Based Rendering: From Theory to Implementation and C Interfaces and Implementations. The first is C++ and the second C. Ty.
    • \<gs-101> Personally, when working with Org files, I have a better time dealing with interpreted languages, such as Python, because you can initialize a session and the code is all conected. You can divide blocks however you seem fit.
  • Q: related to above, do you use inline org function calls and org babel library and such?
    • A: Used to use more org-babel inline functions, found sound bugs (maybe) 8yrs ago, right now my literate dev-ops is calling a lot of backend programs so the org-babel has limited help in that regard while in the emacs session things are "just available" so that hasn't help much either
  • Q: How do you handle the cases where org markup may sometimes interfere with some of the code, in places where you can't use "escapes" (\~ or = or | \<- vertical bar), doubly so if you use modes to not show these but the styled text instead, and so on?
  • I think an example is in C when you assign to a pointer  *p = &i;   (In Org, you need to write (*p) or ,*p = &i; or it will be mistaken for a headline *
    • A: (clarifying) when I'm making pros and I'm talking about a function I've written somewhere else I'll use tildes and look for those things so I can strip them off.  Is that the spirit of the question.  (confirmed).  Yes, I'll strip that off after finding the function name, so I can still mark it correctly.
    • Q: clarifying: when in code inside an org buffer, you don't get to use \~ or = (verbatim/etc), and any font-locking interferes with the proper display in the src blocks, that kind of interference.
  • Q: You said at the start that literate didn't catch on in corporate DevOps - why not?
    • A: I guess the big thing is not everyone is using Emacs and org is needed to make it work really well.
  • Q: I gotta ask: why not that full stack on Markdown, I'm sure it's crossed your mind at least a few times how the same setup on Markdown would be more interop-friendly with colleagues and such?
    • A:  It's a real good idea.
  • Q: How does your management of "TODOs" (projects/tasks) interact with this literate mindset, any insightful things you do on that front?
    • A: Yeah. Okay. So, uh, on to do's and that sort of thing, um. You know, a lot is the same as it was 10 years ago. At the beginning of every sprint, my company, we're using JIRA. All jokes are valid when I say that. But I just go to its web page. I copy all the tasks that I need to do that sprint. I wrote a function that takes that code and reformats it as a bunch of org tasks for me to do that I can just now have all my to-dos. And I start to work off of that from that point on. So it'll reference all the projects, and I do symlinks to every code base that I need to use. Most of the to-dos I put inside my code block is kind of just for me, just so I can remember it. Because the work that I have to do needs to be tracked a little more fine-grained, I guess. Yeah.
  • Q: \<Donovan> Do you LP also on larger projects? (More files & nested directories) 
    • A:  I haven't done nested directories, but I can now. Now that i've realized I have the feature where I can just jump to any projects and all the org-files and all the headings just show up, that works in nested directories, that that's fun.
  • Q: Have you used Cucumber/Gherkin/BDD and do you think it has a strong overlap to what you talked about here?
    • A: I tend to put the tests right next to the function, I like tangle it out to different files; keeping things together is nice.  Many frameworks assume we'll have things seperated out in a way that isn't useful to me.  I like to go old-school on that?
  • Q: What granularity are you looking for re your org files and contents, with respect to a codebase that it tangles to, or in non-coding contexts?
    • A:  Great questions, really subjective.  I change that all time.  I have an idea, I start to refine it. My goal at one point was to have an emacs config that was really small and simple and that just really doesn't happen, it's full of ideas and things that are half-baked and i pull them out and polish them up bit by bit so it ends up being like any code-base it just keeps getting refined.  Sub-trees, archiving are useful.
    • I've found it useful to prune the init file back to minimal every once in a while (actually, AI has been surprisingly helpful - perhaps it helps that Emacs is ancient and hence there is a lot of doc out there and much of it ... correct?)

Notes

  • My literate programming code extensions: https://www.howardabrams.com/git/howard/hamacs/src/branch/main/ha-org-literate.org
  • My Emacs configuration written in a literate style: https://github.com/howardabrams/hamacs
  • See it rendered here: https://howardabrams.com/hamacs/
  • My JOPS (Jump to Project Sections) code that "searches Org headers": https://www.howardabrams.com/git/howard/jops ... temporary location?
  • snippet on \<ssl: Cool one. 
    • 0_0 I need to do this.
  • This (evaluating babel blocks) is also possible with the Avy + Embark combo developed by karthik: (https://karthinks.com/software/avy-can-do-anything/#avy-plus-embark-any-action-anywhere), just jump to a block and then "RET".
  • Really good talk.  I need to find out how to extend xref to handle org files!
    • same here, I asked a long-winded question that was about that (before he touched a bit on it), but feel there's more in terms of wrappers and such
  • Denote has some pretty good use of dynamic blocks I think (https://protesilaos.com/emacs/denote#h:8b542c50-dcc9-4bca-8037-a36599b22779)
  • there is an RFC in for 'cargo-script' which allows building single-file crates - i think that will be quite useful in ob-rust

  • Not a Q, just a comment that we need more of your insightful posts and videos! :) (sic)

  • I am both hyped and scared by Howard's talk, some of his past insights into using org-mode (literate everything, many small spreadsheets, etc.) have changed the way I operate so thoroughly...

  • 19th century? Isn't that in the 1800s?
    • 21 1/4 century?
    • you are not incorrect I believe - easy mistake to make, it's not intuitive!
    • YouTube comment: 1:00 So Knuth invented literate programming in the 19th century? I knew he was old, but not that old!
  • i think the only time i have worked with literate programming is Inform 7
  • I find that the Julia support in org (babel) is good enough. Most of my programming is in Julia.
    • re: Julia in Org: For ob-julia support I wrote an ob-julia that does a few more things than the one that ships with ESS. https://github.com/karthink/ob-julia

  • (I'm here trying to milk Howard for all his fantastic insights in the Etherpad, hope that's not a faux-pas...!)

    • I, at least, am enjoying the Q&A session so not a faux-pas as far as I'm concerned.

  • YouTube comment: The best reason I've heard explaining why most programmers don't (and won't) use literate programming. It requires them to be literate in three languages: the programming language, the markup language, and most challenging of all, English.

  • YouTube comment: Q: Literate programming is very appealing in theory, but it's difficult with languages like Go where your code is split across multiple files. Any suggestions?

Feedback:

  • very cool
  • Lovely talk, thank you!
  • Awesome, thank you!
  • That was wonderful, thank you! 😊
  • Thank you Howard!
  • Thanks for your continued work, Howard!
  • A legend! ... loved the Ironsworn presentation from previous year.
  • excellent presentation indeed
  • Thank you for the marvelous talk!!
  • Thanks for the presentation
  • Your way of delivering is inspiring.
  • god I wish I was that good a presenter
  • I for one had been looking forward to this particular talk, so there's that :)
  • YouTube comment: Legend!
  • YouTube comment: Howard! Your videos have been such an amazing source of information. You voice is engrained in my brains haha

Transcript

[00:00:00.000] Introduction
Can you believe it's been a decade since I started pontificating on literate programming? I am Howard Abrams. In 2015, I spoke at this EmacsConf where I described my challenges I called Literate DevOps. The conference wasn't completely virtual, even though I was. My city of Portland was suffering a citywide electrical outage and I was without power, so I gave the talk in a corner of my friend's living room. People online asking questions and wondering about literate programming... I also see comments explaining why literate programming hasn't caught on in corporate practice. I often don't engage. I mean, is the online arguments and chatter over ignorance or preference? Sure, we're wired differently. I mean, my favorite programming languages put the parentheses before the function name. Literate programming has come a long way since Knuth proposed it in the 19th century. I feel it's come a long way just in the last 10 years. Obviously, this interest is due to Org. I don't think I would bother if all I had was Knuth's original preprocessor. But since I'm talking to fellow nerds about an open source project without corporate backing, let me change the title of my talk and re-pitch Literate Programming in the 24th and a Half Century!
[00:01:35.253] Do I still literate?
People often ask if I still program that way. I guess they want to know if there's any long-term benefits, for many of our tools and our workflows, while initially tantalizing, often don't last. But yes, when I sit down to write a program, I create a file with an extension of .org. I guess you can say I program literally. Let me be transparent. Do I use literate programming during my day job? Yes, but only for personal tools or for initial investigation. At the end of the sprint, I tangle the file and git commit that. My personal projects, on the other hand, are Org files. Since I can't show you the code from my day job, I'm afraid my example code will have a lot of parentheses. I'm sure you won't mind. I like having my Emacs configuration in Org. It's pretty bling. It has over 8,000 lines of code. I know, I can hear the screams and gasps over the network. However, the surrounding prose in Org adds 10,000 lines, and those lines are non-wrapped paragraphs. I mean, is that large? Sure, we've all worked on larger, so I guess it's not huge. Come on, it's still significant.
[00:03:06.332] Advantages
Advantages? Look who I'm talking to. I'm sure you know the advantages, but indulge me. I feel that one advantage of literate programming, especially with large code bases, is how you can organize and manage the complexity. Most programming languages tame large bases by putting code in separate files. While Org can too, with Org, we can group related functions together under expandable headlines. Here's one. You can see that I've got different sections grouped together. In my original talk, I mentioned how I would attempt to organize my thoughts before coding. I appreciate how I can look back at my notes. In my Emacs configuration, I review the prose to help memorize key bindings. My section on getting email working with Emacs using notmuch means creating small collections of scripts and configuration files. I can tangle them all from one Org file. I like that I can explain each part separately. You just can't beat having links back to Stack Overflow or that GitHub repo where you stole, I mean, became inspired to write your code.
[00:04:28.720] Disadvantages
Literate programming may push the boundaries of our workflows and revealing some abrasion, but we aren't solely working with Org. We have the flexibility of a Lisp engine to file down those rough parts. You may have your concerns. Perhaps you could reach out to me, and with particular issues, maybe we can figure something out. Here is my list of frictions, and the rest of my talk demonstrates my answers and my hacks. The goal in literate programming with Org is that it should not require more effort than non-literate programming. For instance, I shouldn't have to type much more than regular programming to get my code literate. I also shouldn't have to worry about the state between my Org file and the source code. I want to be able to jump around my code just as easily.
[00:05:24.133] Ease of typing
Let me explain more. I've created some templates using yasnippet. Since I was used to the old org-tempo feature, my habit has all the snippets starting with a < character. I'm not sure if I should demonstrate all of them as you may be doing something similar. I like to build on top of characters to remind me that if I just enter a <s, I need to put in the language. But if I append a mnemonic, I can get a full language. Why not do that with a full function definition? In this case, I'm smooshing one yasnippet inside another one in order to save myself some typing. My point here is to pay attention to what slows you down or hinders you from getting the advantages you want.
[00:06:24.720] Keep tangled code sync'd
Do you ever forget to tangle your code? You can append this code to the bottom of your Org file so that it gets tangled every time you save. I've written a function so I can visit that tangled file and then return. I've grouped all my functions together. I've taken a cue from Charles Choi, you know, kickingvegas, and his Casual feature set. But instead of Transient, I've just made a hydra using the major-mode-hydra package. Anyway, this allows me to use and remember my micro-optimizations. If you set the :comments property to link, the tangled output is back-connected. This allows us to edit the tangled code and have it update the Org file. Personally, I don't like this. My source of truth is the Org file, and I tangle as a one-way diode.
[00:07:22.501] Code evaluation
Often a block of code will reference a variable or call a function to find in another block of code. In my original literate DevOps talk, I discussed how to use the output from one block into another block by naming the first block and referencing it with a :var for the second. However, if all the blocks use the same language, you can use sessions, which create a persistent REPL behind the scenes. Let's evaluate the blocks of Python code in this file. The evaluation created a Python REPL. It's available in another buffer. This buffer matches the name of the session, but with surrounding asterisks. Evaluating a code block sends it into the REPL, and now I can work with my code blocks interactively. (That's not quite right.)
[00:08:19.960] Has that block been eval'd?
I primarily hack on Emacs Lisp, and textual changes to variables, functions, or macros--unless you habitually type C-c C-c--may not represent the state of your machine. A similar effect happens in any language that uses sessions. Sure, I can move the point to a block and evaluate, but I have three functions that allow me to evaluate all blocks in a buffer or all blocks in a subtree, or I can, without moving the point, evaluate any block I see. Now, this function here evaluates all blocks in a buffer. Someone mentioned calling this function when you first load a file. I'm not sure that's a good policy. I mean, have you not written a bug?
[00:09:05.239] Evaluating code in a subtree
Since this function right here evaluates only visible blocks, we can limit what Emacs evaluates to a single Org mode section. For instance, with the cursor in one section, I can evaluate just the blocks in that header section.
[00:09:26.872] Evaluating code from a distance
If I can see a block, why clumsily navigate to it when I can extend the avy project to just jump to it? For instance, let's pull this file up. I can jump to any of the four blocks. I think that's quite slick. Now why navigate to a code block solely to evaluate it? Yes, this is a terrible example, but these three blocks set a variable to different values. So without moving the point, I can evaluate any one of them. To be honest, the reason why I wrote this is because I often forget to evaluate a block after editing it. I've moved on, and I just don't want to jump back. Now, I can just evaluate from a distance. I apologize for the previous terrible examples, but I'm quite pleased with this feature.
[00:10:26.020] Navigating by headers
As I mentioned earlier, in a large code base, we organize code by library or module, and each file contains a class composed of methods, functions, variables, fields, et cetera. Literate programming in Org files allows me to add a semantic organization layer where I can group related concepts under headlines. Now, while this isn't specific to literate programming, I wrote a little user interface to allow me to jump to any heading in any Org file in a particular project. These are the headings in my Emacs configuration project. Notice the file name beforehand, before the colon character. The header name and its parent headers are after. Let me search for the LSP sections. Maybe I only want the one for Python. Now I use ripgrep to search the files and then some Lisp to parse the output. Unless someone has already done this, I should package this up on MELPA.
[00:11:26.794] Navigating by function names
What about jumping directly to the definition of a function, variable, or what have you? We can use Emacs's built-in xref library, but these functions don't understand that the source code is in Org files. When I started using Emacs 30-something years ago, I would pre-index my source into tag files, but the dumb-jump project uses the newfangled and faster text search programs like ripgrep to find a symbol in real time. I followed this pattern and wrote an extension to the xref API. Now, I want to jump around my code from both code block or in the surrounding prose. I'm sure it comes as no surprise that my presentation is just an Org file. Let's suppose my cursor is on this symbol. I wrote this function for this demonstration. We can jump to the definition and I can jump back. Notice it jumped into an Org file and back out. References, unlike definitions, is where something is defined and where it's used. Well, you know how the xref system works. Here, I can jump to the definition or where it's used. Of course, and jump back. I think this is cool. This should be a nifty package on MELPA. But my code is specific to Lisp, and I'm not completely sure how to make it general. For instance, what is a symbol? If you know the language, this is obvious. But what should the language be when your cursor is in the prose of an Org file? Python only supports sequences of alphanumeric and underscores, but in Lisp, a symbol can be almost any character sequence. I've been stewing on how to do this. I have ideas like prompting during the first query or scanning the language based on the nearest code block. I think I'm babbling.
[00:13:40.480] Why literate programming?
In true geek fashion, I dived into the details before answering some better questions. In my original Literate DevOps talk, I explained the advantages of initially writing down your thoughts, your plans, goals... the user requirements. But what do you do with all that luscious prose afterwards? Well, you do the same thing you do to your initial code. You refactor that prose. Just because the tech surrounding your code is now a first-class citizen doesn't excuse bad code. You want something more from both your code and your prose.
[00:14:23.166] LP prose isn't comments
The prose of your literate program isn't just regurgitation of the code in the block. You want something more helpful. You're really writing a research paper to yourself. I know what you're thinking. You've seen my Git repos. I'm guilty and not always the best example. However, I do get great joy when I see someone ask about something in Emacs and my response is little more than a link to my online repo that I've rendered as a website.
[00:14:55.800] Summary
I'm out of time. I hope this has been interesting philosophically as well as practically, as I think literate programming is the cat's meow. I'm afraid this summary slide is about my home-baked solutions that fit my needs, but hopefully you can recognize your pain points and address them. If you don't need my Literate DevOps-specific techniques for connecting code blocks, I suggest using sessions by default. I highly recommend looking at your workflow and writing snippets to give you less typing for Org blocks. I now jump by headlines in my projects, but extending xref to support Org files made literate programming as easy as programming the old-fashioned way. I do need to make it more general to put up on MELPA, though. Thanks for watching. Happy hacking, my friends.

Captioner: sachac

Q&A transcript (unedited)

I'm doing great. Good to hear from you guys. Yeah, I mean, you've been a regular of the show, I would say, for quite a while, and it's good. I really like the retrospective that you did the first time you attended EmacsConf. And when was it? Was it 2015? I can't actually remember. 2015 was when I presented that, but I did an earlier talk the year before. So yeah, that's why it's kind of thing. It's like, wow, it's been a decade. It's been a while. Yeah, and I'm glad that you are in a much better recording situation than you were in 2015 at the back of your colleague's house. It's good to see you at your own place. Well, it's also nice to pre-record and then just share. Yeah, I mean, that's why we keep talking about pre-recording. You know, when we really pushed pre-recordings, I think it was five years ago when we started, people were not really liking it. They thought, oh, but most conferences are just asking you to come and chat. And, you know, I don't need to do the pesky pre-recording. But now, and especially for you regulars, you're really starting to realize that pre-recordings are actually amazing for everyone. You're less stressed on the day of the conference. We, the organizers, are way less stressed. So it's a win-win scenario for absolutely everyone in the equation. Okay, so there's some confirmation bias at work, but I'm not really arguing with what Leo says for the record. It may be that we are just that this stoked that we're lying to ourselves by the opportunity that we have to chat with each of you. Sure. Sure. Sure. I have really appreciated all of the, the transcripts. It's been great that, you know, to be able to read and copy and paste and all that kind of thing. Yeah, it's great. So yeah, let's answer some questions here. Let's see. Yeah, just feel free to jump onto the Etherpad and just drop on the questions. I'll keep answering them all day kind of thing and get better answers as we go along.
[00:02:07.400] Q: What's the largest code base you've ever tackled with the literate approach (esp. Emacs + Org-mode)?
So yeah, the first question here is, what's my largest code base? I kind of touched it in my presentation. However, I think someone just went through and did a better count than it looks like. a little bit more. So as I joked about in the video, we've all worked on larger code bases, but that's still, it's kind of significant. So it's not, so yeah, anyway. So let's talk a little bit about searching for functions here. If you've got your functions in your code or code block, well, you can search for the descriptions, you know, as long as you've evaluated it. And so that wasn't so much of a problem, but I did want to like look at the commentary around any function that I would define. And that's where, yeah, you know, the C-x or the C-f to define that function and display it. It won't pull out the literate stuff around it, obviously, it'll only do the doc header. But if you could search for it and find it quick, then it might be just as useful. And I guess that's why I started getting into seeing if I can't tie that into the X ref. you know, just the standard ways. My first part, I noticed that evil has a different way of searching through for looking for code definitions. That was my first foray into it. It's like, oh, that's not a bad idea. Let's hook it up under, you know, in the standard way.
[00:03:58.080] Q: Have you ever used org-transclusion?
Let's see. Have I ever used org-transclusion? No, I haven't, but I think I want to take a look at it. I haven't heard of it before.
[00:04:08.440] Q: What is your usage of dynamic blocks in such workflows?
Let's see. What's my usage of dynamic blocks in my workflows? You know, that's a real good question. I haven't really used them a lot in all of my workflows, but yeah, it's an interesting, you know, we always start. At first, all we're doing is just taking a few notes, then we start organizing, then we've got links, and then we're writing code blocks, and then agendas, and table, you know, we keep building up on it. I guess I haven't really added that to my workflow, but maybe I'll have to come back and report on that.
[00:04:48.840] Q: Is the minibuffer being deliberately hidden in this video?
Is the mini buffer being deliberately hidden in my video? Not deliberately. I recorded it first and accidentally not clipped the video and turned that all off. So I went through and started fixing all of it. But yeah, so you'll notice the mini buffer kind of comes and goes in the video. Sorry about that. So yeah, it wasn't intentional at all.
[00:05:17.341] Q: What's your take on Emacs+Org vs. Jupyter notebooks (for interactive programming)?
What's my take on Emacs Org versus Jupyter? That's a real good question. I've been toying about seeing if I could do everything that some people do with Jupyter notebooks in order. I don't need to do it as much because I'm still kind of doing a lot of list programming and a lot of my DevOps, which is a lot of shell work and a lot of Python. Most of the Jupyter notebooks, they automatically include a lot of things in from either Julia or Python, like NumPy and all the dynamic stuff that's going on there. Since it has a server, it seems like it's running a lot more stuff and then just shipping it back off into the web app. That has been the biggest challenge as I've tried to kind of duplicate that. I don't really know NumPy and all the math stuff and that sort of stuff that well. So trying to do like matrix multiplications and that kind of thing that seemed to be pretty easy in the Jupyter, it's not very easy in Org. I've attempted to kind of work with Julia and the Julia support for Org is really limited it seems. That's the, So, yeah, there's some improvements we could probably make. So, I think if somebody is comfortable with Jupiter, and that seems to work, you may have to stay there. I don't know how easy. I would be curious to try to see what we could do as a community to kind of make that transition a lot easier.
[00:07:07.800] Q: Do you think any programming language is more suited to literate programming than another?
Do you think any programming language is more suited to literate programming than another? Hey, that's a real good question. A lot of mine is with Emacs Lisp, and obviously that's a first-class citizen in Emacs. Go figure, right? So that one certainly works. A lot depends on the support in the OB version for the Babel work. Python is really well supported, and yeah, Emacs Lisp. Others, Shell, you know, it works, but it's hard to get both the values, the variables out of the environment variables into your thing. So most everything is just with the results of output. So, you know, some languages aren't quite as good as others for sure. I would love to see better support for more interesting languages, especially ones that are compiled. I've been dabbling with Rust and like, yeah, what can we do to have Rust more iterative as well? So...
[00:08:21.560] Q: Do you use inline org function calls and org babel library and such?
Do you use inline org function calls in Org Babel library? So I used to do a lot of Org Babel libraries to kind of share things. Yeah. What I. But yeah. Sorry, I'm just kind of thinking through here. I've done a lot of it in the past. There were a lot of bugs in it. Oh, I don't know, eight years ago when I was playing with it. I bet things have improved a lot more and it probably saved me a lot more effort. Right now, most of the literate devops, the devops kind of stuff that I'm doing, is calling a lot of back-end programs and that sort of thing. So the Org Babel kind of has limited help in that regard. And as far as my Emacs work, well, it's just part of the session. So it's going to be available or not. So that hasn't been as helpful either. So it's kind of dropped off. But I bet if you're using something like, say, Python, it might be a lot more helpful.
[00:09:36.970] Q: How do you handle the cases where org markup may sometimes interfere with some of the code?
Another question, how do you handle the cases where markup may sometimes interfere with some of the code in places where you can't escape? Oh, okay. All right. Let me see if I understand the question correctly. When I, like, when I'm making prose, and I'm talking about a function that I've written somewhere else, like, oh, I need to work on this, or this calls this, and yeah, I'll use the tildes or the equal signs on either side to make sure it's. that it's clear that it's code. In my xref, when I'm doing a thing-at-point so it can go look for that symbol, I do look for those things at the end and strip them off so that I can mark it up and still search and it's still okay. I'm not sure if that's the answer to the question. Yeah, clarifying. I'm making prose. I'm talking about a function I've written somewhere else. Yeah, yeah, yeah, that is what I'm talking about, where a function with tildes on either end, and if I do an xref search find for that symbol, it'll strip that off and then still look for the function name, so I can still mark it correctly.
[00:11:06.220] Q: You said at the start that literate didn't catch on in corporate DevOps - why not?
You said at the start that literate didn't catch on in corporate devops, why not? You know, I guess the big thing is not everybody's using Emacs, and I don't think anybody would be doing literate programming if you didn't have Emacs. I mean, you got to have Org, or it just doesn't work that well, right?
[00:11:29.421] Q: Why not that full stack on Markdown?
Why not that full stack on Markdown? Yeah, so I often do a lot of Markdown. I have got, so I've gone through and make my markdown look a lot like Org. So my headers are the same size as org. When I mark, do the triple back ticks, you know, and specify your language, we can now colorize that. So you can kind of now have it syntactically marked up. It would be interesting if we kind of expanded on Markdown so that I could do a C-c twice on it and run that code block. That's a real good idea. Maybe we should work on that.
[00:12:22.120] Corwin's aside on orgvm
How does your management of TODOs, projects, and tasks interact with this literate mindset? I can't believe I'm going to do this, but Howard, I'm going to break right in there because I just can't. I need to talk about that. So this is I bought a domain. And really what I mean is I have this domain if somebody gets some work done in this area. I'm fascinated by this. That's mostly really actually my whole comment. But I just wanted to say, indeed, that is something maybe we should work on. Well said. I think the idea of having something that's full-stack baked around Emacs is pretty fascinating, just thinking about it as some command line tool that we can run to get some work done, and then there are documents involved, and as you know coming back to the kind of corporate sense of some of the things like there... that's a little dicey the same way a free software project raising money for itself is a little dicey. It might be a necessary way to get through certain kinds of things and like what the corporate side of the world brings to the table Here in my mind is that idea of I've got a formal process, like the process itself is an asset. If I were selling my company, that would be visibly of value to outside auditors that would help us arbitrate the sale. It's just empirical. Then tie that together with free software is maybe a little unstated ethos of doing our work in the open, and you might even deal with some of the ick factor. that you know that for a handler. Maybe so, but yeah, the downside to that is you'd still need to use Emacs to run through it, right? You know, for all those cool features, yes, it would be kind of a yet another killer app that we would have. So my domain name is orgvm, and my concept is really put org into a box, build a UI around it, it executes things that manipulate files that generate, let's say, And that's just one conceptual solution. But I really do think this is a super fun topic. And that's the space. I'm not sure my idea is that wonderful. I personally haven't soaked too many weekends into it. But I just love the way you're thinking in terms of, well, we should think more about that. Gosh, that's great to hear.
[00:14:49.520] Org and Markdown fragmentation
Yeah. Yeah, I mean, just like the talk yesterday on the future of org, it's like, let's standardize it. Because, yeah, we don't want to have org fragmented like Markdown is. Markdown is terribly fragmented. You know, everybody's got different implementations. And Yeah, what I've ended up doing, I don't know if you've seen my blog, but I often I'm always just writing an org and talking to myself and writing questions and answers and essays come out of that. And then I'll put them on my web page and easy. It's easy to publish to a web page. Well, I do the same thing at work, but I need to have it marked down. I don't start Markdown though. I'm always starting an org because that, you know, I can really work through the problem space, you know, executing code blocks and making sure they work and all of that sort of stuff. And then I just export to Markdown and then go cleaned up a little bit if I need to. And that makes sense. But I am editing Markdown afterwards. So yeah, it would be nice. It would be nice. It's just, it's all done over here as opposed to, you know, Adding it over there, but yeah, let's think about every bit as much as as that. You know, I jumped in in the spirit of yes, let's think about it. You know, your, your comments about, let's think about the workflows that work for us. Wow. Does that hit home? Right? So great. Yeah. Yeah.
[00:16:17.920] Q: How does your management of "TODOs" (projects/tasks) interact with this literate mindset, any insightful things you do on that front?
Yeah. Okay. So, uh, on to do's and that sort of thing, um. You know, a lot is the same as it was 10 years ago. At the beginning of every sprint, my company, we're using JIRA. All jokes are valid when I say that. But I just go to its web page. I copy all the tasks that I need to do that sprint. I wrote a function that takes that code and reformats it as a bunch of org tasks for me to do that I can just now have all my to-dos. And I start to work off of that from that point on. So it'll reference all the projects, and I do symlinks to every code base that I need to use. Most of the to-dos I put inside my code block is kind of just for me, just so I can remember it. Because the work that I have to do needs to be tracked a little more fine-grained, I guess. Yeah.
[00:17:30.630] Q: Do you LP also on larger projects?
Do I use LP also on larger projects? More files and nested directories? Yeah, I haven't really done nested directories, but I can now. Now that I've now kind of realized that I have the feature that I demonstrated where I can just jump to any projects for a particular project, all the org files and all the headings show up, that works in nested directories. Oh, okay, yeah, somebody's typing it in. Yeah, so it's definitely possible. And I have now, since I recorded the talk, I've been busy typing in and trying to extract that out of my init file and make it into a package. I'm looking for a good name right now. I'm calling it jobs, jump to project sections. Anyway, but yeah, I think that's kind of a nice, useful feature.
[00:18:38.936] Q: Have you used Cucumber/Gherkin/BDD and do you think it has a strong overlap to what you talked about here?
Have I used Cucumber, Gherkin, and other behavior-driven development? Have I, I mean, I do a lot of... more test driven. I do a lot of unit tests and I just put it right in my code base because if it's in my pros, I can just shove it in there. I've got the test right next to the function that I'm testing. I like that so I can kind of see it. I don't tangle it out to that same source file. I tangle it somewhere else if I tangle it at all because I don't need to. I can just, you know, C-c that and run it. I like that aspect of it. Yeah, just keeping things together. That's the big problem I have with a lot of tests is the tests are somewhere else. It's separate from the code. I like seeing it all together, but a lot depends on how extensive it is. So for a lot of the, you know, the tests that we end up doing where, you know, you've got to do a lot of mocks and a lot of startup and tear down, that kind of thing. Yeah, that doesn't help as much. And you probably have to kind of go old school on that.
[00:19:54.600] Q: What granularity are you looking for re your org files and contents, with respect to a codebase that it tangles to, or in non-coding contexts?
What granularity I'm looking for with my org files and my subcontent? That's a real good question and really subjective. I change that over time. I begin by just dumping. I just dump everything in my head into a file and then I go through and start to refine it. My goal at one point was to have an Emacs config file that was really small and simple. And boy, that just doesn't happen. It's just too fun to add things and try new ideas. And I found my files were just littered with good ideas that were half-baked and not finished. And so it's like, oh, I need to extract these and put them into different files so I can keep it clean. And so I think it's like with any code base, you're just going to keep refactoring. It's an iterative process. So having the ability to, like with an Org file, where you can archive a subtree. Okay, this idea didn't pan out. Archive it. I can come back to it later. Being able to just kind of shove it somewhere else is really a nice little feature. All right. Not a question, just a comment. We need more of your insightful posts and videos. Oh, thanks. I appreciate it. Yes, it's been a fun journey. And I think we are coming up on our time, actually. So that is just an amazing quiz. And it helps me from manipulating, as I said, or as I put it earlier, like my chance as an organizer, or not manipulating, monopolizing my chance as an organizer to just get a chance to pick your brain on all kinds of things, which is too kind, too kind. Appreciate it. It's been fun. All right, I'll try answering more questions online as well. All right, talk to you later. And we appreciate your being in a huge resource that you are to the community. Thanks to everyone for tuning in and we'll be coming over to our next talk in just a moment here. And so bear with us through the transition as we slip into, I think it's our, is that our penultimate or our ultimate talk? We have one or two left, I'm not sure. I think we've got a couple of talks left. I'm back. Thank you, Howard. And we will be moving to the next talk in about six minutes. So if you need to grab some coffee, some tea, use the time wisely. And to confirm, we will have one, two, three more talks this afternoon. So there's still plenty to go around. See you in a bit, folks.

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

Back to the talks Previous by track: PGmacs: browsing and editing PostgreSQL databases from Emacs Next by track: An example of a cohesive student workflow in Emacs Track: General