(Un)entangling projects and repos

Alexey Bochkarev (he/him) - https://www.bochkarev.io, @bochkarev@qoto.org (Mastodon)

Format: 13-min talk; Q&A: Etherpad
Status: Q&A to be extracted from the room recordings

Description

Emacs provides a few excellent tools for working on projects through all their key stages. Orgmode is great for brainstorming, structuring and maintaining TODO lists, tracking time, organizing notes, and writing memos or reports. Many major modes help writing code, magit makes version control almost frictionless, and projectile helps with project management and navigation. However, I found a few situations when I wanted to separate the concepts of "project" and "source storage" (say, having a few version control repositories associated with a single "generalized project").

In this talk, I would like:

1. to describe a specific example of such situation,

2. discuss a workflow aimed at managing such "generalized" projects and present my solution, based on a very simple ELisp "glue" on top of the functionality provided by package projectile.

For example, consider a research project (think: applied mathematics with a heavy part of computational experiments). It might consist of:

• The ``paper'' draft: some sort of final report source, usually in LaTeX format, or orgmode exported to PDF via LaTeX. Version controlled by git.

• Numerical experiments: a separate folder, or even a separate git repo. Contains the source code for numerical experiments and the related technical documentation. Will be published along with the paper.

• A collection of intermediate memos (notes) sent to collaborators.

• A collection of "raw" notes (lab journal), regarding what did I try and especially what did NOT work and in which ways.

This setting raises a few problems that all boil down to the necessity of having an easily accessible private notes file(s) associated with a few repositories at the same time outside of these repos. This way one can:

• Maintain more granular project structure and TODOs while still having more concise TODO lists for the colleagues on a per-repository basis.

• Maintain (project-specific) private technical notes, and maybe a full lab journal both describing the "big picture" of the project and containing the technical information.

• Keep time tracking data private and outside of the source repositories,

• Capture thoughts and TODOs to a single place from across a few specific repositories.

I propose to solve this problem by associating a single "notes folder" and a main .org file to each repository using the standard mechanism of directory-local variables on top of what is already provided by projectile package.

Discussion

Questions and answers

• Q: Do you use these unentangling techniques in a blog or hosting a zettelkasten?
  • A: Well, I try to keep my "private notes" in something that might qualify as a Zettelkasten, yes. I wouldn't say I 'host' it --- it's not online. But yes, the whole point is that these "private" notes are interconnected in a Zettelkasten-y way (using org-roam package)
    • Denote notes Silo features might be useful with your workflow
      • https://protesilaos.com/emacs/denote#h:e43baf95-f201-4fec-8620-c0eb5eaa1c85
        • oh, thanks --- I'll have a look!
• Q: What is the biggest unhappiness you haven't figured out for your current workflow?
  • A: Maybe I am still on the fence re: where do I structure my TODOs and clock time. I tried to play around with the idea that I structure the work in a repo, and then when I "clock in" it saves time to a separate notes file instead... but it seemed a little too complicated, to my taste.
    • I feel that the time tracking also kind of annoying, especially you forgot to clock on and all the things mess up. So right now I'm just using a Pomodoro technique, 25 minutes, done, rest, 25 minutes, rest, and kind of repeating that. And I'm quite happy with that.
      • wait, what's that? 'org-pomodoro'?. sounds interesting...
        • It's not, you know, special for Org Mode. It's kind of a general technique which you focus on a small task for just 25 minutes, but at the time you're super focused, 100% focused, and after that five minutes you rest, and you're kind of repeating these patterns over long sections. You can do four, five, six of those sections, and it helps me to focus over relateive long time.
          • I also feel this might be something really useful. Just haven't found a way to incorporate it into my workflow
            • for me it's quite simple is I can just use a simple stopwatch that every 25 minutes stop and reminde me  a rest. I believe there's a lot of fancy clock specialized on this this type of technique it's at the core of this concept is really not a complex idea.
              • wait, I'm confused. So, that's outside Emacs right?
                • Yes, the concept is outside of Emacs, but I saw people using this package. Let me search,: https://github.com/marcinkoziej/org-pomodoro <-- yeah, that one. Maybe I'll have a look, thanks!
                  • Yeah, it's, again, if you're familiar with the sports, it's kind of making your long hard working, breaking into a small section, but I feel it's, you have more kind of energy over a long term, yeah.
    • I like Using a weekly GTD log files for my TODO. That way I can look back at them and not have my GTD to big. I like to pull daily tasks from agenda
      • and what do you do to transfer stuff between the weeks --- a manual review? 
• Q: Do you use project.el features as well, or just projectile.el ones?
  • A: Ugh. OK, I am at that point where I am not sure any more it is pretty well integrated to my Doom Emacs, so I am not sure which one is that...

Notes

• GNU Hyperbole already supports this with directory-specific quick access button files (which can be Org files).  These can connect to any number and type of document artifacts, including projects, repos, directories, etc.  You don't need to put any code in dir-locals either.  The directory/project-specific tags jumping (automatically selecting appropriate TAGS files) is also built-in. Have a look.
  • Yes, there's clearly a few ways to achieve this. I have a feeling Hyperbole achieves this, and much more. I wanted to have something simpler, somehow.   (Yes, you seem to have some very efficient techniques down; maybe you could utilize both).  Thanks for the talk, it was good. Thanks for the suggestion, tho!

Transcript (unedited)

Hello, I'm Alexey Bychkadov, and I'm talking about unentangling projects and repositories, or maybe entangling them, depending on how you look at that. So there's going to be a short workflow note. I work as a researcher, So there are 3 main components to my work, I guess. First, I think, so I try to come up with a new ideas that usually results in some collection of notes I have. Second, I try things out. So it usually means that I write code. And third, I communicate. So I prepare papers, presentations, memos, and so on and so forth. And so The workflow problem I had is sometimes all this does not really fit into a concept of a single repository per project. So I might want to have, for example, a source code in 1 repository and then I would like to have a paper in another 1 and then I want to have a collection of notes somewhere unrelated to those 2. Emacs is pretty good at supporting your workflows and I figured I should share what I used and what works for me. So, from the technical perspective, things are pretty easy. So I use a collection of pretty standard components of Emacs. So it's a projectile org mode with this capture templates and other things. Then I sustained a collection of nodes in something that is called org-roam, which is essentially it's a glorified collection of org mode files. Then I used directory local variables, maybe a C text to jump through the source code and very, very little LELisp glue to make this all work, but that's not really rocket science. So that's the workflow I would like to talk about today. So what I mean by all that, it's pretty straightforward to make Emacs, to make it easy to jump around a single repository in Emacs. So if I, Now I have Doom Emacs, but that's not really specific to a Doom that'll work in any Emacs configuration. Well, key bindings might be different, but that's not the point, I guess, for the workflow. So if I hit space 2 times, I have all the list of files within my project, right? So if I create a couple of custom shortcuts, so if I press a magic button, hyper-OP, don't worry about hyper-key. So I want it to have a modifier key all to myself, so that would, no program on my computer would use that except Emacs. Emacs would use that only when I tell it to, so I have a hyper key instead of caps lock. That's pretty easy to do in GNU Linux system. So when I press this magic keys, I have a menu that's a normal key binding. Yeah, essentially an Emacs. And if I hit, for example, R, I end up in a readme file within this specific repository I was sitting in, right? So if I want to document something real quick, I go to the readme file. Then I could go to a change log file, right? So I have a list of changes and the way it works usually, for example, if I'm working in some code, I created a couple of dummy files in there, so I'm working in some code and then I implemented something and I can just use the org mode capture mechanisms to keep track of what I want to discuss with colleagues next time. For example, I could just hit capture repo specific changelog entry and I implemented a feature and I can continue working without this context switching. And then if I want to go to the change log, well, it is there. And next time I talk to the colleagues about the source code, I can open the change log and go through entries 1 by 1 and discuss what I haven't implemented last time. I could go to project specific, sorry, to repo specific to-do list. And I have list of to-dos that would leave within a repository. And for example, I could have a high level structure here, work distribution between team members and other things that sort of face outer world, so to speak. And of course, there are very many ways to jump through the source code conveniently. I ended up not using language servers I use a special program called ctags and so the way it works is just I call projectile regenerate tags and it creates the special tags file within the repository and then I can again run it I usually just hit a single keystroke and here is all the symbols that are there in my source code, regardless of the language, right? So I can jump to the main function and that'll be a C++ file. Or I could go to the super function, which I had in my Python file. And this comes in pretty convenient if I have a mixture of languages. Sometimes I can have some algorithm specific code in Julia, and then I can have some Python glue within the same source code repository, it makes it really convenient to jump between all of those. But I have a few problems here. So just to give you a little bit of context, for example, here is a real project that corresponds to real paper. I have a single note about that project where I keep all the things related to that project here, but that's a private note. So for example, again, I hit a special key that invokes my org-roam function that gives me a menu of my notes. And so here is the paper, essentially. And I can have a paper timeline, and I can have a list of all the dates what happened to the paper with links to my email, right? So for example if I hit this link that will open a specific email and that doesn't work outside of my computer, doesn't make any sense to keep it in the outer world facing repository, for example. So that's something to myself, right? Sometimes I want to have like this list of working notes, right, that contain like, for example, yeah, I might produce this kind of things for internal discussion, right? It has some marks, it has some margin notes and things like that. Maybe again, health-based ideas that may or may not end up in a repository, in the final paper or in a source code, but still I want to have it somewhere. And well, long story short, I need a project folder that would be unrelated to the source code or to the source code repository or to the paper itself or a final report, right? And 1 way, as usual, there are multiple ways to achieve that, I suppose. And 1 way to do that is, so I create a special folder within my org-roam storage. So it's a special folder outside of Henry Postories that got backed up to my hard drive with certain redundancy, but I don't really need like version control, full blown version control for that. I'm okay with just having a couple of backups, right? So this is the folder you see here. So PKB stands for personal knowledge base, and I have a folder project notes in there, right? So, and How does it work? So I have a folder per project in there, essentially. And here I can have all the stuff that kind of belongs to me and I do not publish it anywhere. And then, For example, a source code repository knows about that folder and a paper repository knows about that folder. And anything else that might leave in separate places all over my system can know about that folder. How do I achieve that? Well, essentially this is 1 of the use cases for the directory local variables, right? So for example, how does it work from the user perspective? So if I hit a special key, oh, sorry, if I hit a special key, that would be open project. And then for example, org mode file, right? So this is my personal notes about the maxconf, not specifically about this very talk, but I can have, you know, the house baked ideas here again, presentation tools and things like that. And how does that happen? If we try to like look at the code, the e-list magic here, what is happening is it's just a couple of lines of code, in fact, so let me just press Control, help key. And so the key I was pressing is open project or my file. And so what we see here, there is a single, so it's just a call to a find file function. So I opened that file and there is a special function that figures out what is the like umbrella project nose file and that's, again, that's very easy. So essentially if a variable describing this, the name for that project is defined, then I use that as my project folder name. If not, I take the project name from the project tile. Well, that's pretty much it. And how do I define this variable? Is essentially there is this magical file in a folder called dear locals, elist. And I just put it there. And then whenever I go into that folder or any of its children folders, I get this variable defined. And that's pretty much it. That's how it works for me. I guess 1 thing that I wanted to emphasize specifically about that is of course, it is a time tracking, right? So what is I find especially important when I work in something and I want to clock time, I usually do not want this information to be in a source code repository or in a paper repository because other people I work with will not be particularly happy about that, especially if most of them do not use Emacs and they'll see this long list of org clocked data and that doesn't look nice in a plain text format. So what I usually do if I want to clock in some time and then later analyze what I've been spending time on, so I go to my org mode file and I go to the, my current project to-dos and I clock in there. And that's how it works. So again, what comes in handy, if I hit Control O, I just go back to the file I jumped in into and that's I jumped from so that's also pretty handy. So again no no rocket science in there. So I create a directory local variable that helps me to figure out what umbrella project does this particular folder belongs to. And this way I make Emacs aware of, for example, facts like, so this source code belongs to that project. And this paper, this repository with a paper also belongs to that project. And I can have capture templates that would save my notes into the my private notes file and my to-dos and go to my private note files and so on and so forth. So I find it pretty simple but that really helps to reduce this context switching. And I don't believe it allows me to save time, but that probably helps me to stay focused. And this is what is really important, I believe. So thank you very much. And if you have any comments or suggestions to that, please do jump into the discussion. Yeah, after the talk, thank you.

Questions or comments? Please e-mail emacsconf-org-private@gnu.org