Getting detached from Emacs

Niklas Eklund (he/him, niklas.eklund@posteo.net)

In this talk, Niklas Eklund shows how to use detached to manage long-running asynchronous processes, including how to search through and compare session results. Afterwards, he will handle questions via BigBlueButton.

The following image shows where the talk is in the schedule for Sun 2022-12-04. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 11-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-detached
Discuss on IRC: #emacsconf-dev
Status: TO_INDEX_QA

Talk

00:00.000 Intro
00:27.120 The problem
01:15.800 My solution: detached
02:30.840 Shell
03:30.800 Compile
04:27.854 Detached list sessions
05:21.440 Narrow criteria
06:07.080 Diff sessions
06:34.040 Rich interface with properties
07:04.521 Annotation
07:45.088 Searching through sessions

Q&A

00:32.880 Can it replace ssh+tmux for persistent sessions on remote hosts?
01:16.880 I see integration with projectile in the readme. Does it also integrate with project.el?
03:51.240 Can you detach a session from shell-mode and reattach from eshell/vterm/term-mode? Or start a compile in shell-mode and attach it from compilation-mode?
05:45.120 How do you talk to detached? Could it be feasible to run a child Emacs instead of detached?
07:45.880 How does it handle processes that require user input?
09:20.000 Can you rerun a command (session?) but in another directory?
12:22.880 What are some other places where this might be useful?
16:59.720 What are you currently excited about in Emacs?
19:27.400 Bug in detached re: eshell and quotes

Description

As an enthusiastic Emacs user I would find myself replacing the usage of an external terminal emulator with the Emacs alternative interfaces. This included using packages such as dired for file management, magit for git and proced for processes. However I always kept a terminal around for running shell commands. These were commands that I knew would either, take a long time to run, produce a lot of rapid text output or that I would run on a remote machine. In the remote case I would rely on tmux to be able to detach and let the command run even when I wasn't connected.

To rid me of the need for the terminal emulator in these situations I developed the detached.el package. It is a package that builds on top of the dtach program, which provides the ability to detach and re-attach to processes, to offload Emacs from these processes. The package seamlessly integrates the ability to detach and attach into Emacs, and offers integration with many built in features such as shell, eshell, compile, org and dired.

In this talk I will demonstrate the features of this alternative way to run detached processes and how the package can leverage built in Emacs functionality to provide a great experience. The user interface will be showcased and how the processes essentially becomes text, which fits very well into Emacs.

Bio

• Blog: https://niklaseklund.srht.site/
• Source code: https://sr.ht/~niklaseklund/

My name is Niklas Eklund. I am 35 years young and I live in Gothenburg (Sweden) with my wife and our dog. In my daily work, I write code in C++ and Python. My free time I dedicate to music, board-games, improvement, and of course, Emacs.

More than 4 years has passed since I first started using Emacs and there is so much to like about it: its community, the ethics and how it encourage me to experiment and explore. Whether it is about writing a small function, or a package, the ability to mold Emacs to what makes sense to us as individuals is something to cherish.

Discussion

Questions and answers

• Q: Can it replace ssh+tmux for persistent sessions on remote hosts?
  • A: Not yet; being considered for future work.
• Q: I see integration with projectile in the readme, does it also integrate with project.el?
  • A: Not yet but should be easy to add.
• Q:  Can you detach a session from shell-mode and reattach from eshell/vterm/term-mode?  Or start a compile in shell-mode and attach it from compilation-mode?
  • A: 
• Q: How do you talk to detached? Could it be feasible to run a child emacs instead of detached? Would it make sense? Better communication maybe?
  • A:
• Q: How does it handle processes that require user input? (Usually to type y/n, etc). M-x compile is great but can't handle user input.
  • A:
• Q: Can you rerun a command (session?) but in another directory?
  • A:
• Q: What are some other places where this might be useful?  mu4e fetching mail?  Git processes started by Magit?  What things would you like to see working in a 1-2 year timeframe?
  • A: 
• Q: What are you currently excited about in emacs? 

Transcript

[00:00:00.000] Hello, everyone! Welcome to my talk, "Getting detached from Emacs". When I started to use Emacs, I quickly gravitated towards using it as much as I could. Magit, Org, Dired, a lot of new possibilities opened up. However, there was a workflow that was difficult for me to replace.

[00:00:27.120] The problem for me was running shell commands in sub-processes of Emacs, which in some situations led me to stick to using an external terminal. These situations, often revolved around long-running shell commands, either on my local machine or on a remote host. When I was on a remote host, I would also rely on using the program tmux to be able to detach from the remote process. My main concern at the time was that I didn't want to having to avoid restarting Emacs, because I needed to wait for a process to complete. However, there was of course a lot of things I was missing out on by not using Emacs.

[00:01:15.800] Therefore, my solution to resolving the issue of occasionally having to leave Emacs led me down the path of developing the package Detached. The package allows Emacs to delegate the responsibility of creating processes to the program dtach. It also makes sure to write the output of the process to a file, which we will see later on how that is being used. The package makes Emacs capable of attaching to these processes as well as managing them. In the package, each process is called a session, and inside of Emacs that is just an object with properties such as what command is being run, what working directory is used, where the output is stored, etc. The important aspect is also that these objects are being persistent, so they are stored over time. Today, I'm going to walk you through how I use the package and what advantages there are of treating processes like text.

[00:02:30.840] I'm going to start by opening up M-x shell, and I will run a command to update my package manager. Instead of pressing return, I'll simply press shift return to let Emacs delegate the execution to the dtach program. Emacs will immediately attach itself to the process, and we therefore don't perceive any difference from when running the command as a subprocess. We now have the option though to detach from the session, and later on we can of course reattach Emacs to the session. For me, this addresses the core of the problem that I had. But let’s see what’s more the new workflow inside of Emacs can bring.

[00:03:30.800] The package supports multiple user interfaces such as Eshell and Compile. I will therefore switch to the Detached project, and I will run the build command that I use. I will run it with detached-compile with the difference that I can detach from the compilation. One benefit of this new workflow is that I can get a system notification shown up here once a session has finished. Previously, I was either forced to have the terminal open so I could see it or hiding it, but then risking having forgotten it.

[00:04:27.854] How do we then see the output of a session? Get the detached-list-sessions command, and here we see the command that we just built, and we see the guix pull with an asterisk indicating that it is continuously running. If I press enter, we will get the output of the session here. And since it was run using compile, we also have compilation mode enabled here, so we could navigate between potential warnings or errors. And we see that there is a warning here.

[00:05:21.440] One thing that I have prepared here is that if I open up the user interface, we only see two sessions, but that is because we applied a filter here. So, we have actually the only sessions that are created within the last 12 hours and that are considered unique. So, if I remove the uniqueness, we also see that we have a previous build running on the main branch. So, I think that's typically normal that you might have that.

[00:06:07.080] And since the sessions can be considered text, we can just mark these two and check, does this warning exist on the main branch or not? So, we can just diff these ones and we see that the warning is only present on the emacsconf branch.

[00:06:34.040] Now, another benefit, in my opinion, of the new way of working is that I have these properties being displayed in the user interface. I can quickly see which commands are still running, what hosts they are running on, where they are running, and for how long they have been running. And if they have run, how long did it take?

[00:07:04.521] Occasionally, though, there might be even more input needed to distinguish sessions. So, what I typically do then is press A to annotate the session, I would add a "Warning found at emacsconf" And then it will show up this annotation in the echo area when I select the session.

[00:07:45.088] Another great improvement of using these sessions and consider them being text is, now we also see the guix pull completed here. Then we can also select and see that, okay, there is a lot of updates in this command, but let's not look at it now. Let's instead remember that previously last week when I ran a guix pull, I saw an Emacs package that looks interesting. I rather don't remember its full name, but it has something to do with collection. So, what I then can do is remove the 12 hour narrowing criteria, and we can see here I got sessions ranging back even to 28th of October. Since these are just to be considered text, I can-- now let's first narrow the sessions to only show the ones that run guix pull. Then I would narrow based on the output containing a regular expression. So, I remember it was something with collection. And we got one hit. Here it should be something with collection. It was emacs-flymake-collection. This is something that is a very nice feature to have. It doesn't create any overhead of having these old sessions lying around and occasionally, it can be interesting to search through them as well. So, for me, this is another example of when bringing workflows into Emacs, it often opens up new exciting possibilities. Thanks a lot for listening.

Captioner: anush

Questions or comments? Please e-mail niklas.eklund@posteo.net