Back to the talks Previous by track: Watering my (digital) plant with Emacs timers Next by track: Re-imagining the Emacs user experience with Casual Suite Track: General

Emacs as a Shell

Christopher Howard (he/him) - IRC: lispmacs, christopher@librehacker.com

Format: 38-min talk ; Q&A: IRC
Status: TO_FOLLOW_UP

00:02.940 Introduction
00:37.881 What do I mean by shell?
01:38.560 What I do not mean
04:50.160 What is a shell?
09:26.912 Launching external processes
11:57.300 Environment variables
14:54.400 Processes
17:00.180 Redirecting and pipelining input and output
20:09.440 Scripts
21:11.780 File system management
23:43.560 Networking
24:30.120 A brief tour of Eshell
34:21.128 Login shell
36:36.980 Resources
Duration: 37:13 minutes

Description

A shell, such as Bash, is fundamentally an interface to your operating system. It allows you to run programs, direct I/O, manage processes, and interact with the file system, as well as script such activities. Allowing for a few caveats, we can see that Emacs is capable of doing all these things, and therefore Emacs can be used a practical replacement for the traditional shell. This talk aims to explain this philosophy, to explore Emacs' basic shell functionality, and to address various caveats.

See also these other talks by the same speaker:

Discussion

  • a nice interface for using process filters directly sounds really useful, reminds me of emacs-piper

  • Uniline?

    • lispmacs[work]: gs-101: yes
  • feels like a museum to see someone using helm x)
    • lispmacs[work]: heh heh
    • lispmacs[work]: it was the first one I learned, now I'm hooked
    • gs-101: First one I used was Ivy + Counsel, but then I moved to the Vertico + Consult stack as it was newer.
  • lispmacs[work]: I haven't had any pressuring motivations to try anything else yet, but am open minded
  • Is there a convenient way to share shell history between Eshell and the system shell (fish in my case)?
    • lispmacs[work]: I know there is, but I don't use it. let me check the eshell modules list
    • I use atuin and eshell-atuin to share shell history across programs, shells and machines.
    • Ah yes, I gave atuin a try some time ago but I hate that it takes over full terminal for history. Not sure if that changed now though.
    • lispmacs[work]: I'm not seeing something like that in the built-in Eshell Modules List unless eshell-hist does something like that
    • i use my consult-shell-command package which is a small wrapper around async-shell-command that also suggests shell history :D https://codeberg.org/mekeor/consult-shell-command
    • atuin is a CLI utility; eshell-atuin is a third party Emacs package. (https://github.com/SqrtMinusOne/eshell-atuin/)
    • there is also https://github.com/svaante/recall by dape.el-developer with a similar goal
  • I need to look more into Eshell. Just started using native Emacs on Windows and switched from cmd.exe in shell mode to PowerShell, but it would be nice to have a better shell that I can use both on my home machines and the two work machines that run GNU/Linux that would also work on my work Windows laptop.
    • Yes, I saw that's one of the main use cases for eshell. But on windows, for some reason, git provides a bash shell. https://gitforwindows.org/
    • Yes, I was thinking to maybe hook that into shell mode. The advantage of Eshell would be that I would not have to configure shell mode at all and just use the built in functionality of Emacs no matter what system I am on.
  • Using buffers for input and output is such a killer feature
    • Do you mean in eshell or more generally in Emacs
    • both! But particularly in eshell
    • How do you use a buffer as input in eshell?
  • That looks really cool!
  • wow!
  • I don't see an eshell/@ command, I think I missed something
    • lispmacs[work]: https://codeberg.org/infrared/emacsconf-2024/src/branch/main/shell-talk.el
  • Oh, it's piping (buffer-string). I think this should be (buffer-substring-no-properties (point-min) (point-max)) Otherwise you'll pipe propertized text causing issues with shell commands. (Unless the eshell pipe somehow strips text properties, which I doubt.)
    • lispmacs[work]: oh okay, I wonder if the eshell pipe gets rid of that on its own
  • There's also https://github.com/szermatt/emacsclient-commands which has an epipe utility
  • Input redirection from buffers in eshell would be great -- actually, input redirection at all would be great.
    • isn't the "solution" to use cat input | ...?
      • What is input here?
  • what's the use case of #\ or #\<NAME> notation in eshell, as described at (info "(eshell) Arguments")?
    • It's useful for redirection. run-foo > #\
    • ah right, described at (info "(eshell) Pipelines")
  • I just can’t get eshell to stick … I keep running back to vterm with my tail between my legs
    • eat fan here -- it doesn't require an additional external c library :D
    • mekeor: I need to try it out, I read on its README that vterm is faster but I'm not sure if I actually need that speed.
    • Eat seems to handle buffer resizes better
  • lispmacs[work]: I think an important point is to just use Eshell where you find it useful - same with all the other Emacs tools. Of course, you should explore some of Eshell's Emacs/Elisp integration features
  • The integrations with the rest of Emacs is probably what makes Eshell worth it once you get used to it. I am going to read up and try it out.

  • Thanks

    • einar_m: Thank you! :-)
    • lounge-664: thanks chris good talk
    • chum-cha: This was fantastic, thanks lispmacs[work]!
    • mretka: M-x clap 👏👏👏
    • dubs: Great talk
    • [14:32:57] * gs-101 claps
    • lispmacs[work]: thank you
    • johnhamelink: Thank you :)
    • jsiegel62: Thanks!
    • lispmacs[work]: 🙇
    • karthik: Thank you
    • mraabo: Very nice, thank you!
    • lounge-267: ty, lispmacs.
    • [14:33:42] * inkpotmonkey 🙇
    • oylenshpeegul: 👏
    • ankit: Thank you, I've picked up a lot of things that I'll try to use in my workflow.
    • einar_m: Thank you for the inspiration, lismacs!

  • YouTube comments:

    • 11:28 This should have been a shell-command instead of an eshell-command.
    • is the voice generated by a software program?
      • not judging by the no-pop-filter-noises
      • bro forgot he was giving a talk and thought he was narrating an audiobook lmao
      • be kind
    • Surprised you didn't mention "vterm" or "multi-vterm" especially if some shell commands have fancy ansi output.
    • I have a question regarding command line argument passing with emacsclient. In my fish.config file create a function called "e' so I can just type e sometext.txt emacsclient -c "$argv[1]" But I want this actually emacsclient -c -a "" --eval "(set-frame-size (selected-frame) 106 42)" "$argv[1]" so $ e sometext.txt

      That works but below doesn't work and I get an elisp message

      If I use that emacsclient with --eval

      $ e sometext.txt

      says "nil"... and I don't get emacsclient to open up with the text file I want. It is very annoying and I guess once you put --eval you can't do argument passing but goes into an elisp mode I assume.

      • I think the scope of this talk was mentioned in the beginning.
    • 0:20 I think of Emacs as an OS

Transcript

[00:00:02.940] Introduction
Welcome to Emacs as a Shell, a talk by Christopher Howard for Emacs Conference 2024. In this talk, I would like to explore, or advocate for, a particular perspective. I want to encourage people to think of Emacs not as simply an editor or a development environment, but rather as a shell, or at least something that allows us to do most of the things that we might otherwise want to do from a shell.
[00:00:37.881] What do I mean by shell?
What do I mean by shell? By shell, I mean basically an interface that allows us to interact with the rest of our system by entering commands. That definition is, perhaps, a little too broad, and so I will try to narrow it down with a list of features that, historically, we have come to expect from a shell. The Bash shell is one very portable and well-known shell, and for many of us it is maybe the prototypical example. But in the past there have been many other shells, and there are other shells available today. If we are willing to be flexible in our thinking, we can think of Emacs as being a shell, or at least providing most of the functionality that we expect from a shell.
[00:01:38.560] What I do not mean
Before further expanding on this idea, I must emphasize what I do not mean. First of all, I am not talking about running Bash, or some other external shell, from within Emacs, although this is certainly possible. I am arguing, rather, for using Emacs as a shell, instead of other shells. Second, I do not mean running a terminal emulator from within Emacs. Emacs has a built-in terminal emulator, but this is not what I mean. A terminal emulator is essentially a program designed to control the cursor and text appearance in response to various control codes in order to mimic a terminal display device. There are certainly legitimate reasons to do this. Nevertheless, in general, it does not make much sense to run a terminal emulator within Emacs, because Emacs has its own commands for controlling the cursor and text appearance. Also, due to the way Emacs was designed historically, Emacs itself believes that it is running on a terminal. So you end up with layers upon layers of terminal emulation. Anyhow, at the end of the day, Emacs will not perform as well as a dedicated terminal emulator program. I also think that, as we try to force ANSI terminal emulation into our Emacs workflow, this ultimately will be a hindrance to us in taking advantage of the natural and pleasant interfaces that are already available to us within Emacs. In brief, if your goal is simply to figure out how to be able to do all your normal Bash command line wizardry from within an Emacs window instead of a GNOME console window, you are headed down a different set of train tracks than I am. Also, something which I fear may confuse the issue for some viewers is the fact that Emacs ships with its own unique built-in shell, called the Emacs shell, or Eshell. Eshell aims to be a legitimate shell, and provides a very similar experience to other shells like Bash, while being well integrated into the Emacs interface, and without giving up the power of the Emacs Lisp engine. Eshell will be mentioned multiple times in this talk. The entire talk could, in fact, be about Eshell, except that I want the talk to cover all aspects of Emacs shell-like functionality through its other tools, such as interactive commands and special modes. So, hopefully we can keep distinct in our mind the ideas of Emacs as a shell versus the Emacs shell, though the latter is an important part of the former.
[00:04:50.160] What is a shell?
Let's get back to the fundamental idea of what is a shell. In the broadest definition, a shell is an interface which allows you to interact with your operating system through commands. However, from a historical perspective, there are a few basic capabilities which we expect to be part of every shell. First of all, the shell provides a means of launching external programs. Some internal or built-in commands might also be made available. Second, the shell provides a means of managing environment variables. In the past, environment variables often played a critical role as a means of passing in options, file names, device names, and suchlike to external programs. This is not quite as common today, but the environment still plays a critical role in managing things such as the path to executables and libraries, as well as various other user, desktop, and system settings. The shell modifies the environment and passes it on to external programs. Historically, job control was expected to be either a function of the shell, or easily accessible from it. Usually today, our personal computing is not batch-oriented. But typically, shells can run multiple processes simultaneously, as well as provide means to suspend and terminate processes, which are useful features. Shells should be able to redirect and pipeline process input and output. This allows the user to connect process input and output with files, devices, or other processes. Finally, shells are expected to have some limited scripting capability, such as the POSIX-compliant set of program statements and conditionals that Bash provides. As command-line wizards, there are a number of tasks we expect to be able to do quickly and easily from our shell, even though these tasks are not the domain of the shell itself. A common task is file management and navigation. We quickly navigate and manipulate the file system with standard utilities that do things like change the current working directory, rename files, move files, and delete files. We usually expect to have access to some additional process management utilities. These allow us to do things such as find out the status of all processes running on the system, and send signals to processes. Finally, we expect to have access to some basic networking utilities. For example, we should be able to run commands that set up network interfaces, ping computers, and download files. With a little reflection, we can see that Emacs can provide all or nearly all of the functionality we have described so far. And the functionality can be called conveniently through one of several methods. Either a normal interactive call, like M-x something, or a call to an elisp function, or through Eshell commands, or through some special buffer mode, such as the directory editor, which provides its own interface to some functionality. It may be going too far to say that Emacs is a full replacement for shells like Bash. Nevertheless, we can see that Emacs can do most of the things that we might otherwise do with our shell.
[00:09:26.912] Launching external processes
Let us give some examples. First, can Emacs launch external commands? Of course. Now, there are something like a half-dozen different ways to do that within Emacs, and some are more convenient than others. From any Elisp program, we can call functions like make-process and call-process to launch external processes. These, however, generally are not convenient for quick, one-off commands. Another option would be to run Eshell, which would allow us to call the external program from a familiar command line prompt. If we do not actually want to drop into Eshell just to run one command, we also have the interactive command, eshell-command, which would allow us to call the external program from a familiar command line prompt. If we do not actually want to drop an Eshell just to run one command, as I just mentioned, we also have the interactive command eshell-command, which allows us to enter in a one-off command and run that immediately. And finally, there is also an interactive command called simply shell-command. Shell command is like Eshell command, but instead passes the command off to our system shell, for example, bash. This is cheating, of course, but it might be useful or convenient in some scenarios.
[00:11:57.300] Environment variables
Regarding environment variables, Emacs can read and manipulate the environment variables, which in turn get passed on to processes which it launches. The general-purpose interactive commands for this are getenv and setenv. These commands deal with the one environment that is available throughout all parts of your running Emacs session. In other words, these functions deal with a global environment, which is the same wherever you are running getenv or setenv. An important exception is that every instance of Eshell maintains a distinct environment that will not be affected by setenv calls run in other buffers. Also, Eshell has some additional syntax for dealing with its environment, including the set and export syntax. Regarding job control and process management, Emacs does not provide job control in the way that Bash users are used to. We can, however, launch asynchronous processes, and do various things to them. From Eshell, or an eshell-command call, we can append the ampersand symbol to the command, and this will cause the process to run asynchronously in a dedicated buffer. Now, if the command is launched from Eshell, it will not actually run in a separate buffer, but the output will go to the Eshell buffer.
[00:14:54.400] Processes
We can run the interactive command list-processes to see all the processes running for our current Emacs session. In Eshell, we can run the command "jobs" to get the same list. This will show the process name, process buffer name, process ID, and some other information. We can select the process buffer in the process list to bring up that process buffer. We can also use the interactive command signal-process to send any signal to a process, including "stop" to suspend the process, "continue" to resume the process, and "interrupt" or kill to terminate the process.
[00:17:00.180] Redirecting and pipelining input and output
Regarding redirecting and pipelining input and output, Eshell does support redirection similar to Bash, so you can overwrite and append to files and some other objects. Input redirection is not yet implemented, but it is on the Eshell to-do list. Eshell also has pipes. The default pipe, which uses the familiar vertical bar symbol, pipes the data between the commands using an intermediate Emacs buffer. This, while usually quite practical, is less efficient than the system pipe. Therefore, Eshell also makes available a star-modified version, which uses the system pipe through a call to your system shell. So we can do things like direct output to a file. We're unfortunately not able to do input redirection, but we can use pipes. Elisp can manipulate and tie together processes in various ways, such as process filters and pipe processes, but I won't attempt to cover that. I feel like you should mention again that we have two kinds of pipes here available. So this pipe, the standard one, will pipe the data through Emacs buffers. That's very practical in most cases, but it is less efficient than piping through the system pipe. So Eshell makes available another symbol for that, star, vertical bar, that allows you to explicitly use the system pipe. Regarding scripting: Of course, using Emacs makes available all the power of the Elisp API and third-party packages, so we have that out of the gate. Eshell also has control flow statements, like an "if" construct and a "for" construct. See the Eshell info manual, section 3.7, for more details.
[00:20:09.440] Scripts
And if you wish to write a script entirely in Eshell syntax, and store it in a separate file, this is possible with recent versions of Emacs. Here's an example of a brief script that I wrote. Unfortunately, an eshell mode for proper syntax highlighting is not yet available, but hopefully that will be forthcoming. Note that Eshell syntax allows elisp forms to be interspersed with regular command form for additional scripting power. We will discuss this a little more later.
[00:21:11.780] File system management
Regarding file system management. In Emacs, many of the common file system operations are available as interactive commands. For example, M-x cd, to change your buffer's current working directory, and other M-x commands such as make-directory chmod, and delete-file. Of course, you can also drop into Eshell, or use M-x eshell-command to run the usual external commands for file system manipulation. Also, a file manager is built into Emacs, which can be run by calling M-x dired. The directory editor is powerful, but it is a bit strange to folks expecting something like Midnight Commander or the GNOME file manager. It gives us a number of helpful features like the ability to mark files, and to run elisp functions on them, and some other interesting ways to manipulate and rename the files. However, third-party Emacs extensions such as Midnight Commander Mode and Sunrise Commander are available to provide a Midnight Commander experience, for those who prefer that sort of file management. Emacs also has the nifty TRAMP functionality built in, which allows you, most of the time, to easily edit files on other computers, as well as manipulate the file system. This transparently works through SSH and some other protocols that you can specify.
[00:23:43.560] Networking
Regarding networking features, I don't have a lot of interesting things to say about this at the present, so I'll skip through this quickly. But if you do a little research, you will see that Emacs has a lot of functionality relating to making network connections, interacting with the web, and such like, both built-in and in available packages, as well as modes for doing things like Web browsing and Gemini browsing. And of course, you can run the usual standard networking commands for your system through Eshell.
[00:24:30.120] A brief tour of Eshell
So having put forward the main arguments for this talk, I would like to take some time now to give a brief tour of a few of the features of Eshell, the Emacs shell. It bears emphasizing that Eshell is not a drop-in replacement for Bash, or even a Bash clone, though I believe the developers are trying to make much of the syntax very similar. Also, Eshell is not a terminal emulator, and it will not display correctly applications which use advanced ANSI control codes. However, Eshell can be configured to be aware of such applications, and to run them automatically within the Emacs terminal emulator when launched. See section 5.1 of the Eshell manual titled Visual Commands. Though Eshell is not Bash, it has multiple features, pertaining mainly to its by-design Emacs integration, which may make it more appealing to use than Bash or another shell. For one, Eshell allows entering commands on the command line that are space and new line separated, without parentheses. Of course, all the other shells do this. But within Eshell, it is possible to enter internal Emacs functions, as well as external commands. This allows us to do things like this. As far as I understand, it is possible to enter any Emacs function on the Eshell command line. However, some special syntax may be required if you are trying to pass in something that is not a string or a number. As you might have noticed in the last example, Eshell makes it possible to use an Emacs buffer as a sink for output. It also allows using a buffer as a source of input, though this is slightly more complicated, since the buffer must be converted to a string first. I have distilled this down into my own function, named with the "at" symbol. And I will provide the brief snippet of code for this later. So to give an example, here's our messages buffer. And from Eshell, we can do something like this. Let's say here we wanted to grab our messages buffer to see everything that we had been loading during the startup process. So you can see how that could be very handy in a number of scenarios. I wanted to briefly mention that we have a helpful function here called eshell-insert-buffer-name, which allows us to insert a buffer name into the current buffer at point using completion, which can save you a lot of typing. Another nice feature of Eshell is that it allows integrating ELisp into the command line call. Let's give another example. Say we wanted to echo the date to an event file or an event log. I should probably take a moment to explain this asterisk that I'm occasionally using. So since Emacs, or excuse me, since Eshell can use internal or external Emacs, excuse me, internal Emacs commands or external commands, it may sometimes be necessary to clarify which one you want to use, since the names may overlap. Since my Eshell is configured by default to prefer the internal Emacs functions, then sometimes I have to use the asterisk to specify that I want the external version. Here I can insert a bit of Elisp, and then redirect the output to the event log. Last, I want to mention that there are some optional Eshell modules in Emacs, not turned on by default, which provide additional nifty features. On my system, I have most of the optional modules turned on. An interesting module is eshell-smart, which does various things with cursor positioning and scrolling, so as to make editing commands and reviewing output easier. Let's say I was to change directory to my boot directory and use a command which involves lots of output. You'll notice right away that the cursor positioning is set such that I'm immediately able to view the top of the output. Also, I'm able to use the space bar to page through the output. So this is an opinionated feature, which assumes that you're likely going to want to review the output immediately, or that you often will. Of course, you can always jump to the end. Also, after a command is entered, the cursor is immediately repositioned to make it easy to edit the command. And also, if I don't want to edit the command, and I do not want to review the output, I can simply start typing another command. So that covers the brief tour of Eshell features. And that basically ends my talk.
[00:34:21.128] Login shell
However, a handful of viewers might be wondering, is it possible to set Emacs to be my login shell to completely replace bash in your login experience? The answer is yes, but in practice there are various difficulties involved which might make it not worth the trouble. Before doing this, you'll have to answer a few initial questions. Do you want to make a new Emacs instance every time you log in, or do you want it to connect to an Emacs server? Which is popular among Emacs users, to reuse the session, or to connect to the existing session. Also, do you want a different result, whether in graphical or a terminal environment? And are you okay with your initialization file being run every time you log in, including every new tab you open in a terminal emulator? If we assume that you are using a system with /etc/passwd user management, you get one field to specify the name of the shell program that you want to use, and no arguments are allowed. So maybe you can see how this might be challenging, depending on your answers to the previous questions. You can work around these issues in various ways, like modifying the authentication system, or by specifying a script for your login shell. But if your normal workflow is to simply log in and start Emacs and run that Emacs session until your next reboot, then it probably isn't worth the bother.
[00:36:36.980] Resources
So thank you for listening to my talk, Emacs as a Shell, by Christopher Howard for Emacs Conference 2024. At the bottom of this page, you can see a link to the repository containing the brief amount of code that was featured here in this video, as well as a link to my personal Gemini gemlog, as well as to a Web portal version of that. Thank you.

Captioner: mark

Questions or comments? Please e-mail christopher@librehacker.com

Back to the talks Previous by track: Watering my (digital) plant with Emacs timers Next by track: Re-imagining the Emacs user experience with Casual Suite Track: General