Back to the talks Previous by track: Getting detached from Emacs Next by track: Org workflows for developers Track: Development

Top 10 reasons why you should be using Eshell

Howard Abrams (he/him)

In this talk, Howard Abrams shows how eshell combines the best of Emacs Lisp and shells. Afterwards, he will handle questions via BigBlueButton.

The following image shows where the talk is in the schedule for Sun 2022-12-04. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 15-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-eshell
Discuss on IRC: #emacsconf-dev
Status: TO_INDEX_QA

Times in different timezones:

Sunday, Dec 4 2022, ~1:40 PM - 1:55 PM EST (US/Eastern)

which is the same as:

Sunday, Dec 4 2022, ~12:40 PM - 12:55 PM CST (US/Central)
Sunday, Dec 4 2022, ~11:40 AM - 11:55 AM MST (US/Mountain)
Sunday, Dec 4 2022, ~10:40 AM - 10:55 AM PST (US/Pacific)
Sunday, Dec 4 2022, ~6:40 PM - 6:55 PM UTC
Sunday, Dec 4 2022, ~7:40 PM - 7:55 PM CET (Europe/Paris)
Sunday, Dec 4 2022, ~8:40 PM - 8:55 PM EET (Europe/Athens)
Monday, Dec 5 2022, ~12:10 AM - 12:25 AM IST (Asia/Kolkata)
Monday, Dec 5 2022, ~2:40 AM - 2:55 AM +08 (Asia/Singapore)
Monday, Dec 5 2022, ~3:40 AM - 3:55 AM JST (Asia/Tokyo)

Find out how to watch and participate

Talk

00:00.000 Introduction 00:29.000 1. It’s an Emacs REPL 00:48.600 2. It’s also a shell 01:10.120 3. You can mix these two modes 03:27.560 4. Emacs is better than shell 04:36.080 5. Better regular expressions 06:13.480 6. Loops are better with predicates 07:39.640 7. Output of last command 09:08.520 8. Redirection back to Emacs 10:26.880 9. Using Emacs buffers 12:28.400 10. cd to remote systems 12:59.360 Summary

Q&A

00:31.280 Do you fall back to vterm only when needing terminal emulation? 01:56.320 Have you thought about adding the Eshell manual? 02:43.200 Can Eshell be used from Elisp? 03:33.880 How does that interplay with literate devops? 04:42.880 Do you have a strategy for getting around Eshell's lack of support for input redirection? 07:35.040 Do you have a preferred method for getting argument completion for shell commands? 09:14.320 Similarly, is it possible to get Eldoc-based completion for Elisp calls in Eshell? 10:33.720 Integrating functions into Emacs core 12:51.760 Are you the maintainer of Eshell now? No, just an interested bystander. 18:13.880 Do you ever fall back to terminals/shells outside Emacs, and if so, in what circumstances?

Listen to just the audio:

Description

While Eshell is this quick and dirty way to run external commands, its dirtiness plays into the Lisp’s malleable big ball of mud metaphor, and I have a number of quick hacks that will make you want to play in this puddle.

This will be a lightning talk that I will pre-record to show off some features in eshell I found while diving into the source code … stuff you can’t do in another terminals. Did you know that $$ is a special variable that contains the output from the last command?

Update from Howard: I wrote an expanded transcript with more code and functional links. See http://howardism.org/Technical/Emacs/eshell-why.html Want all the code? See my literate dotfiles for #emacs at https://github.com/howardabrams/hamacs/blob/main/ha-eshell.org

Discussion

Notes

Full code: https://github.com/howardabrams/hamacs/blob/main/ha-eshell.org
Longer transcript: http://howardism.org/Technical/Emacs/eshell-why.html
Yes eshell is usefull! Please help polishing and showing this stuff you found out.
Alvaro Ramirez has been doing the DWIM stuff
Regarding the not so well oiled parts of eshell. There are many efforts doing a better shell. I have the feeling we already have that in emacs already and it is just unfinished. But maybe that is just a statment about emacs in general.
Reach out to me if anyone wants to pair up and make a eshell-ext with many of the feature improvements I mentioned in my talk, that probably shouldn't clutter up the default eshell implementation.
eshell is great for running top and htop (except I can't figure out how to input the function keys)
- haha yeah i don't either
vterm isn't distracting - it has no new features to speak of

Questions and answers

Q: Do you fallback to vterm only when needing terminal emulation (ncurses/etc)? Or are there use cases or contexts where you use vterm over eshell beyond just terminal emulation needs?
- A: I do vterm mostly for SSH, but Docker builds and Ansible commands can cause a real mess of the screen, so I often run those commands in vterm ... but I'm not really working with that output.
Q: One issue I've had with eshell's TRAMP integration is that cd is host agnostic (as you point out). This means typing cd on a remote machine will cd back to $HOME on your local machine. Is there a way to cd to $HOME on the remote machine?
- A: It just isn't the way it behaves. While Eshell, with a Tramp-based cd command, will ssh "under the hood", it is temporary, as all the buffer work is local. I usually don't know what will happen, so I often need to switch to vterm for all ssh work. Which gets me upset when I encounter something that I would then like to use Eshell for (like piping the output back to my local Emacs buffer).
Q: Thank you for the missing Why eshell. Have you thought about adding it to the eshell manual?
- A: maybe I should team up with someone and improve on it
Q:Do you know if the eshell {} can be used from elisp? It could make for a nice elisp shell interface.
- A: Yes. Start with `eshell-command' and some variations on that.
Q: How does that interplay with your "literate-devops" approach, where things are done in an org buffer/document first instead of directly in the shell/terminal?
- A: the 2 are different. I use as REPL to test stuff
Q: Do you have a strategy for getting around eshell's lack of support for input redirection? (I also miss process substitution.)
- A: I have started sending output to Emacs buffers, where I can have more fun editing them than trying to get a pipe command sequence working. I wrote a function to pull a buffer back into Eshell to pipe back to something else. Pipes are problematic in Eshell.
Q: Can you call elisp functions as well (ie, not just commands)?
- A: Yes. Functions that start with eshell/ are called as if they were commands. However, all functions are available in eshell ... that is what makes it more interesting than the other comint-based term shells.
~~Q: Aren't buffers the superior pipes? --> that was meant as a comment when he was asked about pipes, not a question per se --> alright~~
- A:Howard: yes
Q:Do you have a preferred method for getting argument completion for shell commands in Eshell?
- A: Check out https://github.com/howardabrams/hamacs/blob/main/ha-eshell.org#getopts
Q: Similarly, is it possible to get Eldoc-based completion for Elisp calls in Eshell?
- A: dont know. would be great, though
Q:Do you have thoughts about https://www.masteringemacs.org/article/complete-guide-mastering-eshell#plan-9-smart-shell ? Summary: it effecitly adds a "| less" to every command so you get to see paged output if needed, except it is built into eshell.
- A: It is a cool idea, but while I tried it when Mickey first published that idea, it didn't stay in my workflow.
Q: Is $$ a built-in feature of eshell or did you add it?
- A: The Eshell built-in version of $$ doesn't always work, so I wrote an updated version that seems to work better (see https://github.com/howardabrams/hamacs/blob/main/ha-eshell.org#last-results) ... I'm pretty sure that if you do a command with a lot of output, it may not work at all, not just get the last of that output. Mine is just a better hack.
Q: Do you ever fallback to terminals/shells outside Emacs, and if so in what circumstances?
- A: I boot up with a Terminal to mount remote file systems, as my Emacs configuration isn't always stored locally on my machine. I'll admit that I sometimes leave the Emacs Garden, but doing anything interesting become frustrating when you have to leave the keyboard for the mouse.
Q: What are the less well-oiled parts of Eshell or edge case issues that you encounter if any?
- A: We should make a list and start working on them.
Q:Do you have ways to improve eshell vterm interop like sharing command history and directory tracking?
- A: I don't. If I am going to SSH somewhere, I just start vterm, and haven't thought about any interop.
Q: Where can I find your eshell/do command? Probably you also have another bunch of interesting Eshell helpers.
- https://gitlab.com/howardabrams/hamacs/-/blob/main/ha-eshell.org?plain=1#L741-761

Other comments from IRC:

Impressive. eshell is an emacs REPL! I knew I could issue some emacs commands but not this level of interactivity. Thanks!
The real elisp REPL is ielm, but eshell is more generally useful.
eshell is a REPL focused on the specific niche of shell. ielm's the pure elisp repl, and it rocks.
yes. i know ielm. just hadn't realised how powerful eshell is.
howard-abrams : every time I've watched a talk of yours over the years, Emacs/Org-mode has absorbed one more use cases of mine, and made them be in literate form. I'm down to Emacs and a web browser, so I'm looking ahead to your talk about the Web in Emacs
You can also leverage org-mode source blocks tu turn outputs into inputs to other blocks so, plenty of alternatives to pipes
Wow! Eshell is awesome! I have just learnt more tips! Thanks howard-abrams!
I think of Eshell as my universal machine REPL, i.e. not just ielm for emacs/elisp nor a shell for the machine, but both emacs/elisp and the OS/env.. In that way it's quite neat.
howard-abrams: thank you, very inspiring! I've always found 'normal' command line usage somewhat cumbersome and am certanly going to look at the code
I really like eshell but I sometimes find the aliases a bit hard to write. alias f if $ {find-file $1; for i in {cdr {flatten-tree $}} {find-file-other-window $i}} {echo "No files"} - Should probably use Elisp instead
- I think the aliases are almost completely broken, and only seem to work in the barest of cases.
  - Yes, in particular the Lisp part is broken. In combination with $*
    - I agree that you should write a function for that. Because as I mentioned in my talk, aliases don't accept $* at all.
      - Oh, they do. This alias works. I don't recall the reason why I wrote this as an alias. I probably just wanted to use an alias where possible.
        
        Huh ... that works? I need to try it out. I couldn't get that working, so that is why I wrote the function that I did.
watching your talk now; TIL /dev/kill and /devl/clip

Transcript

[00:00:00.000] I have 10 minutes to talk you into giving Eshell a second chance. Have the right perspective and expectation, and I think you’ll really enjoy it. Just remember eshell is a shell, not a terminal emulator. I use both Eshell and vterm. I’m going to talk and type fast, as I have 10 reasons for you to try Eshell again.

[00:00:29.000]

It’s an Emacs REPL.

I mean, check this out. Let’s start up Eshell here. Let’s just type a Lisp expression. It works. As a shell, the parens are kinda optional.

[00:00:48.600]

It’s also a shell.

While eshell may look like a shell, like Bash you should view it as a REPL with parenthesis-less s-expressions. This makes sense, because a shell command with options, like this ls command, looks like an s-expression.

[00:01:10.120]

You can mix these two modes.

Shells can call subshells which return their output like a function call, like this Bash command. In this Eshell example, I use the output of a text file as command line arguments to ripgrep. Notice how I use braces to state that it is a call to an eshell expression. We can mix Lisp-expressions and Shell-expressions. Allow me a contrived example. Notice I use good ol' setq to create a variable. Yes, those are global Emacs variables available everywhere. In Eshell, the wildcard actually creates a list. This variable assignment doesn’t work as you might expect, as setq in Eshell is still setq, and it assigns variables in pairs. To make a list in Eshell, we use listify: Without parens, Eshell is in “shell mode”, which means that words are strings, and variables need to be prefixed with dollar signs. A command can have both Eshell and Lisp expressions. As you can see here, I have a call to ripgrep, but part of it is an s-expression. Remember the differences: With parens, eshell treats it as Lisp, like the last line in my example. With braces, eshell follows these shell-like rules: First, if it looks like a number, it's a number. Otherwise, eshell converts it to a string (quotes, like a shell, groups words). What about this mix between functions and executables for the first word? Functions that begin with eshell are called first. Next in priority are executables on your $PATH, then matching Lisp functions. You can actually switch this order with the eshell-prefer-lisp-functions variable.

[00:03:27.560]

Emacs is actually better than shell.

If the following works, why would you call expr or bc or dc, or any of those other calculators? You can just call a Lisp expression. Why call less or more when you could call view-file? Here, I’ve aliased less to view-file. Load it up, and it shows up in an Emacs mode. Just like with less, if you hit q, you go back to your Eshell terminal. I do have an improvement, though. The problem with view-file is it takes a single file as an argument. In a shell, we might want to view more than one. So let’s make a solution to that. This function will call the first function with the first argument, and the second function with each of the rest. This allows me to make a version of less that calls view-file on the first [argument] given, but open in another window for each additional file.

[00:04:36.080]

Better regular expressions.

Can’t remember regular expressions when calling grep or some other search function? Use the rx macro. Here I call ripgrep again, but this time, I’m using a Lisp expression calling the rx macro to look for UUIDs in the files in my current directory. But I have another improvement for this. While the rx macro is freaking cool for Emacs Lisp, it doesn’t always translate to regular expressions accepted by most commands. The (I have no idea how to pronounce this) pcre2el project can convert from a Lisp regular expression to Perl-compatible regular expressions (PCRE) acceptable by most search commands. I’ve created a new macro here, prx, that translates the output of the rx macro. This allows me to type something much more readable, and probably easier to remember. Certainly easier than this freaking regular expression. I’ve got an even better improvement. The rx macro with regular expression snippets can be assigned to key words that I can then take advantage of. Now our command would be much simpler to type.

[00:06:13.480]

Loops are better with predicates.

Let’s say you want to remove the execute bit from files that have it. In a shell like bash, you need both a for loop and an if, as you can see in this example. With eshell, use a predicate to combine into a simple loop. The paren x after a file glob filters for only files marked as executable. Now here is another improvement. Since we often type loops to execute on one command, what about creating a function that can do this all in one go? This do function splits the arguments on that double colon, where the left side is a single statement to run, and the right side is a list of files. I have to append and flatten it in order for it to work. It loops through each file, creating an eshell command with the file appended. With this, I can remove the execute bit on all CSV files that have it. I see that my example wasn’t too good, as most commands like chmod accept multiple files, but you get the idea. In my final, larger form on my website, I don’t assume the command expression accepts a file as a final argument, as I can also replace underscores with the filename.

[00:07:39.640]

Output of last command.

Most shells have a special variable like $? for the exit code of the last command. While reading through the source code, I noticed that the $$ refers to the output of the last command. This seems pretty cool. However, Eshell returns true or nil when running external commands, so accessing the output from a call to ls doesn’t work as expected. But this is Emacs. We can fix that. After running any command, eshell sets these four variables. I can hook a function call after every Eshell command. Using buffer-substring, I store the output into a global variable, and extend Eshell’s special variables list. In my Emacs configuration, I turned this variable into a ring, so while $$ works, so does array sub-scripting on that variable. This allows me to run a command and use the output from that command more than once. The code for this is a bit longer, so you’ll need to see my Emacs configuration for details.

[00:09:08.520]

Redirection back to Emacs.

Output of any command can go to kill-ring (or the clipboard). Think of the implications. You don’t have to go into text selection mode. Just grab the output. In fact, with our $$ improvement, we can always copy the output from the last command to the clipboard. Better yet, let’s write the output to our engineering notebook. Here’s my idea. First, create a capture template that takes a string, or if called interactively, the region, and that does an immediate-finish after inserting that string to the default notes file. Next, create a wrapper function to call org-capture-string to run that template. Finally, we add our new function to eshell-virtual-targets. Let’s see this in action. I have a CSV file of user information. I can use grep and cut to extract some of that and write it out to this month’s engineering notebook.

[00:10:26.880]

Using Emacs buffers.

Why leave the results of eshell commands in the eshell buffer? Send the output into a buffer where you can use it. Here’s a call to ripgrep that searches for lines with email addresses using a complicated regular expression that I added to my prx macro. When I switch to this almost-grep buffer, I can turn on grep-mode. Now I can jump around as if I just called grep directly. Perhaps I’m proficient with my prx macro to filter out entries, but not good with shell commands that I can use in pipes to extract just one… the address column, for instance? Let’s just extract it, send it to a buffer called email-list, and now I can use Emacs commands that I know and love to edit the data directly. We currently have an over-sight that the Eshell’s built-in cat command doesn’t pipe buffer contents as standard in. So I created a bcat, a buffer cat, function to do this. So this command works to grab my email addresses I just extracted and send them to another program. If you’re interested, I have a more elaborate and yet simpler workflow surrounding sending data back and forth from Eshell to Emacs buffers.

[00:12:28.400]

Did I mention that you can cd to remote systems?

This command uses SSH to jump to my host, goblin, start a root session, and jump to the etc directory. Remember that Tramp can be finicky if you start blinging your remote hosts with oh-my-zshell, and funky prompts and things like that, so your mileage may vary.

[00:12:59.360] In summary: Use eshell if you want a quick way to run commands and Emacs functions as a REPL, or to run an OS program but process the output with Emacs. Keep in mind that Eshell has two types of subshells, and you can mix and match during a command call. The rx macro is really cool. Eshell loops are better with filters and predicates … if you can remember them. Take advantage of Emacs buffers to really enhance your shell experience. You’ve now seen that just like Emacs, I’ve crafted Eshell to be my own shell creation, tailored to my workflow. So, steal my spells, cast your own magic, but feel free to share your incantations back to me. I’ve gone over my time allotment, so we’ll have to continue this discussion on the intertubes. Why yes, I have joined the birdless diaspora, so toot me over there. Thanks.

Captioner: howard

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

Back to the talks Previous by track: Getting detached from Emacs Next by track: Org workflows for developers Track: Development