Beguiling Emacs: Guile-Emacs relaunched!
Robin Templeton (they/them)
IRC: robin on libera.chat; Fediverse: @lispwitch@octodon.social; Matrix: @terpri:matrix.org; website: http://terpri.org/; mail: robin@terpri.org
Format: 16-min talk ; Q&A: BigBlueButton conference room
Etherpad: https://pad.emacsconf.org/2024-guile
Status: TO_CAPTION_QA
Talk
Q&A
03:01.480 Q: About fibers: My understanding is that the problem with making Elisp concurrent is that none of the data structures (buffer, cons, vector, window etc) are concurrency-safe. How do fibers help with this? 04:28.080 Q: Do you have a rough idea of how much of Guile is written in C? 06:19.240 Q: A Common Lisp implementation for Guile sounds really cool! Is there already work on this underway? 08:34.480 Q: Did switching from guile 2 to 3 give any performance benefits? 10:46.440 Q: Do you know if the Emacs maintainers are interested in switching to Guile as the engine for Emacs Lisp? 12:30.000 Q: Do you think guile-emacs will be able to use or (collaborate with) some of the other awesome projects around Emacs Lisp? 15:04.200 Q: SBCL, ...You mentioned Robert Strandh's SICL along with SBCL---does that work help with the implementation of CL in Guile?
Description
The Guile-Emacs project seeks to develop new foundations for Emacs, building on forty years of development to prepare the way for the next forty. Guile-Emacs brings Emacs and Guile together by providing a new Elisp implementation based on Guile's compiler technology, serving as the basis for a more expressive and extensible version of Elisp. We envision Guile and Emacs being co-developed in a sort of mutualism, with Emacs becoming the heart of a "Lisp machine for the 21st century" and with Guile fulfilling the promised role of Lisp from the GNU Manifesto.
Guile-Emacs is now being developed by a democratic workers cooperative, focused on development of the project itself and related components of the GNU system.
In this talk, I'll cover:
- What is Guile-Emacs, in terms of its goals and general architecture? Why is Guile well-suited to the goals of the project?
- How did the Guile-Emacs project begin, and what is its status today?
- What are the immediate tasks for Guile-Emacs development, and how will they improve Guile-Emacs itself as well as its component projects?
- How are our long-term goals for Guile-Emacs connected to the spirit of Emacs and the GNU Project in general? What do we envision for the future of GNU Emacs?
- How can you get involved with and support this effort?
Along the way, we'll show live demos of Guile-Emacs itself and its extensions to Emacs Lisp.
About the speaker:
Robin Templeton is a free software advocate who enjoys programming language design and exploring system architecture. Their fascination with Emacs and Lisp lead them to begin work on the then-hypothetical Guile-Emacs project during their university studies. If given the opportunity and interest, they will bless an interested listener with a treasure trove of obscure Lisp history.
Guile-Emacs seeks to provide new foundations for Emacs, integrating Emacs and Guile via a new Elisp implementation. We envision a significant role for Emacs within the GNU Project, becoming a central part of a "Lisp machine for the 21st century".
Discussion
Questions and answers
- Q: About fibers: My understanding is that the problem with making Elisp concurrent is that none of the data structures (buffer, cons, vector, window etc) are concurrency-safe. How do fibers help with this?
- A: Fibers do not provide thread-safety for any of the existing data structures. They are useful for building things that don\'t use Emacs data structures, like a network client that reads a \"stream\". Fibers can introduce new data structures that are thread-safe, like \"thread-local buffers\".
- Q: \<ramin> You mentnioned that Emacs is roughly 25% written in C.
Do you have a rough idea of how much of Guile is written in C? Could
it be an improvement to make libguile a dependency of Emacs?
- A: The problem is not the quantity of C, but that there is C involved at every single level/layer of computation in Emacs. This makes it difficult to use concepts like delimited continuations, that can make it easy to implement Emacs concepts more simply.
- A: About guile: it\'s about 1/2 C.
- Q: A Common Lisp implementation for Guile sounds really cool! Is there already work on this underway?
- A: Yes, Robin is working on it, but focuses more on research on how to do a polyglot lisp environnement where elisp and common-lisp can work together (with lisp 1 vs lisp 2 issues with different name spaces, package and module system interaction, ...). If you are interested, the guile project will work on it.
- Q: As someone who\'s kinda new to the concept of Guile, is the
primary goal to transpile Emacs Lisp into Guile bytecode rather than
primarily focusing on adding support for writing code in Guile\'s
Scheme interface in particular?
- A:
- Q: Did switching from guile 2 to 3 give any performance benefits?
- A: Not benchmarking stuff here yet, because guile emacs has too much overhead involved in conforming to Emacs. Have not noticed a perceptible change. Based on the Gabriel benchmark results, it might have benefited a bit, but for Emacs we don\'t know yet. Lowering the overhead is the best place to focus on to optimize guile-emacs.
- Q: Do you know if the Emacs maintainers are interested in switching to Guile as the engine for Emacs Lisp?
- A: Previous maintainers were cautiously optimistic and interested. Issues might be cross-platform compatibility.
- Q: Do you think guile-emacs will be able to use or (collaborate
with) some of the other awesome projects around Emacs-Lisp, like the
gypsum project presented earlier today, or Andrea Corallo\'s efforts
to make Emacs-Lisp more suitable for native compilation? Or even
something like the renewed PreScheme efforts?
- A: Gypsum has a different focus. Guile tries to improve lisp instead of replacing lisp in any way. But some code can be shared, especially if some parts of emacs are rewritten in lisp
- A: For libgccjit: it is accelerating the interpreter, which is not great in the first place. So no direct relationship.
- A: For prescheme: it is a useful tool to look at, making it easier to upstream one day.
- Q: SBCL, ...You mentioned Robert Strandh\'s SICL along with SBCL---does that work help with the implementation of CL in Guile?
- A: Time consuming part are the DSL from common lisp. So they can be used. No plan on sharing code yet, as they are open-source.
- Q: Can you comment more on relation with hoot project and advantages
that might bring to guile-emacs?
- A: Hoot is only tested on scheme decompilation. But it is a completely different project. One could image compile emacs to wasm and maybe with a different garbage collector.
- Q:
- A:
Notes
- https://guile-emacs.org/
- Some more motivation: avoid FFI to increase performance and allow for more optimizations (including type annotations)
Transcript
Hello everyone. I'm Robin Templeton, and I'm going to talk about Emacs Beguiled and recent progress on the Guile-Emacs project. First of all, if you're not familiar with Guile, it's an implementation of the Scheme programming language, which is a dialect of Lisp, and in the same family as Emacs Lisp, and Guile is GNU's official extension language. The goal of the Guile-Emacs project is to use Guile as the basis for Emacs's Lisp support. It has two main components: a new Emacs Lisp compiler built on top of Guile, and a variant of Emacs in which the built-in Lisp implementation is entirely replaced with Guile Elisp. We expect the combination of these two projects to have several benefits. One is improved performance. Another is increased expressiveness for Elisp and making it easier to extend and experiment with the language. Finally, it will reduce Emacs's reliance on C for two reasons. Guile will be responsible for the language implementation, so Emacs will no longer have to include a Lisp interpreter. It will also become possible to implement much more of Emacs in Lisp than is currently feasible. Of course, this raises the question of why Guile is suitable for this project. And we chose Guile for a few reasons. Guile is primarily a Scheme implementation, but it also has built-in support for multiple languages using its compiler tower. To add support for a new language to Guile, you only have to write a compiler from the source language to Tree-IL, which is essentially a low-level, minimal representation of Scheme. All of Guile's compiler optimizations occur at the Tree-IL layer or lower, so you don't need to worry about the lower-level details of the compiler when initially implementing your language. Guile also has some Lisp features that are very rare in Scheme implementations. For example, it has a nil value that counts as both false and an empty list, just like in Elisp, and it also has a version of the Common Lisp Object System and its metaobject protocol, which is called GOOPS. The idea of Guile-Emacs has a pretty long history, going back at least three decades. There have been about half a dozen previous implementation attempts. But the current iteration began with a series of six Summer of Code internships: Daniel Kraft's in 2009, and then my internships from 2010 to 2014. My basic implementation strategy was pretty straightforward. I implemented a core subset of Elisp, which was enough to run some batch mode programs outside of Emacs. In Emacs, I modified the garbage collector and the data structures for Lisp objects to use their libguile equivalents. I replaced Emacs' Lisp evaluator with the one provided by Guile Elisp. After a little over a year of work, at the end of the 2014 internship, I ended up with a fully functional prototype of Guile-Emacs. It used Guile Elisp alone as its Lisp implementation and was completely compatible with Emacs functionality and with external extensions. One caveat was that performance was pretty bad, because I was focused on correctness, as well as ease of integration with the Emacs C code. But it was nonetheless a major milestone for the project. Let's take just a moment to look at Guile-Elisp. For starters, we have access to Guile modules. If we call Guile's version function, we can see that we're running under Guile 3.0. We have access to some of the numeric tower via the arithmetic functions. We also have multiple values. We have to be careful to use Guile's values procedure here, not the CL library's, but you can see that this works properly rather than being an emulation. Finally, we have tail call elimination. Naturally, we're going to use factorial to demonstrate it. If n is zero, return the answer, else recurse with n less one and n times a. Of course, this definition works correctly, but it gets more interesting if we communicate the answer with an error, in order to look at a backtrace. You can see here that there are no calls to fact visible in between the request to evaluate and the error communicating the answer. That's because this tail call has been optimized into effectively a goto. This is essential for any kind of serious functional programming. That's a peek at Guile-Elisp. In 2015, I left university to go work on web technologies, and the project was dormant for a very long time. But that's been changing recently. During the last few months, I've been working with Larry Valkama to rebase Guile-Emacs onto the development branch of upstream Emacs, including the past decade's worth of upstream development. What we've ended up with is a series of rebases onto different versions of Emacs. The older ones tend to work pretty well. The newer ones have increasingly bad problems where they haven't been properly adjusted for changes in the Emacs implementation. But we do have by now a version of Emacs 30 which boots correctly and can be used for interactive debugging, as well as the ability to bisect the revisions of Emacs and find out where regressions were introduced. Our immediate goal is of course to complete the rebase. At the same time, we want to improve Guile Elisp's performance to at least be competitive with ordinary Emacs Lisp. Just to characterize the performance situation, Guile Elisp is usually about half as fast as ordinary Elisp, while Guile Scheme is quite often an order of magnitude faster than ordinary Elisp, and that's based on micro benchmarks like the Gabriel benchmarks. But there's clearly a lot of room to improve our compiler's output. If you want to mark your calendars, we're expecting to have a usable version of Guile-Emacs 30 out sometime next spring. We're also going to put some effort into either extracting old work or doing new work that could be contributed upstream. On the Guile side, we'll probably start out with optimizing the dynamic binding facilities, which are used very seldom in Scheme, but are used all the time in traditional Lisp dialects. On the Emacs side, we'll be working initially on abstracting away the details of the Lisp implementation where they're not relevant. And that will clean up the Emacs code base a bit. It'll make it easier to integrate Emacs and Guile Elisp. It will probably be helpful for anyone who is working on ordinary Elisp on their own. We're also going to be adding new features to Emacs Lisp. We've seen a few of them already. The numeric tower, tail call optimization, Common Lisp compatibility. We're also going to provide access to Fibers, which is a Guile library based on ideas from Concurrent ML that provides much more powerful facilities for concurrent and parallel programming than what Emacs currently offers. This plan meets Guile-Emacs' basic goals, and it's work that we could maybe get integrated upstream in a reasonable amount of time. But it's also worth considering what more we can do, and what effect Guile-Emacs might have on Emacs if it becomes simply Emacs. For context, the amount of C code in Emacs has increased by around 50% in the last decade, and now it constitutes around a quarter of the code base. C can be a bit of a barrier to customizing and extending Emacs. For example, there are about 1500 C subroutines. Around 500 are used in C code, as well as available to Lisp code, and being written in C means that they can't be practically redefined. The use of C can become a barrier to extending Emacs or customizing its behavior. We might consider writing as much of Emacs as possible in Lisp. One way to speed up this process would be to provide a Common Lisp implementation for Guile. Note that between Guile Elisp and Guile Scheme, we have all of the essential ingredients for a Common Lisp environment. We can also share code with other Common Lisp implementations such as SBCL and SICL. Overall, the duration of the project will be better measured in months rather than years, despite Common Lisp's reputation for being a large language. This could have multiple uses, of course. It could be a model for future improvements to Elisp, because Elisp and CL can interact directly without problems. And it would be very easy for Elisp to borrow language features from Common Lisp. But for the purpose of a C to Lisp transition, it would also provide us with instant access to a huge number of high-quality libraries for things that Guile is not necessarily equipped to deal with, such as access to low-level Windows APIs, as well as lots of other libraries, such as interfaces to GUI toolkits for a variety of operating systems. At a certain point, this has technical advantages. If most of Emacs is written in Lisp, then we could consider using Guile Hoot to compile Emacs to WebAssembly, making it available perhaps in web browsers or on systems with the WebAssembly System Interface. But it would also be a great victory for practical software freedom. That's the idea that Freedom One, the freedom to study and modify programs, should not just be legally and technically possible, but should be actively encouraged by our computing environments. Emacs is really one of the archetypal examples of this, but we can and should go further. When Emacs is implemented primarily in Lisp, the entirety of the system will be transparent to examination and open to modification. Every part of Emacs will be instantaneously inspectable, redefinable, and debuggable. This will be a fundamental change in what is possible to do with Emacs extensions. For example, one experiment I'd be interested in is using the Common Lisp Interface Manager as the basis for Emacs's user interface. Screwlisp is giving a talk about McCLIM later today, but for present purposes, just think of it as a super-powered version of Emacs's concept of interactive functions. It would be a pretty long-term project in Emacs as it currently exists, but it would be almost trivial if Emacs were customizable at the lowest layers via Lisp. We'll certainly be looking at the practicality of these kinds of changes as we continue developing Guile-Emacs. Finally, how can you get involved with and support Guile Emacs? One way to help is just by trying it out and letting us know what your experiences are like. There will be a snapshot available on the Codeberg project site of the version that I'm using to give this presentation. It will be available both as a Guix package and as a portable tarball. This will be more interesting as we get closer to a complete rebase. We're also always happy to talk to potential contributors or potential collaborators from other projects. We can always use bug reports, and we're interested in what kind of features people actually want to see in Guile-Emacs. Guile-Emacs is also being developed by a small worker cooperative, so donations are a pretty direct way to support the project. If you do nothing else, I recommend going to the website and subscribing to our mailing lists so that you can keep up with news on the project. If you're watching this at EmacsConf, there will be a Q&A session immediately following this, and thanks for watching!Captioner: sachac and robin
Q&A transcript (unedited)
All right. Hey, thanks for bearing with us there. We had a couple of bumps in the road, a cross between a couple of different versions of our program that we deliver here, different ways that we bring this stream together between the recorded content that that speakers are putting together in advance in the live content, such as what you're seeing right here. So thanks go to Sacha and Leo, and everybody behind the stages gluing it all together. And we're back here now, and I'm speaking with Robin, who us ready to take on some of your questions and address some of the comments over here on the etherpad. If you want to jump in there, there's links in the chat. And thanks so much, Robin, for your talk. And it's also been a pleasure chatting with you just a little bit over the last couple of months on IRC. Yeah, absolutely. Great meeting you. All right. All right, everyone. I think I am streaming now. So let's look at it. Let's see. I see the IRC scrolling. So let's see where that's going. Yes, the Common Lisp is what I thought would piss people off. And because it's not part of either community, but I think it would be a good compromise for building a Lisp into a language that's more suitable for building large systems like the kind that we are building in Emacs today. I also left out an important part of the talk, which is part of the motivation for transitioning from C to Lisp. And that's the performance characteristics fundamentally change when you get a modern and high performance Lisp system involved. it starts getting less practical to just call out to C to speed up every operation. Among other things, you lose the ability to use more advanced control structures, like the limited continuations. And you also have to pay the overhead of calling out to our foreign function. So it gets to be an increasingly better deal to optimize your list implementation and provide ways for building faster list programs, such as type annotations, once you've gotten over a certain threshold of performance.Questions or comments? Please e-mail robin@terpri.org