Beguiling Emacs: Guile-Emacs relaunched!

Robin Templeton (they/them)

IRC: robin on libera.chat; Fediverse: @lispwitch@octodon.social; Matrix: @terpri:matrix.org; website: http://terpri.org/; mail: robin@terpri.org

The following image shows where the talk is in the schedule for Sat 2024-12-07. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 16-min talk ; Q&A: BigBlueButton conference room https://media.emacsconf.org/2024/current/bbb-guile.html Etherpad: https://pad.emacsconf.org/2024-guile
Etherpad: https://pad.emacsconf.org/2024-guile
Discuss on IRC: #emacsconf-dev
Status: Q&A open for participation

Description

The Guile-Emacs project seeks to develop new foundations for Emacs, building on forty years of development to prepare the way for the next forty. Guile-Emacs brings Emacs and Guile together by providing a new Elisp implementation based on Guile's compiler technology, serving as the basis for a more expressive and extensible version of Elisp. We envision Guile and Emacs being co-developed in a sort of mutualism, with Emacs becoming the heart of a "Lisp machine for the 21st century" and with Guile fulfilling the promised role of Lisp from the GNU Manifesto.

Guile-Emacs is now being developed by a democratic workers cooperative, focused on development of the project itself and related components of the GNU system.

In this talk, I'll cover:

• What is Guile-Emacs, in terms of its goals and general architecture? Why is Guile well-suited to the goals of the project?
• How did the Guile-Emacs project begin, and what is its status today?
• What are the immediate tasks for Guile-Emacs development, and how will they improve Guile-Emacs itself as well as its component projects?
• How are our long-term goals for Guile-Emacs connected to the spirit of Emacs and the GNU Project in general? What do we envision for the future of GNU Emacs?
• How can you get involved with and support this effort?

Along the way, we'll show live demos of Guile-Emacs itself and its extensions to Emacs Lisp.

About the speaker:

Robin Templeton is a free software advocate who enjoys programming language design and exploring system architecture. Their fascination with Emacs and Lisp lead them to begin work on the then-hypothetical Guile-Emacs project during their university studies. If given the opportunity and interest, they will bless an interested listener with a treasure trove of obscure Lisp history.

Guile-Emacs seeks to provide new foundations for Emacs, integrating Emacs and Guile via a new Elisp implementation. We envision a significant role for Emacs within the GNU Project, becoming a central part of a "Lisp machine for the 21st century".

Transcript

Hello everyone. I'm Robin Templeton, and I'm going to talk about Emacs Beguiled and recent progress on the Guile-Emacs project. First of all, if you're not familiar with Guile, it's an implementation of the Scheme programming language, which is a dialect of Lisp, and in the same family as Emacs Lisp, and Guile is GNU's official extension language. The goal of the Guile-Emacs project is to use Guile as the basis for Emacs's Lisp support. It has two main components: a new Emacs Lisp compiler built on top of Guile, and a variant of Emacs in which the built-in Lisp implementation is entirely replaced with Guile Elisp. We expect the combination of these two projects to have several benefits. One is improved performance. Another is increased expressiveness for Elisp and making it easier to extend and experiment with the language. Finally, it will reduce Emacs's reliance on C for two reasons. Guile will be responsible for the language implementation, so Emacs will no longer have to include a Lisp interpreter. It will also become possible to implement much more of Emacs in Lisp than is currently feasible. Of course, this raises the question of why Guile is suitable for this project. And we chose Guile for a few reasons. Guile is primarily a Scheme implementation, but it also has built-in support for multiple languages using its compiler tower. To add support for a new language to Guile, you only have to write a compiler from the source language to Tree-IL, which is essentially a low-level, minimal representation of Scheme. All of Guile's compiler optimizations occur at the Tree-IL layer or lower, so you don't need to worry about the lower-level details of the compiler when initially implementing your language. Guile also has some Lisp features that are very rare in Scheme implementations. For example, it has a nil value that counts as both false and an empty list, just like in Elisp, and it also has a version of the Common Lisp Object System and its metaobject protocol, which is called GOOPS. The idea of Guile-Emacs has a pretty long history, going back at least three decades. There have been about half a dozen previous implementation attempts. But the current iteration began with a series of six Summer of Code internships: Daniel Kraft's in 2009, and then my internships from 2010 to 2014. My basic implementation strategy was pretty straightforward. I implemented a core subset of Elisp, which was enough to run some batch mode programs outside of Emacs. In Emacs, I modified the garbage collector and the data structures for Lisp objects to use their libguile equivalents. I replaced Emacs' Lisp evaluator with the one provided by Guile Elisp. After a little over a year of work, at the end of the 2014 internship, I ended up with a fully functional prototype of Guile-Emacs. It used Guile Elisp alone as its Lisp implementation and was completely compatible with Emacs functionality and with external extensions. One caveat was that performance was pretty bad, because I was focused on correctness, as well as ease of integration with the Emacs C code. But it was nonetheless a major milestone for the project. Let's take just a moment to look at Guile-Elisp. For starters, we have access to Guile modules. If we call Guile's version function, we can see that we're running under Guile 3.0. We have access to some of the numeric tower via the arithmetic functions. We also have multiple values. We have to be careful to use Guile's values procedure here, not the CL library's, but you can see that this works properly rather than being an emulation. Finally, we have tail call elimination. Naturally, we're going to use factorial to demonstrate it. If n is zero, return the answer, else recurse with n less one and n times a. Of course, this definition works correctly, but it gets more interesting if we communicate the answer with an error, in order to look at a backtrace. You can see here that there are no calls to fact visible in between the request to evaluate and the error communicating the answer. That's because this tail call has been optimized into effectively a goto. This is essential for any kind of serious functional programming. That's a peek at Guile-Elisp. In 2015, I left university to go work on web technologies, and the project was dormant for a very long time. But that's been changing recently. During the last few months, I've been working with Larry Valkama to rebase Guile-Emacs onto the development branch of upstream Emacs, including the past decade's worth of upstream development. What we've ended up with is a series of rebases onto different versions of Emacs. The older ones tend to work pretty well. The newer ones have increasingly bad problems where they haven't been properly adjusted for changes in the Emacs implementation. But we do have by now a version of Emacs 30 which boots correctly and can be used for interactive debugging, as well as the ability to bisect the revisions of Emacs and find out where regressions were introduced. Our immediate goal is of course to complete the rebase. At the same time, we want to improve Guile Elisp's performance to at least be competitive with ordinary Emacs Lisp. Just to characterize the performance situation, Guile Elisp is usually about half as fast as ordinary Elisp, while Guile Scheme is quite often an order of magnitude faster than ordinary Elisp, and that's based on micro benchmarks like the Gabriel benchmarks. But there's clearly a lot of room to improve our compiler's output. If you want to mark your calendars, we're expecting to have a usable version of Guile-Emacs 30 out sometime next spring. We're also going to put some effort into either extracting old work or doing new work that could be contributed upstream. On the Guile side, we'll probably start out with optimizing the dynamic binding facilities, which are used very seldom in Scheme, but are used all the time in traditional Lisp dialects. On the Emacs side, we'll be working initially on abstracting away the details of the Lisp implementation where they're not relevant. And that will clean up the Emacs code base a bit. It'll make it easier to integrate Emacs and Guile Elisp. It will probably be helpful for anyone who is working on ordinary Elisp on their own. We're also going to be adding new features to Emacs Lisp. We've seen a few of them already. The numeric tower, tail call optimization, Common Lisp compatibility. We're also going to provide access to Fibers, which is a Guile library based on ideas from Concurrent ML that provides much more powerful facilities for concurrent and parallel programming than what Emacs currently offers. This plan meets Guile-Emacs' basic goals, and it's work that we could maybe get integrated upstream in a reasonable amount of time. But it's also worth considering what more we can do, and what effect Guile-Emacs might have on Emacs if it becomes simply Emacs. For context, the amount of C code in Emacs has increased by around 50% in the last decade, and now it constitutes around a quarter of the code base. C can be a bit of a barrier to customizing and extending Emacs. For example, there are about 1500 C subroutines. Around 500 are used in C code, as well as available to Lisp code, and being written in C means that they can't be practically redefined. The use of C can become a barrier to extending Emacs or customizing its behavior. We might consider writing as much of Emacs as possible in Lisp. One way to speed up this process would be to provide a Common Lisp implementation for Guile. Note that between Guile Elisp and Guile Scheme, we have all of the essential ingredients for a Common Lisp environment. We can also share code with other Common Lisp implementations such as SBCL and SICL. Overall, the duration of the project will be better measured in months rather than years, despite Common Lisp's reputation for being a large language. This could have multiple uses, of course. It could be a model for future improvements to Elisp, because Elisp and CL can interact directly without problems. And it would be very easy for Elisp to borrow language features from Common Lisp. But for the purpose of a C to Lisp transition, it would also provide us with instant access to a huge number of high-quality libraries for things that Guile is not necessarily equipped to deal with, such as access to low-level Windows APIs, as well as lots of other libraries, such as interfaces to GUI toolkits for a variety of operating systems. At a certain point, this has technical advantages. If most of Emacs is written in Lisp, then we could consider using Guile Hoot to compile Emacs to WebAssembly, making it available perhaps in web browsers or on systems with the WebAssembly System Interface. But it would also be a great victory for practical software freedom. That's the idea that Freedom One, the freedom to study and modify programs, should not just be legally and technically possible, but should be actively encouraged by our computing environments. Emacs is really one of the archetypal examples of this, but we can and should go further. When Emacs is implemented primarily in Lisp, the entirety of the system will be transparent to examination and open to modification. Every part of Emacs will be instantaneously inspectable, redefinable, and debuggable. This will be a fundamental change in what is possible to do with Emacs extensions. For example, one experiment I'd be interested in is using the Common Lisp Interface Manager as the basis for Emacs's user interface. Screwlisp is giving a talk about McCLIM later today, but for present purposes, just think of it as a super-powered version of Emacs's concept of interactive functions. It would be a pretty long-term project in Emacs as it currently exists, but it would be almost trivial if Emacs were customizable at the lowest layers via Lisp. We'll certainly be looking at the practicality of these kinds of changes as we continue developing Guile-Emacs. Finally, how can you get involved with and support Guile Emacs? One way to help is just by trying it out and letting us know what your experiences are like. There will be a snapshot available on the Codeberg project site of the version that I'm using to give this presentation. It will be available both as a Guix package and as a portable tarball. This will be more interesting as we get closer to a complete rebase. We're also always happy to talk to potential contributors or potential collaborators from other projects. We can always use bug reports, and we're interested in what kind of features people actually want to see in Guile-Emacs. Guile-Emacs is also being developed by a small worker cooperative, so donations are a pretty direct way to support the project. If you do nothing else, I recommend going to the website and subscribing to our mailing lists so that you can keep up with news on the project. If you're watching this at EmacsConf, there will be a Q&A session immediately following this, and thanks for watching!

Questions or comments? Please e-mail robin@terpri.org