Back to the talks Previous by track: Windows into Freedom Next by track: GNU Emacs: A World of Possibilities Track: Development - Watch

Bringing joy to Scheme programming

Andrew Tropin - Nick: abcdw, Site: https://trop.in/, Fediverse: https://fosstodon.org/@abcdw

Format: 22-min talk ; Q&A: BigBlueButton conference room
Status: Q&A to be extracted from the room recordings

00:02.120 Introduction
00:23.280 Interactive development
01:18.180 REPL: Read Eval Print Loop
02:53.720 Long-lasting loops
04:07.600 Not interruptible
05:23.160 No protocol
05:51.480 Not scalable
07:25.860 nREPL
09:01.740 Arei, Ares, and how to try
10:34.180 Demo
11:27.640 Continuations
12:32.460 Reading from stdin
13:33.420 Fancy example with continuations
15:13.160 Guix API
17:42.060 Support
17:57.020 Future steps - Multiple simultaneous evaluations in different contexts
18:46.220 Tree-sitter integration
18:56.880 Full-fledged debugger
19:22.760 FAQ - Does it support other Scheme implementations?
19:58.380 Is it possible to use it with other text editors?
20:22.121 Conclusion
20:45.880 Contacts
Duration: 21:01 minutes

Description

Emacs is usually a primary tool people remember, when talk about development environment for lisp languages. It serves the purpose great for Common Lisp, Clojure and of course Emacs Lisp, but what about Scheme? Let's talk about current state of the things, recent improvements, and emerging tools in this field.

My talk covers the following:

  • What does a usual Scheme developer day look like? And how it can be made more enjoyable?
  • Important developer duties, their automation and acceleration.
  • Interactive development and its benefits.
  • Emacs setup for Scheme development.

Links:

About the speaker:

Talk about Lisp development workflows, REPLs, and modern Scheme tooling for Emacs. Author of Guix Home, maintainer of rde, FOSS developer.

Discussion

Questions and answers

  • Q: How much do you use these repels remotely ex using a server or desktop from your laptop?
    • A: I don't use it remotely at the moment, but it should work perfectly fine (except maybe lookup and other similiar functions). I also want to add a shepherd service for ares-rs, so you can connect to GNU Shepherd and systems based on it (like GNU Guix) from you emacs process and interact fluently with guile code.
  • Q: Can this be integrated with eglot?
    • A: I'm not sure how this integration could look like. Theoretically, it's possible to expose many of ares-rs functions via LSP custom actions (or whatever it called). Anyway, contact me on IRC or https://trop.in/contact to discuss it in more details if you have something in mind.
  • Q: How hard is it to add support for something else than Guile? Does it make sense to contribute at this early stage of development? I've written several packages for CHICKEN Scheme before and would like to try this one.
    • A: It's a matter of implementing the whole chicken-ares-rs :) Many of the code can be reused, but not all, unfortunately. emacs-arei doesn't need any (or almost any) changes.
  • Q: (One day late sorry) Is nREPL more extensible than what SLIME/SLY use in Common Lisp world (I think it's comint.el) ?

Notes and discussion

  • brilliant work for scheme
  • yeah, this is overdue. the only real alternative is slime-r7rs

Transcript

[00:00:02.120] Introduction
Hello and welcome everyone on EmacsConf 2023. I'm Andrew Tropin. I work on operating systems and programming languages. Today, we discuss Lisps, Schemes, REPLs, interactive development, and how to make your own cozy development environment.
[00:00:23.280] Interactive development
Let's start from interactive development. Lisps are famous for a nice Interactive Development Experience. They have REPLs. Emacs Lisp has its own Lisp machine, and a lot of cool IDE with different functionality is already here and providing a nice and pleasant experience. The question is, is it enough? In most cases, yes, but for some languages, we have some white spaces, some missing pieces. And for example, in Scheme world, we already have a few tools. We have REPL, we have integration for REPL in Emacs, but is it enough? Let's see.
[00:01:18.180] REPL: Read Eval Print Loop
We know that Emacs is very good for Lisps and REPL. Lisp and Emacs should be a perfect setup. But let's see how REPL basically works. It's an event loop which does three things. It reads an expression, it evaluates the expression, and it prints the result. We can take a simple expression, input it into REPL, and evaluate it and see the result. Very nice, very convenient. You can experiment and see immediately what is happening. You can even run a long-running process which does something. You can interrupt it and everything will be okay. But the problem appears when you start to develop a bigger project. And in most cases, you don't do your whole development in REPL. You do only a small part of it. In most cases, you just write the source code in text files, and after that, you run those snippets of code from those text files, or run the whole project. It's not very convenient to copy and paste every time the snippets of code to the REPL, see the result, modify the snippet of code, copy it again, and so on. So people invented some integration between REPL and your text editor. So you can evaluate expressions inside your text editor and see the result here.
[00:02:53.720] Long-lasting loops
Works good so far, but what happens if we run a long-lasting loop, which does a lot of operations. As you can see here with a simple example, the output of the function, stdout of the function is presented here, and the resulting value is here. If you run a long-running process, you don't see anything happening. And you see there's a watch instead of my cursor. Maybe you don't see it, but nothing actually happens, at least from the point of view of the user. But if we interrupt the evaluation, we will see that some process in the background was launched, but we didn't see anything. Because the REPL is a single-threaded blocking process, which reads stdin and prints stdout, make the integration between the REPL and your text editor is not an easy task. And even if you do it, you have a lot of downsides, usually.
[00:04:07.600] Not interruptible
First of all, the process is not interruptible. If you have a remote process which listens on the socket to which you connect from your development environment, and you run some infinite loop, for example, you can't interrupt it. Because interruption is done via signals, and signals to remote processes are not usually the thing in such integrations.
[00:04:38.760] Output is not interactive
Output is also not interactive. Usually, for example, here you can see when I evaluate the expression, the output is captured on the evaluation side, and after that, after the whole evaluation of the whole expression finished, I get the result, all the stdout at once. And if I run the process which evaluates for 5 seconds, I will see the first signs of the life only after 5 seconds of evaluation. Okay, what else?
[00:05:23.160] No protocol
When you do such integrations, you have no protocol, you have just stdin and stdout. You print to stdin from your text editor. You read from stdout of the process. It's hard to tell if evaluation is finished, if it requires stdin, and how to extend the REPL to make it more featureful, and so on.
[00:05:51.480] Not scalable
And also, such integrations are usually not very scalable. For example, if you want to have a completion, you type something, you have the completion. Cool. But if you run the process and at the same time try to have a completion, you don't have it, because the evaluation is in progress, and you can't calculate the completion candidates at the same time. To make it more obvious, I will start a completion here. You see the completion pop-ups. I start the evaluation process, and when I try to complete something, the evaluation freezes and there is no completion. Not very convenient. Usually, you have some long-running processes and you want them to continue while you have your go to definition, completion, and other things. Overall, those issues make it quite inconvenient to integrate REPL in text editors or development environments, so you need something else to make the work comfortable.
[00:07:25.860] nREPL
There is already a solution called nREPL. It's a synchronous protocol which allows to send operations to the server and receive responses in a synchronous manner. And here is a simple example of a few operations. First one is cloning the existing session, and as a response you will get a new session. Also you send the evaluation request with code that you want to evaluate, and you get two responses. First one says that output is captured and it's equal to "hi\n", and after that, you receive an "Evaluation completed", the value of this expression. This protocol was developed for CIDER development environment. It's a Clojure development environment for Emacs. It's very cool, featureful, reliable, and I would say production-ready. A lot of professional Clojure developers use it. The nREPL protocol is very simple. It has a few operations out of the box, and you can extend it with any arbitrary operation you want. I work a lot on Guix codebase and other Scheme projects, so the experience I had previously with nREPL was not satisfying. I decided to just implement nREPL protocol.
[00:09:01.740] Arei, Ares, and how to try
First of all, I implemented nREPL server in Guile. I called it guile-ares-rs, and used it with a generic nREPL client for Emacs. It worked. It had some rough edges, but overall it was okay. And after that, to add more features to make the implementation more complete, I wrote my own nREPL client for Emacs and called it arei. And I got almost complete Guile IDE in two months. So ares-rs is nREPL server implementation. arei is Emacs client, which uses the same nREPL protocol. It utilizes sesman package for managing sessions, the association of buffers with nREPL connection. It has some roots. The implementation has some roots in Geiser, CIDER, Monroe, and Rail. I took small snippets for some parts of functionality. I used the CAPF and xref infrastructure for completion at point and cross-reference capabilities. And by the time of conference, I hope that README will be complete enough so you will be able to try it yourself.
[00:10:34.180] Demo
Let's see what is possible with it already. Let's connect to nREPL server. After that, you can evaluate the expression. And you see the stdout and the result. Very nice, very convenient. You have different expression, you evaluate it, you get the value of the evaluation. You can run an infinite loop which prints to stderr and stdout and you see all necessary stuff. Very cool. But also, you can interrupt the evaluation, which is very convenient if you accidentally run an infinite loop.
[00:11:27.640] Continuations
Also, do you remember here we have a few more examples that we didn't try yet? For example, on usual REPL implementation, if I evaluate this expression, I get return value. I make a continuation and save it to this variable and I try to call this evaluation and I get an exception, because the environment in which this continuation was created was different and it has redefined stdout and stderr to capture it. But when I run it one more time, when I resume the continuation, the environment changed and it doesn't work. What happens in arei? I define continuation, I save the continuation for the simple expression and I resume the continuation with a new argument, and you can see at the top of the screen that it works perfectly fine.
[00:12:32.460] Reading from stdin
Also, with a usual REPL implementation, let's see what happens when we have a process which reads from stdin. I evaluate the expression and nothing visible happens. I can try to type C-g, C-c, and after some time it will say user interrupt. What actually I expect in such a case to have a minibuffer which prompts me for the input. When I evaluate the same expression in the arei, you see the prompt at the minibuffer and here I can tell, "Hello I'm a message from minibuffer". Cool. You will see that this message is printed to stdout, and unspecified was returned as a result of this expression.
[00:13:33.420] Fancy example with continuations
Let's make some fancy example with continuations. Continuations is a very cool mechanism which is not the topic of today's talk, but you can find a lot of interesting information in Scheme documentation or in related books, and I advise you to do it because it's really nice thing that is actually applicable in many different programming languages. Here you can see the infinite loop which just prints values increasing one by one. And here we save a continuation on each iteration. I can call the continuation and it will resume from the previous saved step. And you can see, it resumed from the same step

we interrupted earlier, but we provided a new value for it. another value for it.

 We can provide another value and it resumed from the same spot it was saved earlier. But I also can provide a read-i value and if I provide read-i value, the infinite loop will read the input from stdin and will continue the evaluation with a different i provided in this input. So let's try to type some arbitrary value and you see that the loop continued with this value. Very nice. And every time we could easily interrupt it.
[00:15:13.160] Guix API
Okay, what most annoying thing that I had previously with the usual REPL implementation that I have a quite nice Guix API where I can build packages, systems and other stuff. But if I evaluate this expression, I will get an error. Okay. I will get an error because I don't have an appropriate environment. But what I can do, I can connect to the remote REPL by creating a server with guix repl --listen command and connecting to it with geiser-connect command. And now I can evaluate this expression. Right? Wow. Okay. It actually doesn't matter for my example. I will explain how it doesn't work easily. This is a long-running process which prints something and it can take up to a few minutes. And for the whole few minutes I don't see any results, the same as with this infinite loop which prints to stdout but I don't see anything interactively. With arei, I can run the evaluation of the same expression, and you will see instantly that stdout is presented here in slightly yellowish color. I can interrupt the evaluation if I don't want to wait until it's finished, and just after that, I can evaluate another value. So that's cool. And let's see one more thing. We have an infinite loop and we have some completion here. And completion still works, very nice, while the infinite loop is running. Okay.
[00:17:42.060] Support
Actually it took me around two months of full-time work funded by my own savings, and you can support and help to the project using OpenCollective or by contributing on SourceHut.
[00:17:57.020] Future steps - Multiple simultaneous evaluations in different contexts
The future steps for the project include an experimental workflow where you have multiple simultaneous evaluation in different contexts. For example, you have Fibers, you have Goblins, you have some HTTP server or some other thing, and you want to run all of them independently in slightly isolated sessions, and you want to have the ability to still interact with them. For example, if they require standard input or something else, you want to be able to provide it. You want to see the stderr and stdout of those long-running processes and so on.
[00:18:46.220] Tree-sitter integration
The second thing is tree-sitter integration for better syntax highlighting, code navigation, and other features.
[00:18:56.880] Full-fledged debugger
And after that, probably we will do a full-fledged debugger so you can jump expressions one by one and see the results and see some intermediate values during the evaluation. And it's very possible because nREPL is a very extensible protocol and you can implement whatever you want on top of it.
[00:19:22.760] FAQ - Does it support other Scheme implementations?
I will answer two probably very frequent questions. Does it support other Scheme implementations? At the moment, it doesn't, but the Scheme implementation is not restricted. You have a server which is implemented in your language and you have a client--in our case, arei-- which communicates with this protocol. So if you implement nREPL server in a different language, it should work with already implemented arei client.
[00:19:58.380] Is it possible to use it with other text editors?
And is it possible to use the same functionality in other text editors, for example in VS Code, Vim, whatever? Yes, it's possible and the case is similar here. You have already implemented nREPL server and you can write your own nREPL client in a different text editor and it will work.
[00:20:22.121] Conclusion
I would like to thank the authors and maintainers and contributors of Guile, Geiser, CIDER, Clojure, and Emacs, and all other people who are somehow related to the work on those projects involved in this talk. And I hope the Scheme programming will be enjoyable.
[00:20:45.880] Contacts
If you want to contact me, join #tropin IRC channel at libera.chat, or drop me a message via email or feediverse using andrew@trop.in handle. I will see you in a bit in Q&A session.

Captioner: sachac

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

Back to the talks Previous by track: Windows into Freedom Next by track: GNU Emacs: A World of Possibilities Track: Development - Watch