Back to the talks Previous by track: Swanky Python: Interactive development for Python Next by track: Emacs, editors, and LLM driven workflows Track: Development

Interactive Python programming in Emacs

David Vujic (he/him) - Mastodon https://mastodon.nu/@davidvujic

Format: 20-min talk ; Q&A: BigBlueButton conference room Etherpad: https://pad.emacsconf.org/2025-python
Etherpad: https://pad.emacsconf.org/2025-python
Status: TO_REVIEW_QA

Talk

Duration: 19:52 minutes

Q&A

00:00.000 Developing Python in Emacs
01:16.680 Q: How long have you been using Emacs?
02:39.520 Q: What editor were you primarily using before?
04:00.280 Q: Do you have your Emacs connect to the IPython kernel over the XMPP socket that IPython sets up?
05:38.480 Q: Not sure if you've explored Org Babel, but what are the benefits of using your approach over Org?
07:05.800 Q: Have you seen marimo.io notebooks? It uses standard Python.
08:20.560 Q: what LLM did you say you've used?
10:28.480 Q: Are you building your own Emacs, or using the system packages?
12:40.080 Q: Are you involved with local Emacs meetups?
14:24.640 Q: Can you repeat the name of the LLM that you specifically mentioned there? - ECA
18:08.742 Wrapping up
Listen to just the audio:
Duration: 18:44 minutes

Description

Learning Lisp was an eye opener for me. The Interactive way of writing code is Amazing. Emacs has great support for this in general already. For Python, there's limitations and I have looked for ways to make Python development more interactive (and fun). Python is what I do at work and in Open Source projects. It has lead me to develop Emacs features specific for Python: evaluating code with visual feedback, modify a running Python app without restarts, and some LLM support. I will demo and talk about what I have developed to make Python development interactive and joyful.

About the speaker:

My name is David and I'm a software developer. Colleagues and friends may know me as an early adopter of agile ideas and test driven development. I am passionate about things like that, and share the things I learn to the community and the people I work with. My favorite programming languages are Python and Clojure. On my spare time I practice outdoor Parkour & contribute to Open Source.

Transcript

Okay, so welcome to this session about interactive Python programming. My name is David Vujic and I live and work in Stockholm, Sweden. a developer and today I focus mainly on Python software development. So I do this at work and I also do this on my spare time in my open source projects. Before that, I've been part of the Lisp community. I've been a Clojure developer, and also, like, way back, I was in the Microsoft world and developed C# and .NET stuff. What I've been doing lately is to try to improve the developer experience when you write Python code. So what I want to talk about is this, but also I want to begin with feedback loops because I think it's very related to this interactive programming style, like having this nice feedback when you write code. So I'm going to begin with that.
[00:01:10.534] Feedback loops
So this image, you know, this circle is supposed to be a visualization of a feedback loop. Let's say we write our code and then we deploy it to production. Then when it's running there, we can check if things work, or if maybe someone else will let us know. Maybe our customers will let us know. That's a pretty slow feedback loop with potential risks of damaging your business or whatever. This is obvious, of course. So a faster feedback loop probably is to have some kind of automation when you do commits or maybe you have this pull request things and even reviews. So maybe not always as fast as deploy, don't deploy directly to production, but it's probably safer and often you get this automated feedback faster anyway. But it's still kind of slow. You have to wait. You have to push things to GitHub maybe and wait. So there's faster ways for sure to get feedback. So a much faster way is to write code, and write some unit tests, and run those unit tests. So then you do everything on your local machine and you will fairly quickly learn if your code does what you think it does or if it doesn't. I want to zoom in to this test write code and test flow a bit. Let's do that.
[00:02:56.000] Test-driven development
As a developer, I have used a thing called test-driven development for quite some time. I find that this way of working is very fast when it comes to getting feedback on what your code does and how you should continue the development. So, test-driven development, basically that you start writing a test for something that you want to develop, and then you continue developing that, and then you go back to the test, and modify and modify the code, and you go back and forth between the tests and the code. It's sort of like a ping-pong game. I find this very effective when you want to get feedback and to know how to continue the development. The most important thing that I feel is that you know what the code does. You learn very quickly.
[00:04:05.560] REPL-driven development
Let's zoom into this TDD flow a little bit. The last couple of years, I've been doing a slightly different thing which is called REPL-driven development. REPL-driven development is very similar to test-driven development, but I find it even quicker. You get feedback even quicker than with a regular TDD setup. So REPL-driven development is about writing and evaluating code in a REPL basically. And you can do experiments and you can refactor and re-evaluate and you get instant feedback on what the code does and what you need to change. So I think that's even faster than test-driven development. Okay, REPL driven development. Let's go back. What's the REPL? Most of developers know what a REPL is. The most common setup is you open this shell and you use the REPL for your programming language. In this case I'm using the Python REPL or the IPython REPL which is an enhanced REPL for Python development. So what happens here is that we start a REPL session in isolation. So this session knows about the Python environment. So it knows about the Python language basically. So as soon as we start writing things, adding variables or creating writing functions or even doing imports. Then the session will be more and more aware of the code so we will add things to the to the session and then that means that we can run functions we can print out these variables and things like that. But with REPL driven development it's not really that well at least not what I mean with REPL driven development. So what I'm thinking of is that you are in your code editor where you have your autocomplete, and you have your syntax highlighting and your favorite theme, color theme, and all of those things. But instead, you have this running REPL in the background or in a smaller window or buffer. So that means that you write code and you can send that code to the running REPL, to the REPL session. You write and do everything as you would do when writing your code basically. In this case, in this example, I have evaluated these two functions. I've sent them to the REPL session so it's aware of these functions. Then I switched to a separate different module and evaluated that one. So the REPL session now knows about these two functions and also these two variables. That means that I can evaluate the state of those variables and change code and re-evaluate and things like that. So in this example if you look in the smaller area there you see that I have evaluated this res variable on line 6 and the output was that it's a dictionary with two keys and two values basically. So this setup works in basically any of your favorite code editors. So you can do this in Visual Studio Code, you can do this in PyCharm or Vim. But what I have done is that... More like what I have missed is that when I write code and do this evaluation, this is really cool, but then I need to switch context if I want to see the result. I have to switch context to this other window. I have my focus on the code and then I have to look in a different place to know the results. And if it's a larger output, then maybe I need to scroll. So I wanted to find out if it was possible to make this even smoother and faster, this feedback loop even faster, so I don't have to switch context. What I've done here is that... I can select a row or a region and I can evaluate and then an overlay, a small pop-up shows up with the evaluated result right next to it. So I can change code and re-evaluate and quickly see the result of it without doing this context switching. So the way I've done it is that I wanted to reuse the existing tooling that I already had. I know that my in-editor REPL, the IPython REPL, already does this evaluation. So I figured maybe I can extract the data and do this visualization as a separate thing. That's how I've done it. What I've done is that I've created this overlay and placed it where my cursor currently is, right next to the code. Then I've extracted the evaluated result and put it in this overlay. I also want this overlay to have this nice looking syntax, so I've set it to this Python mode, so we get this syntax highlighting. Make it look very readable. And as a nice developer experience thing, when you move the cursor, of course you don't want the overlay to be there. You want it to disappear. So those kinds of things I've added. So putting the overlay at the right place and feed it with the evaluated data and then make it disappear when it's not interesting to look at anymore. What I've described so far is something that I use on a daily basis, and it covers most of my needs while doing Python development. But one thing I still miss, and I miss it from my days as a Clojure developer, because over there we could have a running app on our local machine and we can have our editor, and the app and the editor were connected. So when I did some changes in the code, the app would change without any restarts or anything like that. And the same if I would change the state of the app, I can inspect the state from the code. So they were connected. They are connected. So I was thinking, hey, this would be really cool if we could have something like this in Python. And that reminded me of Jupyter and Jupyter notebooks because I think notebooks, the way you do things there, is very similar to what I was trying to achieve. So I was reading up a little bit on how this notebook thing works. It turns out that a notebook is a client that talks to a server, that communicates with a server. It's on the server that all this Python evaluation and all this thing happens. Then what I've done is that instead of starting up IPython in my editor, I start the Jupyter console instead. And then I can give it that unique ID and it will be connected to that running kernel.
[00:12:30.920] FastAPI CRUD
In this example, I've created this FastAPI CRUD app that has this create, read, update, and delete endpoints. It has this, it's locally running, it has this database where you can do all these things. I'm running this FastAPI app in the kernel and then I've connected to, I've connected to the kernel in my editor too. Both of them are connected to the kernel. What I do now is that I want to initially create some data. I'm going to add this, creating this message. What I get back is a message ID. I want to experiment in my browser. What do I get with that message ID? I'm evaluating the read function. I instantly get this evaluated result, which was this hello world text. So what happens if I do some changes in this app? I'm going to grab this message ID and write something else. Now I can evaluate the same thing again, and you can see that the content has changed to this new value. My editor isn't in any debug mode or something like that. It doesn't know what database it is. It doesn't have any environment variables set up or something like that. It is only connected to the kernel, and the kernel is aware of that. It's running the app. It has the connection strings and everything that is needed. So that's how this thing works. Now I want to do some inline hacking because I want to store this input that is sent from this app because I want to work with it afterwards. I can add this dictionary that stores this message. I'm updating the source code of this app, and when I run any of these endpoints again, you will see that the state changes, and the new inputs, I can grab and I can use them for quick evaluation or testing. This example is really simple. It was just an integer. For example, if you are sending a more complex object, maybe a pydantic schema or something, and you want to inspect what's coming in, and if you have some sort of validation that you want to test out. The configuration or the code that I wrote to make this work is a little bit different than just adding an overlay. I'm using this overlay just like with the IPython example, but in this case, when I change code, I have to think about where that code lives, because it's the app that runs the code. So it's in the app context I need to manipulate with the data. If you have started the app from maybe a main function and that module imports namespaces, then you need to, if you want to update a function or something like that, you need to update it in the correct namespace. What I did before in IPython by adding and changing things, everything ends up in the global namespace. But here, if you want the app to actually react to the changes, you need to put it in the right namespace. So that's what I do here. I do some lookups, where is this function, and then I do this reload of this function or module. And when I was developing this, I was thinking, hey, this is really ugly. I'm in this REPL and do some manipulation of the imports and things like that. That didn't feel good. Then I was reminded of the IPython. And IPython has this feature to reload any updated submodules. I was curious how do they do it. I looked in the IPython source code and saw that they also use importlib and reloading of this module. Once I've learned that, then I stopped thinking that my code was hacky. I thought it was good enough at least.
[00:17:37.160] Testing with an LLM
But one thing that has bothered me for a long time is I quite often want to test out and evaluate individual rows that lives in a function. Quite often, this code uses the input to that function like the input parameters. To be able to do that, I need to manually type some fake data and set it to this variable, and then I can evaluate the code. But I think that takes... That slows me down. I was thinking, maybe I can do this in a quicker way, so I have this quicker feedback, so I can run this or evaluate this code much quicker. So my idea was maybe I can use an LLM for this. If I give it the parameters, maybe it can return some random data so I don't have to write it myself. I ended up doing that. I have this source code. I'm loading the REPL with the code. Then I select this function name and the parameters with its data type. I have this prompt that instructs the LLM to come up with fake data based on the tag name and on the data type. And then I can send that to the REPL. I do that with a key command. Then I can proceed by running the code within the function that uses these inputs. This works for all the data types. If there's a custom data type, you need to give the LLM extra context. So that's something to think about. Once it knows the context, it can generate this fake data that very often is good enough just to test out, you know, like I've done here, like string... sorry, list destructuring and parsing and things like that. I think that was all I had, and thank you for listening!

Captioner: sachac

Q&A transcript (unedited)

[00:00:00.000] Developing Python in Emacs
So as we wait for people to drop in their questions and so on, I'll just kind of add a couple of thoughts. I'm not personally much of a Python developer, so I'm just curious as to your thoughts on using Emacs and kind of the development methodology you showed as a way to learn Python. Yeah, I think Emacs is perfect for it. If you're already an Emacs user and you have the keyboard commands and everything uploaded in your brain, then learning Python shouldn't be too difficult, because there's a lot of good packages. For Python, I'm using a thing called elpy, which is really good. for Python development. So you have this connection to a running backend, the Jedi Python backend that will give you autocomplete and some refactoring and things like that. So I think Emacs is really good for Python development. So what are some other...
[00:01:16.680] Q: How long have you been using Emacs?
How long have you been using Emacs? Kind of a general question. Sorry if you covered this and I missed it. It's not that long actually. I think it was in 2019 or 2020. It's when I started to, I was really fortunate enough to join a team that had chosen to develop an app in Lisp, in Clojure. yeah and i didn't know anything about it i didn't know lisp at all so it was like brand new so i really had struggling to to learn it but at the same time i was looking for okay which editor should i pick to to do this and i was before that and like an ide person so i do use this big ideas and this time i figured maybe i should try something different. And I read this book about Clojure development and the author was really into Emacs and he was like, Clojure is really good for, Emacs is really good for Clojure development. So that's how I started. So I like to learn, rewired my brain to learn all how Emacs works as opposed to like the mainstream editors out there. So that's how it started. So like five years ago, maybe. Cool. Um, so did you, what were some other,
[00:02:39.520] Q: What editor were you primarily using before?
what, what, what editor were you primarily using before you? Um, um, back, back then it was like, I think it was like Sublime Text. And also some IntelliJ, sorry, JetBrains, their tools. And before that, I was in the Microsoft world, so it's like Microsoft toolings basically. So I moved towards open source tooling and also open source development. Very cool. So I see we've got a couple people that have joined on the BBB here with us for the live question. You're welcome to just throw your question in this chat, or if you've got a microphone connected, you're welcome to come off mute and throw a question in that way as well. You can join on liberachat IRC. You can join pound emacs comp hyphen dev, and we would happily take your questions there. Meanwhile, I've got a question coming in on the pad right now, so take that next. I'll just read it out. I don't quite see how your setup works with IPython. Sorry if you already answered this.
[00:04:00.280] Q: Do you have your Emacs connect to the IPython kernel over the XMPP socket that IPython sets up?
Do you have your Emacs connect to the IPython kernel over XMPP socket that IPython sets up? Oh, good question. So what I've done is I have two different connections. Like if I'm going to do this regular day-to-day Python work when I want to evaluate code without thinking about the external connection, then I just use IPython and I have my I'm using the LP Python package so that you can set up which of the REPLs you want to start when you evaluate code. So that's IPython. But in the second part of the talk where I connect to an externally running app, then I have set up the same configuration but to start Jupyter, the Jupyter console instead. And when I start it up, it's going to, it's like a dialogue that will ask for the ID of a running kernel. Because if you start a kernel, if you start something in a kernel, you will get this unique ID so you can connect to it. So that's kind of the difference. So I have two different ways of starting up the Python kernel. REPLs, either PyPython, which I do for most of my daily work, or if I want this connection to a running app, I'm using the Jupyter console, basically.
[00:05:38.480] Q: Not sure if you've explored Org Babel, but what are the benefits of using your approach over Org?
Have you explored Org Babel at all? Can you talk about your approach compared to the Babel workflow? Oh, interesting. No, I haven't actually, I haven't done that, but I think it's similar to Jupyter notebooks, so you can do some interactive, you can run Python there, but I haven't, I have too little knowledge about it to elaborate on that. Sorry. Very good. All right. I'm just going to peek over to the other chats and make sure I'm not missing questions. Sorry, I got a lot of screens. A little dance here. All right, and I do see a few people in the chat. I'll just say again, if you've joined us on the BBB, you're more than welcome to jump in with your questions or put questions into the chat here. I'm not a big Python person, so I can't get into the really interesting questions that are probably more relevant, just because a lot of it's kind of over my head.
[00:07:05.800] Q: Have you seen marimo.io notebooks? It uses standard Python.
Have you seen the marimo.io notebooks? These use standard Python? Oh, yes, I've seen, I haven't tried it myself, but I've seen it. It's very, I think it's a very interesting approach that probably will solve a lot of these things with interactive development in general, but because I think it's basically Python files and not the, not this other file format that you have for the regular notebook setup. So I have that in my list of things to try out in future, but it looks really, really cool. Awesome. Yeah, I'm hesitant to just ask you really boilerplate questions. You get an expert, a souffle chef, and you ask about making pasta or some other sort of thing. No, go ahead. Go ahead. I'm glad to ask any questions.
[00:08:20.560] Q: what LLM did you say you've used?
So just generally about Emacs as a coding developer, what are the features that you tend to lean on more heavily? You know completion or you know, what have you... Maybe I could put it better as: what have you spent more time configuring for Emacs to facilitate development? Yes, initially, it was basically taking away some of the UI features, basically disabling it, but fairly quickly, for both Clojure development where I was using CIDER, which is like a really good tool, and for Python, when I quickly found elpy, I'm not sure how to pronounce it, but you get so much good things from these tools... So what I've done additionally is theme and maybe having this environment that is without disturbance. So I really like the code to have my full attention. So I rarely have, maybe I have like a preview sometimes, but mostly, it's the window or the buffer where I have my code, and I've minimized the running REPL buffer. So sometimes it's, I'm hiding it, but it's still active. So basically that's how I configured, spent my configuration doing that. And also lately, trying out some LLM tools. I think I have two packages that I'm using actively today. My favorite is a tool called ECA, which is a LLM chat assistant, which is really good. The developer is also an Emacs enthusiast.
[00:10:28.480] Q: Are you building your own Emacs, or using the system packages?
So are you the type of person that tends to work with like a master builder, build your own Emacs? Are you using the system packages? What's your approach to just get Emacs up and off the ground for yourself? I'm not there yet with doing my own builds. Currently, I'm on macOS, so I'm installing Emacs through brew, basically, homebrew. I've yet so much Emacs things to learn because I'm not even using Org yet. I'm exploring the this editor. So I feel like I'm very much a beginner. I'm not using Emacs to its full potential yet. So I'm counting on that I will be more and more using Emacs for more and more different tasks basically. And I'm just seeing a comment. Ed Stallthroat says, thank you for publishing your configuration and offers a link. Oh, great. Great. Yes, those things that I showed in my talk, I haven't made any, I haven't packaged it yet, but everything is on my GitHub, and I've tried to separate each feature in separate LSP files. So it should be, I hope that it's fairly straightforward to understand how it's set up. Very good. Let me just scroll down, make sure I'm not missing questions here. Okay, I think those are the questions that we have. We can take a couple more minutes if you're open to that, just for people to consider. I know we're coming up on the lunch hour, so it may be that people are jogging off to get some food. Yeah, in Sweden, it's dinner time. It's like 6 p.m. soon. That fits.
[00:12:40.080] Q: Are you involved with local Emacs meetups?
So are you involved with local Emacs meetups? I know that your part of the world has a really brisk community, you know, a lot of, just a lot going on. I'm always seeing in Sacha's Emacs News, all of the different meetups and so on. I'm curious if you're plugged into a local community there at all, No, I haven't. Not in any community. I haven't done that yet, but we are... Like, friends that I've gotten to know through work and through like open source work that also use Emacs. We talk and share our conflicts basically on a regular basis, because we are, at least in the Python community, we're like a minority, like other editors are like the standards. And at work, I'm sharing what I'm doing, just like my talk here to my fellow co-workers that are not on Emacs, but I hope to get their interest up. So many of us Emacs users are Emacs evangelists. Yeah. It's the Venn diagram of Emacs user and Emacs evangelist is a circle, I think. Yeah, I think so too. Well, David, I super appreciate your talk and thank you so much also for... I'm sorry, I saw another question here. Let me cover that.
[00:14:24.640] Q: Can you repeat the name of the LLM that you specifically mentioned there? - ECA
Can you repeat the name of the LLM that you specifically mentioned there? Oh, it's a tool called ECA, E-C-A. It's basically a server and clients, and it's not only Emacs. It has support for other editors, but I think the primary support is for, at least the developer who does it is an Emacs user, and so it's like Emacs first, basically. And his name is Eric Dallo. He's a great developer in the Clojure community. He has done some LSP work in the Clojure world, too. What was the last name? What was Eric's last name? Eric Dallo, D-A-L-L-O. Yeah. So I think that's a great tool, but I also use another tool that for some reason I forgot to have this interactive way of, because I think they are developing that feature too, but I have another LLM that has support for requests and callbacks basically. So you can do something programmatically. So that's how I've solved my LLM thing where I select some Python variables and tell the LLM to populate it with some fake data so I can send that to the REPL. But for some reason, I forgot the name of it. I can look it up. Just give me one minute, sorry. Oh yeah, gptel, that's that's the name of the other tool which is also a good tool, yeah, but ECA is more like a chat companion and this, you can also tell it to write code and things like that, but I mostly use it for reviews and research and asking questions because I want to... My... I don't have a problem with typing code. I don't think that slows me down, but maybe getting some quick feedback on the actual work that I'm doing, that's what I'm using LLM for, on LLM for two, mostly today. That really hits for me. I think somehow having things be so simple and in many cases just automated, Emacs just does the right thing. It makes me hungry to type. I want to type more. It feels like every keystroke does a little more. Speaking of evangelism, thank you. Thank you pop up for the for the question. I'm just going to peek one more time through and I don't see a backlog. There are a couple more comments here linking to the ECA Emacs and gptel. Comment also saying very cool setup. So I suppose I'll hand it over to you if you have kind of final thoughts or additional wisdoms you want to share I really appreciate your taking the time to prepare this presentation especially to do the live Q&A with us.
[00:18:08.742] Wrapping up
I will of course continue evolving my setup, because there's new tools and maybe new ideas also coming up. What I'm aiming for in Python development is the great developer experience of what's called Lisp programming, basically. So that's what I'm aiming at for Python too. Thank you. Thank you so much.

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

Back to the talks Previous by track: Swanky Python: Interactive development for Python Next by track: Emacs, editors, and LLM driven workflows Track: Development