The Wheels on D-Bus

Ian Eure (ee-uhn you-er, he/him/his, IRC: ieure)

The following image shows where the talk is in the schedule for Sun 2022-12-04. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 16-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-dbus
Discuss on IRC: #emacsconf-dev
Status: TO_CAPTION_QA

Talk

00:00.000 What is D-Bus?
01:07.880 Why D-Bus?
01:45.360 The D-Bus Model
02:55.360 Well-known Busses
03:20.000 Common interfaces
04:06.240 Emacs Native D-Bus
05:09.320 Debase
05:23.880 Debase: Objects
06:13.440 Debase: Retarget objects
06:51.080 Debase: Object binding
07:12.480 Debase: Raw binding
07:29.400 Debase: Codegen
08:14.200 Debase: Codegen example
09:08.680 Debase: ObjectManager
09:44.480 Demo: Discomfort
11:01.480 Demo: Remote eval
13:16.280 Demo: Remote Org capture
14:11.880 Future directions

Q&A

00:46.840 D-Feet graphical debugger
01:37.000 Emacs desktop environment
04:58.160 How long has D-Bus been around, and what was in place before that?
07:48.360 Why is everything D-Bus prefixed with "org."?
08:28.480 Do most OS/desktop environment/window managers interoperate well over D-Bus?
10:08.720 There is a lot of criticism against D-Bus out there. Why do you think that might be?
11:37.560 Which system services come to mind when thinking about applications, be it at the OS/DE/WM level?
12:19.720 When it comes to managing devices, how are D-Bus and udev related?
13:33.280 What is something D-Bus does that you couldn't do before? What is a really cool use of D-Bus in a modern desktop environment?
15:08.880 Can you explain briefly what clients and services can do with properties?
17:49.920 Is there such a thing as a D-Bus reflection browser, maybe Emacs-based, that lets you discover all the behavior different D-Bus app participants provide?
18:17.760 Next question, D-Bus seems great for extensibility, but then Emacs has no such mechanism and is fantastically more extensible. Why do you think this is so?
19:25.200 Do you have any other cool D-Bus ideas?
19:54.640 Are there buses besides system and session? Is there anything more to a bus besides a way to group objects?
22:52.280 What do you use it for?
25:12.480 It looks like dBus is mostly useful for Emacs to do IPC -- IIUC, this is how synctex works when working with LaTeX docs. How does it compare with other ways of doing IPC, for example, communicating over a socket with MPD?

Description

In this talk, I’ll explore uses of D-Bus that supercharge your Emacs Operating System.

Discussion

Notes

• • https://codeberg.org/emacs-weirdware/debase

• https://codeberg.org/emacs-weirdware/discomfort

• re: "pushing mindshare"..  Yes, you were very successful on that!

Questions and answers

• Q: This is such a great overview of dbus.  I hadn't been paying attention to this space because it seemed to be in flux (10-15 years age).  How long has dbus been around and what was in place before that?
  • A: D-Bus dates to 2002, but really saw stabilization and adoption in the 2010s.  There wasn't really anything prior to D-Bus, you'd do some of the things it does by shelling out and driving command-line programs --- not a great approach.
• Q: Forgive me if this question is silly: Why is everything dBus prefixed with "org."?
  • A: D-Bus services generally use reverse-FQDN notation, similar to Java packages, and most stuff using it is not-for-profit software, so org. is a very common prefix for those.
• Q: In your investigations, do most OS/DE/WM interop well over d-bus? Which one(s) have proven more challenging, if any?
  • A: Since D-Bus is pretty uniform, DE/WM considerations aren't a large concern.  D-Bus isn't widely used on non-Linux systems, so OS differences have little to no impact.
• Q: Re: using EXWM as a Desktop Environment -- Does EXWM provide a session manager (daemon)?
  • A: No, but it looks like it can work with external X11 session managers.
• Q:There is a lot of critisism against d-bus out there, why do you think that might be?
  • A: Because it's not very good.  It uses XML, which isn't hip, and I think a lot of people have a knee-jerk negative reaction to.
• Q: Which system services come to mind when thinking about applications, be it at the OS/DE/WM level?
  • A: Stuff that interacts with hardware: turning WiFi on and off, connecting to networks, pairing Bluetooth devices.  The kind of stuff you find in the upper-right of the dock/menubar in a traditional DE.
• Q: "If you want to do the kinds of things that dBus does, you're limited": What is something dBus does that you couldn't do before? What is a really cool use of dBus in a modern DE (KDE/Gnome etc)?
  • A: D-Bus is fundamentally about making it easier to do things, and using that increased ease of use to broaden the number of places that kind of thing gets done.  So there's some prior art for doing some of these things, but none of them is as easy as doing it via D-Bus.  As far as specific features: I've been using computers a very long time, and it's still magic to me that you can just plug hardware in and immediately use it without having to do anything --- D-Bus is foundational to that kind of interactivity when it comes to things like storage, network, and Bluetooth-connected devices.
• Q:  When it comes to managing devices, how are dBus and UDev related?
  • A: They're orthogonal.  I believe UDisks2 (which is the D-Bus service for managing disks) talks to UDev2, but this is abstract & not a detail you have to care about as a client of UDisks2.
• Q: As an average GNU/Linux user, I've used signals and methods before but not properties. You gave an example involving properties, but it kind of flew by. Can you explain briefly what clients and services can do with properties?
  • A: Properties are metadata associated with an object interface[.   They can expose r/o information about the object (the name of the host; the UUID of a disk device; the hardware address of a network device), and you can modify certain properties about the object or service by writing to them.  For example, org.freedesktop.NetworkManager.Device has an AutoConnect property, which you can use to enable/disable automatic connection.
• Q: what is the name of the dbus GUI you showed?  defeat? dfeet?
  • A: "d-feet" 
• Q:  Naive Q (me not knowing much about dBus): Is there such a thing as a dBus reflection browser (maybe Emacs based) that lets you discover all the behavior different dBus app participants provide? Thinking something like what macOS Automator does? (actually, wait, think you're showing it)
  • A: d-feet is the one I showed: https://wiki.gnome.org/Apps/DFeet
  • I'd very much like something similar, but built into Emacs.
• Q:dbus seems great for extensibility. But then Emacs has no such mechanism. And is fantastically more extensibile. Why do you think this is so?         
  • A: I think these are different kinds of extensibility.  Emacs is much more malleable than anything D-Bus offers.  You can't change how existing D-Bus services work in the same way as you can with Emacs customizations, variables, advice, or just going and hacking on the code live; you can only add new features by creating new services.
• Q:  Do you have other cool dbus ideas?
  • A: I greatly want to see:
    • A D-Bus browser.
    • discomfort expanded and made better looking, so there isn't a single storage-related task I can't do with it and dired.
    • A NetworkManager interface, so I don't have to use nmtui/nmcli.
    • A Bluez (bluetooth) interface, so I never have to see bluetoothctl again.
• Q: Are there Busses besides System and Session? Is there anything more to a Bus besides a way to group services objects?
  • A: There's always at least a System bus; theres one Session bus per logged-in session (so potentially zero).  You can create and connect to as many busses as you want, and they get identified by the socket they listen on, which can be a local UNIX socket or a TCP socket.  This isn't a common usecase, most things use system and session.
• Q: It looks like dBus is mostly useful for Emacs to do IPC -- IIUC, this is how synctex works when working with LaTeX docs. How does it compare with other ways of doing IPC, for example, communicating over a socket with MPD?
  • A: D-Bus provides a uniform framework for building these services.  MPD's socket interface only works with MPD, and if you want to connect to one, you have to write code that speaks the protocol before you can do high-level things like "play" or "pause."  D-Bus provides the underlying communication layer as well as the model for the API, so you get all that stuff for free.  If a music player has a D-Bus interface, and you're using a language with D-Bus bindings, you can go from zero to "support playing and pausing" in one line of code.
• Q: What mainstream/popular d-bus alternatives have you seen out there, if any, maybe beyond pipes, udev, and such? -> somewhat redundant with the above
  • A: There aren't any alternatives on Linux that I'm aware of.  Other operating systems have different ways of doing things similar to what D-Bus does (such as OSA/AppleScript on macOS).
• Q: I see a python dbus tutorial, does it make sense to have an emacs version? https://dbus.freedesktop.org/doc/dbus-python/tutorial.html 
  • A: That's a really good idea!
• Q: How if at all do the various web browsers interop (well) via d-bus?
  • A: It could be better.  Firefox (and forks) have a simple interface that lets you open a URL, but nothing else.  I'm not sure about other browsers, since LibreWolf is what I use.  If you use d-feet to examine the session bus, you can see!
• Q: Nice, thanks! I had been looking for a udisks tool. Is it available?
  • A: Yes, https://codeberg.org/emacs-weirdware/discomfort It's alpha, but way nicer than using udisksctl.

Other discussions from IRC:

• Very interesting talk Ian!
• Pretty interesting... I am already scratching my head thinking on applications!
• Thanks, fascinating ieure, now onto refactoring everything not yet into Emacs to be managed in Emacs via d-bus...
• One usage example I've recently stumbled over is the scad-dbus package which can control the openscad gui from Emacs, move around the camera etc.
• @ieure : Thanks for the great talk, love these expositions of yours, especially of things that have gone under-appreciated - and the packaging into timeless tunes obviously!

Transcript

[00:00:00.000] Welcome to my EmacsConf 2022 talk, The Wheels on D-Bus. In this talk, we'll cover what D-Bus is, why you might want to use it, and how to use it with Emacs. D-Bus is fundamentally based on passing messages in between processes, using the bus as a mediator. On top of this is built an RPC system with method invocation that has argument lists and return values, like you might find in any programming language. These are commonly used for verb-type actions like "restart my computer." You can also associate a collection of attributes with objects on the bus, and these are called properties. The properties can be read-only, write-only, or read-write. Signals are a way of notifying participants on the bus of updated state, and are the basis for building dynamic user interfaces that react to changes in the system. It has a static and strong type system, so if you send a message with the wrong type signature, it simply gets rejected instead of going through to the remote service. It also manages service life cycles, so you're not running services at all times. They can be started and stopped by D-Bus on demand.

[00:01:07.880] D-Bus has two major use cases. The first is acting as a lower-level substrate for higher-level programs, like a graphical desktop environment. For example, if you want to manage your network connectivity from your graphical environment, instead of having to build all of that from the ground up, you can rely on the D-Bus service to do that and only build the graphical component of it. This gives you consistency between desktop environments and reduces code duplication. Another application is automating desktop programs. If your program offers a D-Bus service, then it can be remote-controlled, and if all of your programs offer D-Bus, you can control your entire desktop.

[00:01:45.360] Let's look at the abstractions that D-Bus provides. The top level object is called a bus, and it's like a partition that messages get exchanged inside of. Messages don't cross buses. Inside of a bus are services. Services are normally identified in reverse FQDN order, so org.foobar.FooService. Each service provides some set of features related to a particular area of functionality. Inside of each service are objects. Objects use a path notation, and usually follow the same reverse FQDN format as the service identifier. Each object has one or more interfaces. An interface is like a facet that you can use to interact with an object, and inside of the interface are properties, methods, and signals, which we covered before. Properties are attributes that can be read or written. Methods are verbs that you can call to invoke an action, and a signal is something that's used to move state in between a service and another participant on the bus. There can be any number of interfaces on an object, any number of objects in a service, and any number of services on a bus, and any number of buses on a system.

[00:02:55.360] There are two well-known busses, and these roughly map to those two use cases I mentioned before. The system bus is for interfacing with hardware and operating-system-level concerns like disks, networks, and so forth. The session bus is tied to a user login, and is more in the desktop automation use case.

[00:03:20.000] There are some common interfaces you'll find if you go exploring D-Bus. The Introspectable interface is the basis of a lot of the reflection features. It has a single method called introspect that returns the XML interface description of whatever you call it on. Peer is used for lower level connectivity, for example, pinging a service to see if it's running. And the Properties interface is the basis of the read-write properties, which are secretly method calls under the cover. Just about every object you interact with on D-Bus will support all three of these interfaces. Additionally, ObjectManager is used for services that manage collections of objects. For example, the disk service has an object for each disk that's attached, and the object manager allows you to enumerate all of those.

[00:04:06.240] Emacs supports D-Bus natively since version 23.1. It's a combination of native bindings with a C library and dbus.el. While there are some ports of D-Bus to non-Linux operating systems, it's probably only available on Linux and almost certainly only usable on Linux. If you want to interact with D-Bus from Emacs, it's fairly straightforward. There's a collection of functions like dbus-get-property or dbus-call-method, et cetera, and they almost all take this same set of four arguments at the beginning: bus, service, path, and interface. In this case, it takes a single additional property, which is the one to read. And what we're calling is the hostname1 service, which gives you just a little bit of information about the system, like its hostname or its chassis. And in this case, you can see I'm running this presentation off my laptop. The problem with this and what I don't like about it is that all of these identifiers are very verbose and very repetitive. And if you end up calling these a lot, it gets old really quickly.

[00:05:09.320] So I wrote a wrapper called Debase, which is convenience on top of the built-in functions. Most of the stock functions have Debase versions just by replacing "dbus" with "debase". And let's look how that works.

[00:05:23.880] The fundamental idea of Debase is that you can bind together all of those arguments into a single object that represents the endpoint. This is an EIEIO class, and it takes keyword arguments, so there's never any chance of mixing up which thing is what. So this sets the endpoint to that object, calls debase-get-property on it, and you can see it works exactly the same. The thing that's really nice about this, though, is it knows that so many of these arguments are very similar that it can compute most of them if you don't provide them all. So if you just say service, it will assume that you want the same object that matches and the same interface that matches, and it works just the same. I find this very, very convenient. You can also reuse the object instead of having to repeat every argument with every function call, which is a really great improvement in ergonomics.

[00:06:13.440] Because so many objects have multiple interfaces, you often find yourself needing to look at a different aspect of that object. This is supported with the built-in EIEIO clone method, which takes an object and a set of keyword arguments to replace. So in this case, we can see we're calling the Properties method, but everything else on that endpoint is the same. And then we're gonna call the method GetAll on that Properties interface, and it's going to return all the properties of the org.freedesktop.hostname1 interface inside of that object. And if we run that, we can see there's the hostname and some other information about the laptop that I'm running this on.

[00:06:51.080] Debase also supports object binding. This creates a lexical context in which the Debase object is the implicit target of any D-Bus function. This is really convenient if you need to fetch multiple properties or otherwise interact with the same endpoint in multiple different ways. And you can see I'm still on a laptop and it's still named meson.

[00:07:12.480] You can also, if you don't want to use the object, you can provide the raw argument list. Under the covers, this is basically an flet where you're currying all of these functions so they start with those argument lists. And you can see I'm running on a Linux machine, which should not be surprising.

[00:07:29.400] Debase also has an experimental code generation feature. It outputs EIEIO code with one class per D-Bus interface. This includes accessors for all of its properties with an in-process cache, so if you read one property, you don't have to go back to the bus to read it again. It also outputs generic functions and method implementations for the D-Bus interface methods. It includes name-mangling options, so you can control how everything is named. And you can generate the code either via introspecting a live system or providing an XML interface description, which is handy if you want to use it as part of a non-interactive build. I think this has a lot of promise, but it doesn't feel quite right yet, so any feedback or contributions are very welcome.

[00:08:14.200] Let's generate some Elisp code for that hostname1 service we were interacting with before. debase-gen-class is the generation class, and it says to create a class that matches this interface, named "hostname1", and then the rest of these arguments are the same ones to target the endpoint, just like with debase-object, because it extends debase-object. debase-gen-code is a generic function that takes any debase-gen class. There are different classes for functions, properties, et cetera, and it creates all of the code for it. If we evaluate it, we can see the results look about like we would expect: creates a defclass named "hostname1", which extends debase-object, has all of the slots and accessors defined, and then methods that define everything that you might want to do with it, including documentation. This is based on introspecting a running system, but as I mentioned, you can provide an XML interface description instead, if you like.

[00:09:08.680] Debase also comes with "debase-objectmanager", which is convenience for the D-Bus ObjectManager interface. This is used in a lot of places in D-Bus, where an object manages other objects. For example, the NetworkManager object manages network hardware objects, and using the ObjectManager interface, you can enumerate all of the network hardware, and by subscribing to the signals, you can be notified when they change. "debase-objectmanager" keeps a local cache, and will fire a callback on any change. So it's the building block for that dynamic user interface, like you would see in a desktop system, but inside of Emacs.

[00:09:44.480] Let's do some demos. Discomfort is an interface I wrote for UDisks2, which is what manages all of the block device hardware. And again, it has that dynamic desktop-like interactivity, and mostly will just do what you mean. This is definitely alpha state. It doesn't have all the features, but it's good enough that I use it daily. So here's Discomfort, and you can see it has a list of all your hardware, what type it is, and where it's mounted. I have a little USB extension cable here, and I'm gonna plug in a disc, just to show you how this works. You can see when I plug it in, just a moment later, it shows up in that list, automatically. I don't have to press any key, I don't have to refresh it, it's just there. If I unplug it, it's gone. Plug it back in, and there it is. And you can see it's an encrypted volume. So in order to do anything with this, I'm going to have to supply a password. Just pressing Enter goes into the "do what I mean" mode, and it asks for the password. In this case, I've chosen the very secure password of "password". I hit Enter, and it unlocks it, and it mounts it, and it opens "dired" looking at it. And here's a little README. Let's see what it says. "Hello, EmacsConf." So that's my demo of discomfort.

[00:11:01.480] In addition to acting as a client for D-Bus, Emacs can also offer services to other D-Bus clients. This is a really interesting opportunity because it allows many different programs to integrate with Emacs in ways that were previously very difficult. You can use this as an alternative to Emacs. The difference is D-Bus provides a full API, so instead of emacsclient being a sort of fire-and-forget system, you can actually get results back from the remote operation. So here's some code. Here's a dbus-eval function, which takes a string, reads it, and evaluates it, and returns whatever that value is. Then we have a debase-bind block that sets up an object on the session bus. Again, that's my user login bus. It offers this D-Bus service Emacs. This is a constant inside of the dbus.el package. And again, the path is a constant in there. And we're gonna create this interface, org.gnu.Emacs.Eval, and then register a method called Eval that calls that dbus-eval function. Pretty straightforward, only a handful of lines of code. To test this out, we're going to use the dbus-send utility. This is a command line program that interacts with D-Bus. We're going to tell it to wait for and print the reply, that the message should be sent to the session bus, that we're going to talk to the org.gnu.Emacs service on that bus, and the /org/gnu/Emacs object inside that service. On that object, we're gonna interact with the org.gnu.Emacs.Eval interface and call its Eval method. We're gonna call that method with a single string argument, which is indicated by the string prefix, and then a form to evaluate. I actually have to run this from a shell, because if I try using it in Org, it wedges. org-babel blocks waiting on completion, which blocks the D-Bus service from responding. I really wish Emacs was multi-threaded. But let's try it out. So if we run this, we can see that we get a return, and that's an unsigned integer of 32 bits with a value of 3. So like I was saying, this is really a two-way API where you can communicate back and forth between Emacs and another program. It's not just fire-and-forget. I think that's really cool.

[00:13:16.280] Let's try another demo. What about a remote org-capture? What if you could trigger an org-capture from any program on your desktop? I think that would be pretty cool. And we can see, there it is. All right, I think I've got that one covered. So I do want to say that remote eval is probably a bad idea from a security perspective, but the point of this is some quick and dirty demonstrations of what can happen and to get people's imaginations flowing, because I think this is something that offers a lot of promise for Emacs. I think having a full-blown Emacs desktop environment where it can do all the things that a GNOME or a KDE environment can do is very exciting. And if you want to have a traditional GUI with Emacs as a more integrated participant of it, its service mechanism offers a lot of ability to do that.

[00:14:11.880] In the micro sense, I think there's a lot of improvements that can be made to either dbus.el or to dbase. The main one is handling of the type system. Lisp's dynamic type system doesn't mesh particularly well with the static strong type system that D-bus offers, and having some convenience to assist that would be very helpful. There's also some weird interfaces. For example, some things return identifiers as an array of integer code points instead of a string, and there should be a common way of handling that. I also think that the service support could be improved. Even though I gave the demo service, it's not really a great D-bus citizen because it doesn't offer that introspection mechanism, and so the actual methods are pretty much invisible to other participants, unless they already know that you're using Emacs. That's my talk. Thank you. You can find me on mastodon.social or on libera.chat.

Captioner: sachac

Questions or comments? Please e-mail emacsconf-org-private@gnu.org