Back to the schedule
Previous: Maxima a computer algebra system in Emacs
Next: WAVEing at Repetitive Repetitive Repetitive Music

Extend Emacs to Modern GUI Applications with EAF

Matthew Zeng

Download video, 22:23, 113M
Download captions

Download compressed .webm video (41.7M)
Download compressed .webm video (28.4M, highly compressed)
View transcript

Emacs Application Framework (EAF) is a customizable and extensible GUI application framework that extends Emacs graphical capabilities using PyQt5. This talk will cover the architecture design of the EAF project, and demonstrate some of its most useful applications: modern browser, PDF viewer, video player, etc.

  • Actual start and end time (EST): Start 2020-11-29T16:05; Stop 2020-11-29T16:28

Questions

Q9: Do you think that this tecnology could to be on core of Emacs any time? or fork of Emacs?

Not yet and I don't know if it ever will, since EAF uses many other dependencies that one needs to install themselves (see README); and I don't think all of them is GPL (though using open source licenses).

Q8: I use PDF-Tools currently for my PDFs inside Emacs, would you consider this a better alternative to that and if so why? Although I am definitely trying it because the browser looks incredible, possibly the best implementation of an Emacs browser I have seen, I would love to hear your opinion on the PDFs compared to something like PDF-Tools.

PDF-Tools is great, it would be an awesome option if you can't run EAF on your machine. However EAF PDF Viewer is just a lot faster and smoother as it uses PyMuPDF as its backend.

  • Oh, thats great actually, I have noticed it being a little choppy at times, I am excited to try EAF in general because it looks awesome and if its faster than pdf-tools I will probably also switch to it for my PDFs. Thanks a lot for the talk, one of my favourites in this EmacsConf, it gave me a lot of great tools to try inside Emacs!!
  • Thank you!!!

Also because PDF-Tools is much older than EAF, it had more attention and more people working on it, so there are definitely more features than the current EAF.

Q7: Can you use the PDF viewer as a viewer for LaTeX files, with reverse search support with e.g. AucTeX?

You could do that with some simple Elisp functions, and EAF PDF Viewer now updates itself automatically whenever there is a change to the file.

Reverse search is currently not available, we need more people to help us work on it! :-)

Q6: What JavaScript engine is the web browser in EAF using? Also, what web browser engine is it using?

QtWebEngine,. (from the Qt Wiki: )

  • Qt WebEngine uses code from the Chromium project. However, it is not containing all of Chrome/Chromium;
  • Auxiliary services that talk to Google platforms are stripped out (nice).
  • The codebase is modularized to allow use of system libraries like OpenSSL.
  • Binary files are stripped out.

Q5: Does the web rendering happen in a subprocess, or can loading a big page cause Emacs to lag?

Not at all! And that's one of the biggest advantage of the EAF project, it utilizes all the powerful Python features, like multithreading.

Q4: Do you have control over the JavaScript that runs on these pages? Also is there a blocklist feature? (True ad blocking might be impossible, I understand)

As my talk just (or will be shortly) mentioned, you can disable JavaScript altogether. So far there isn't a blocklist implemented, but I don't see a reason not to be able to implement this feature in the near future. EAF itself uses JavaScript (free code) to implement some browser features (like the Vimium binding), so turning off JavaScript will make the feature stop working as well.

Q3: (Feel free to ignore this one if it is off-topic) How big is free software movement in China? Is there any organisation like FSF there?

Very recent years there are A LOT of open source movement in China, however not free software strictly speaking;

There are a lot of open source clubs in the chinese unversities now that people actually starting to get interested about open source in general, that's a huge improvement than decades ago I'd say. There are still many places to improve.

Although not Free Software Foundation, literally this year the first ever open source foundation is established in China, called OpenAtom Foundation: https://www.openatom.org/#/ (in chinese).

Q2: Is there anyway to implement EAF without the reparenting behavior from X11?

That's one of the challenges right now to get EAF working on other platforms. We're always looking for people to help out.

  • Are there any ideas on this at all? I can try to help out but don't know what's even been tried (and perhaps has already been ruled out).

Q1: Have you experimented with using Hy (aka Hylang, a Lisp that compiles to/runs in Python) for EAF, to avoid having to write "real Python"?

Not yet! Will have a look into it later :-)

Notes

Transcript

[00:00:03.600] Hello. Hopefully everyone is staying safe and staying home, I feel very grateful to live in a world today that technology and free software can be leveraged to connect people in such disconnected and difficult times, and to have an online conference like this. Hopefully you've all enjoyed this year's EmacsConf so far. Many thanks to all the people that made this possible. Anyways, welcome to my talk "Extend Emacs to Modern GUI Applications with EAF, the Emacs Application Framework". This will be my first ever talk, so apologies for my inexperience, let us begin. About me: my name is Matthew Zeng, you can also call me MT or Mingde. I'm a Chinese Canadian living in Toronto, Ontario. Offline: I'm an undergrad studying mathematics at the University of Waterloo. Online: I'm one of the admins of the Emacs China — the largest Emacs forum in China. So, to all Chinese listening to my talk right now, feel free to check it out. And this is a link to my GitHub profile, (and) to my projects I'm involved in. One's M-EMACS which is I'm the author of — a user-friendly full-featured Emacs configuration distribution, it is what I'm using right now, as well as the Emacs Application Framework which I help to maintain along with the other author lazycat, which of course, is today's topic.

[00:01:35.759] So, as you all might have already noticed I'm currently using Emacs and opening navigating closing all these websites that are rendered properly all within Emacs, it's all thanks to the EAF project. So, we're living in a society that's heavily dependent on the internet and multimedia, it is unavoidable to run to some occasion that you need to open a fancy website that uses JavaScript and CSS, or you need to watch some videos. However, due to the nature and history of Emacs, it cannot render all these modern graphics effectively and efficiently. Emacs is solely a text-based editing environment, and I argue that this is not a bad thing, in fact, it is one of the reasons that me and I believe many of you as well are attracted to Emacs in the first place. Unfortunately, this results in us having to open a dedicated web browser to browse the internet, open a dedicated video player to watch some videos, or a PDF renderer to read some documents. So far Emacs cannot do all these tasks on its own but can only be achieved using other external applications. So, the other author manateelazycat, or lazycat in short, didn't want to use all these external applications, he wanted to have an uninterrupted Emacs experience, he wanted to truly live in Emacs. However, it would be a lot of work to build this modern application from scratch, there's simply no time or research to do that. So, lazycat thought of utilizing existing applications and to try to make it collaborate with Emacs, there are many solutions available, one of it is the Emacs X Windows Manager, and I'm sure a lot of you already know that — the EXWM. However, it didn't work for him, because although EXWM opens the door to use other applications within Emacs, it as a fine window manager cannot modify, customize, or extend other software from Emacs. For example, it cannot modify the behavior when you press a key in Chromium or PDF viewer, therefore it cannot utilize the rich Emacs ecosystem that's been growing for almost 40 years.

[00:03:57.360] On the other hand, in the EAF browser, so, if you M-x eaf-open-browser-with-history, you can see on the lower half of my screen — a list of histories sorted by my personal most visited sites, and you can search for a site that you've been to or search for some keyword in a search engine. So, this is all achieved by utilizing the popular completion framework in the Emacs ecosystem — ivy. So, lazycat decided to develop a solution of his own in 2018, namely the EAF project, so, I joined the development last year, 2019. EAF is a highly customizable and extensible GUI application framework that extends Emacs to graphical capabilities using PyQt5, and it is not a window manager. Alright. So, in the README, you can see a list of GIFs showcasing all the available EAF applications, a browser, a markdown previewer, a video player, a PDF viewer, and more. Today I don't have enough time to demonstrate each one of them, but I will select a couple applications to show you.

[00:05:21.120] So, since we are already using EAF browser, we'll start with this. Besides using the classic Control n (C-n), Control p (C-p) you can also use the Vim style hjkl to move up or down. Also, Meta Shift comma (M-<) or g (moves) to the beginning of page, Meta Shift period (M->) or capital g (moves) to the end of page. Vimium and Surfingkeys are popular keyboard-based browsing techniques in Chrome, and they've imported here as well. You can press f to toggle markers pointing to all the links in the current page, say, I want to visit the wiki — which comes very very handy when you want to configure EAF to your liking, so you see the marker on top of wiki is dd, press dd and Enter (RET), and now you are navigated to this link, so you don't need to use your mouse at all.

[00:06:13.840] So, a full list of key bindings can be found when you (press) Control h m (C-h m), just as any other Emacs major mode, so you don't have to remember everything… all the key bindings I said to you. So, this is a global binding application to every other EAF application as well. You can find it under the wiki in the keybindings section, so press f again and use ns, press Enter (RET), now you're in the keybindings web page. You can see all of the keybindings available in every EAF application, and you can try them out, and you can customize your key bindings using eaf-bind-key, you can customize Control n (C-n) as in the web page to scroll up in the EAF PDF viewer, or you can unbind an existing binding using eaf-bind-key, bind it to nil, so it doesn't bind to anything. Okay, so, here comes the important part, if you want to customize EAF, you should visit the customization page in the wiki. Now, I press Meta b to go back in history, and go to the customization page, press f, press ad, Enter, and now we're in the customization page. So, the first customization option you see is dark mode, let's say, if you want to turn on the dark mode for EAF browser, and you don't want to use your mouse to do all this stuff. You press c, and you can select c to toggle the caret browsing, you can see a lot of markers available, poped up again, but they're not on top of links but instead of paragraphs. You select the paragraph of your choice, in this case you want ls which comes here, and then you just move the cursor like what you always do in Emacs, and now you select everything and use Meta w (M-w) to copy the text. Now, we (run) Meta Shift colon (M-:) to evaluate what we just copied, and set that to true, and press r or F5 to refresh the page, voilà we have the dark mode enabled. So, there are…, well, let's toggle it back off for now. Now, we (run) Meta Shift colon (M-:) again, and we find the one we just used, and change it back to false, and refresh the page, back in the light mode. So, there are many other customization options available, you can either evaluate like what we just did or add it to your Emacs configuration file. So, in this wiki…, you can make the EAF browser to continue where you left off similar to the Chromium setting. You can make EAF the default browser (in) Emacs by aliasing browse-web to eaf-open-browser, or set the browse-url-browser-function to eaf-open-browser, there's just some tricks. And there (is) also an experimental adblocker currently in place, therefore it can block some elements but not all, so we really encourage people to help us test out and add more conditions in. So, you can…, the EAF Browser is able to download any files from the internet, and it will be downloaded using Aria2. You can also customize the eaf-browser-download-path using eaf-setq, it's a function that we defined similar to setq, the normal setq we know. So, by default the download file is stored in your home directory slash downloads, and you can change that whenever you want. You can also disable saving browsing history, so, remember when I press…, when I use M-x eaf-open-browser's history, I see all the histories here, but if you want more privacy, you don't want that to be available at all. You can turn it off easily with eaf-setq, and set that remember-history to false. You can also set your default search engine. Right now we have Google, although not really good but… Google and also DuckDuckGo which is a better search engine, well, yeah, ethically better search engine. So, you can also configure the zoom. The default zoom of your browser is 1.0, you can convert default-zoom to 1.25, so when you open any web page, it will be zoomed by default. You can also disable JavaScript, although I personally don't really suggest you to do, because it will basically break a lot of our features, because a lot of the browser related features must be implemented using JavaScript, but yeah, you can do it if you really want to. And there's also some customization on EAF Camera you can do as well.

[00:11:47.760] Let's move on to EAF PDF Viewer. Now, let's open the PDF file using EAF. So, that's one something already here, but let's open it here. So, eaf-open, and select "Introduction to Programming in Emacs Lisp". I have it already open, but it's okay. So, you have the file, you have other files displayed… you have all the pages display, sorry. There are 273 pages in total, but notice how fast it is to browse all the pages, it is blazingly fast, that's all thanks to Python and MuPDF which you don't really get from Emacs Lisp. So, let's say if I want to jump to page

  1. We press p and Enter 50.
And here we are, we are at page 50. You can look at the lower right to verify the page you're on. You can use i to toggle dark mode as expected. Let's say you want to find table of contents, so use Control s — the Emacs default binding for I-search, and search for a "table of contents", here we are, it is highlighted for you, and you can Control s for more but there's only one match, you (press) Control g (C-g) to disable the highlight, and you see a lot of options for you to go. Okay. Let's say, if you want to go to the preface. That is, you press f which is also similar to EAF browser, you press f for Vimium, and you see the marker, now change to wn, press wn, and then you can go to the preface. Now, we are at the preface. So, now you finish reading, you want to save your progress? No worries, it is already saved for you by EAF. You can safely close the document using x, and opening again, eaf-open, and the file…, see you are at preface again. You're right at where you left of.

[00:14:16.560] You can also use M-x org-store-link, or Control c l (C-c l) — which I prefer, if you want to save a particular page in a Org mode file. Now, I go back to my presentation doc, I don't need this anymore. So, you just (press) Control c Control l (C-c C-l), or I think M-x org-insert-link. You can find the file right here, and you press Enter (RET), and you press Enter (RET) for the description again, and now it's right here, and Control c Control o (C-c C-o) to open it. Voilà! You're back.

[00:14:54.000] Let's now demonstrate the EAF Video Player. So, M-x eaf-open, you use eaf-open whenever you want to open some file. You use eaf-open-browser if you want to use some actual application that's not really related to a file. So, eaf-open, and select the video you want, so, video-demo, I already have a video demo ready, because I recorded a video of the demo of the EAF Camera, have a look. Let's move to the beginning, "Hello people from the future! This is a demo of the EAF Video Player that demos the EAF Camera feature, so, as you can see on the screen of me inside my camera, and the screen is actually with all within Emacs. (Right, the video itself is as well, haha.) You can open this using eaf-open-camera which I'm already into, and you can press p to capture a photo, so, the photo is by default stored at your $HOME/Downloads directory, and you can modify it freely. If you go here, and you can see the camera stored right here." So, you press Space (SPC) to pause, what I used here is the eaf-open-this-from-dired. Basically, in dired you select the file that should be opened by EAF, and I used that. It detects that it wants to use the EAF Image Viewer, so I accidentally tested EAF Image Viewer before I noticed. That gives the image of the photo I just took using EAF Camera. As you can see, you can use hl — the Vim binding to navigate in the timestamp in the video, and I can use jk to change the volumes of the video.

[00:17:02.320] Alright. Now, you've seen all the basic usages of the EAF project, it comes the question of what is the magic behind it. All right. Let's open the hacking page in the wiki, the design is laid out in a diagram here. Let's put it side by side along with my text, so we can follow through. Right, okay. Let me…, sorry, let me drink some water. This page in the wiki went into a lot of detail, due to the time constraint I will just rephrase some of the ideas here, so for anyone interested, please have a look at the wiki yourself. The easiest way to think about EAF is that the actual GUI application is started in the background, then the frame of the application is attached to the appropriate location on the Emacs window. So, EAF linked Qt5 with Emacs using Elisp and Python. On the Python side which is colored yellow in the image, we have QGraphicsView and QGraphicsScene objects. These are used to simulate the Emacs window buffer design where QGraphicsScene is similar to buffers in Emacs, it controls the state and the content details of the application where QGraphicsView is similar to Emacs window. It populates the buffer (QGraphicsScene) to the foreground at the appropriate position. Whenever an EAF mode buffer brings to a background…. Whenever an EAF mode buffer brings to the foreground, sorry, a QGraphicsView instance is created, and whenever the buffer goes to the background the QGraphicsView instance is then deleted, while QGraphicsScene — the actual process — remains running in the background until the EAF mode buffer is killed. GPU compositing is used to ensure that QGraphicsView and QGraphicsScene is synchronized real time. Using QWindow::setParent function the QGraphicsView is attached to the appropriate location on the Emacs frame, so that although GUI applications are not running within Emacs, they look as if they were. When user types on the keyboard it is first received by the Emacs EAF mode buffer, and then Elisp sends the event to QGraphicsScene using D-Bus. When user clicks on the GUI application it is received by the QGraphicsView and processed in Python. Elisp can communicate with Python through D-Bus, in other words you can customize and extend Emacs not just using Elisp, and now you can use Python, this way one can leverage all the Python properties like multi-threading or some other stuff, the entire Python ecosystem can be utilized as well, such as the Qt web engine that is the basis for our EAF Browser, and PyMuPDF is the basis for the EAF PDF Viewer. This really opens the window to many many new possibilities to extend Emacs using EAF.

[00:20:32.720] All right, back here. We are always looking for people to join the development, there are many many more work that needs to be done, such as testing and debug EAF on more Linux distros and window managers such as i3 and stuff, you can also add new EAF applications, or debug and enhance existing EAF applications, or you can port EAF to native Wayland which I just discussed with the emacs-webkit author Akira Kyle, and he told me that EAF doesn't really work on native Wayland, because it uses XWayland, so it doesn't work on the pgtk port of Emacs. And we also need people to port EAF to non-free operating systems including Windows and macOS, and that's because, like, D-Bus is a Linux specific feature, so it doesn't really work on other platform. We need to check, replace it with some alternative, and QGraphicsScene somehow doesn't really work on macOS, and there are many other to-do lists available, so please have a look and see if there is anything you want to work on. All right. So, since this is a pre-recorded talk I won't be able to do the Q & A real time in the video. However, I will be around on the collaborative pad and the IRC #emacsconf,

emacsconf-questions

to answer any questions when it pops up, and you can also submit an issue on the repo, and you can check the wiki for some other guides and tricks. All right. Thank you guys, and hopefully you find this EAF project very interesting, and enjoy the rest of EmacsConf 2020.

Sunday, Nov 29 2020, ~ 4:07 PM - 4:30 PM EST
Sunday, Nov 29 2020, ~ 1:07 PM - 1:30 PM PST
Sunday, Nov 29 2020, ~ 9:07 PM - 9:30 PM UTC
Sunday, Nov 29 2020, ~10:07 PM - 10:30 PM CET
Monday, Nov 30 2020, ~ 5:07 AM - 5:30 AM +08

Back to the schedule
Previous: Maxima a computer algebra system in Emacs
Next: WAVEing at Repetitive Repetitive Repetitive Music