Back to the talks Previous by track: emacs-gc-stats: Does garbage collection actually slow down Emacs? Next by track: Writing a language server in OCaml for Emacs, fun, and profit Track: Development

hyperdrive.el: Peer-to-peer filesystem in Emacs

Joseph Turner and Protesilaos Stavrou, https://ushin.org xmpp:discuss@conference.ushin.org (XMPP MUC for USHIN discussion), joseph@ushin.org

Actually a general-audience talk; just on the development track for scheduling purposes

Format: 41-min talk; Q&A: BigBlueButton conference room
Status: Q&A to be extracted from the room recordings

Talk

Duration: 40:03 minutes

Q&A

Listen to just the audio:
Duration: 28:15 minutes

Description

This talk explores hyperdrive.el, an Emacs interface to hyperdrive, a peer-to-peer shared filesystem.

Peer-to-peer networks give you the freedom to choose your sources of information and to communicate directly with them. Emacs is a freedom-respecting text editor/operating system that puts you in the driver's seat. By bringing these two worlds together, hyperdrive.el aims to inspire deliberation about digital communication freedom.

Hyperdrives are peer-to-peer shared folders, well-suited for data that changes over time, like personal blogs. It complements Bittorrent and IPFS, which work best with unchanging data, like journal archives.

Here's a scenario: Alice creates a new hyperdrive and adds some files. Her computer returns a public key URL that uniquely identifies the hyperdrive. Alice shares that URL with Bob, who can then download Alice's files directly from her computer. No third-party servers are required to route the connection.

Data is distributed among peers; once Bob has loaded Alice's files, Carol can get them from Bob (or anyone else who has a copy) even when Alice is offline. Drives are mutable; when Alice adds/removes/changes files in the drive, Bob can refresh her drive on his machine to get the latest changes. Drives are versioned; anyone with the URL can "check out" prior versions of Alice's drive to see what her files used to look like.

Currently supported features in hyperdrive.el include:

Planned features include:

  • peer discovery (swarming)
  • diffing directories between versions

Check out the manual for installation instructions!

You're welcome to join our public XMPP chat room!

Bugs can be submitted to the ushin issue tracker. Patches, comments or questions can be submitted to the ushin public inbox.

About the speaker:

I'm Joseph Turner. I enjoy fiddle, Aikido, peer-to-peer networks, Emacs, and swimming in cold water. I work with USHIN, a tiny educational US nonprofit whose mission is to promote personal, community, and global health through free and open universal shared information for everybody. This year, we're focusing on the hyperdrive.el project, with the goal of bringing Emacs and peer-to-peer together.

I am Protesilaos. Friends call me "Prot" and you are welcome to do the same. I have been an Emacs user for ~4 years. I use Emacs full-time for practically every aspect of my computing. I am the author and maintainer of several packages for Emacs and am enthusiastic about its potential for user freedom.

Discussion

Questions and answers

  • Q: It's not clear how hyperdrive (not hyperdrive.el) works. Do I need to install something on my computer to use it? Can I use it from my phone?
  • Q: What lessons have you learned while developing hyperdrive.el?
    • A:  Great support and guidance from talented folks like Adam, Mauve, Jonas, Prot!!
  • Q: I use multiple computers and my partner also would like acess to my notes. so 2 questions. How well would this work with using this to editing my zettelkasten hyperdrive using multiple computers
    • A: Hyperdrives are single-writer, so you'd be better off linking between drives.  In the future, we plan to add support for https://github.com/nobiot/org-transclusion to hyperdrive.el.
    • Q2: How well would it work if my and my partner worked on the same hyperdrive zettelkasten
      • A: If you linked between drives, it could work quite well!
  • Q: What would be a good way of getting Hyperdrives if you don't want to install NPM and hava a binary. Could you compile it with deno or the "rust or zig or go?" cli alternative tool? I would prefer to download a single binary.
  • Q: If you had your druthers, what would make your work on hyperdrive.el easier?
    • A: User feedback!! Please try it out :)
  • Q:  Have you tried putting a git repo in hyperdrive? Does it work well?
    • A: If you use a bare repository, take care to gc and create pack files before mirroring to the drive, and only publish periodic updates, then that might be okay. (And make sure not to repack old pack files.)
  • Q: Is data transferred between nodes in the clear or encrypted?
    • A: Encrypted in transit.
  • Q: Is there a searchable catalogue of hyperdrives? 
  • Q: Any plans for FUSE or posix semantics?
  • Q: Any plans for a TRAMP interface?
    • A: Good idea!  A TRAMP interface may make it easier to offer live hyperdrive filename completions.
  • Q: How does this comprare to syncthing?
    • A: Syncthing is useful for sharing files among a small group of trusted peers, like an F2F network (<https://en.wikipedia.org/wiki/Friend-to-friend>).  Hyperdrives are useful for publicly sharing a set of files which you can updated going forward and which others can link to.
  • Q: If you edit a file on the hyperdrive, then edit the same file on the local mirror. How is the conflict handled when you sync the mirror again?
    • A: If I understand correctly, you're asking about what happens when you write to the same hyperdrive from multiple machines. The short answer is, "Please don't do that." However, it appears that the Holepunch team is making progress on https://docs.holepunch.to/building-blocks/autobase for "autohmatically rebasing" hyperdrive history, effectively allowing for multi-writer hyperdrives.
  • Q: wouldn't user be able to collaborate asynchronously by viewing diffs on a serially "shared" file, in other words, a user would copy another peer's text file, edit and upload their changes, share the link to their updated file so that then others can see diffs and in that way co-create an evolving file?

Notes

  • damn, I never even knew about dired-jump (C-x C-j).  Main differences with (C-x d) from the file are one fewer keystroke and having the point on the file you came from.
  • Btw, hyperdrive looks like another one of those things that would be amazing if I collaborated with anyone using emacs
  • Also incidentally hyperdrive is amazing
  • I installed hyperdrive.el and tried to run it and immediately got: transient-setup: Symbol’s function definition is void: transient-prefix-object
    • Need to upgrade transient.el
      • Gah, you are right. I had an obsolete version just hanging out causing pain.
      • Okay, I got hyperdrive.el working and it was super easy. hyper://fwsn55wnznts5mpkee16j89ja38nfz6zne4wijzap1z9ka4jsxio/Dird_a333f1_4884540.jpg
  • I think hyperdrive is a pretty easy way to share files. Easier than Dropbox. But maybe for delivery? I'm not sure.

  • A: What do you mean by "delivery"?

    - I was thinking of asset delivery. I write professionally, so I have things like PDFs and other documents to deliver. Sometimes I have big files to deliver to clients. Using something like hyperdrive would be cooler than Dropbox. ... Hmmmm, is there a hyperdrive web gateway?
  - A: A good non-Emacs option is <https://github.com/AgregoreWeb/agregore-browser> Agregore is Chromium with built-in support for more protocols besides HTTP, like Hyperdrive, Bittorrent, IPFS, Gemini, and Gun. <https://github.com/RangerMauve/hyper-gateway> , the program that hyperdrive.el connects to as a client is, I think, exactly what you asking about. I don't know of a public HTTP gateway, but you could easily run hyper-gateway on a VPS.
  - That would be fun. Then I could control the domain name so it would look cool and professional and like I was extra high-tech.
        - A: hyper-gateway lets you serve multiple drives at various subdomains. For example, you could serve your primary drive at <https://reverik.tld>, and then folks could access, e.g., the USHIN drive at <https://aaj45d88g4eenu76rpmwzjiabsof1w8u6fufq6oogyhjk1ubygxy.reverik.tld>

  • Great work, everyone. Thank you.

  • Prot's presentations are so clear. Perfectly model pedagogy.
  • Just like his code: his .el buffers SQUEAK as you C-v through them.
  • Prot is indeed a model of paying attention to detail.
  • I have learned so much from his videos, and from his code.
  • Prot switching from vim/etc to Emacs seems to correspond to an inflexion point in the current Emacs renaissance does it not? Coincidence, or... I wonder what way the causal arrow goes :)
    • I think the current "Emacs renaissance" probably dates back to around the time that package.el was developed and then when MELPA came online. momentum seems to have been building since then
      • We must also consider Org-mode's inception as a key moment in recent Emacs history
  • My Emacs environment would be much poorer without Prot's contributions, that's for sure.
  • My hypothesis is that Diogenes himself made his way to Cyprus and spent the past immortal couple thousand years as a mountain hermit waiting to return to mortal life when the Emacs community needed him. Humor aside, there are a few key personalities---many of them here this weekend---who have been disproportionately catalytic to the community recently. Prot is certainly one. Thanks to all of you!

Transcript

Hello, I'm Joseph Turner. This talk is about hyperdrive.el, peer-to-peer file system in Emacs. Hyperdrive is a JavaScript library developed by the Holepunch team for sharing files on a peer-to-peer network. It's released under the Apache 2.0 license.

[00:00:19.840] Hyperdrives introduction

Here's how it works. To participate as a peer, you run a node, a lightweight local server that allows you to connect with other nodes on the network. You can create a hyperdrive or multiple hyperdrives, and you can author files within them. Each hyperdrive is automatically assigned a globally unique link that starts with hyper://. When you share that link with someone, they have access to your hyperdrive. Anyone who has that link can load the hyperdrive from the network and view its content. When you load a Hyperdrive file from the network, your node caches that data locally and automatically begins seeding it back to the network, making it available for others to download from you. Hyperdrives are single writer. This means that when you create a new drive, you are the only one who can make changes to it. Others can view it and can seed it back to the network, but you're the only one who can modify it. Hyperdrives are offline first. This means that when you load data from the network, it's stored locally on your machine for you to view later, even when you're disconnected from other peers. You can also create new drives and modify your drives when you're offline, and then share those changes once you connect with peers later. Hyperdrives are local first. This means that when you are connected with other peers on a local area network, even if none of the peers involved are connected to the broader Internet, you can still share files. Hyperdrives are sparsely replicated. This means that you can download individual files from a hyperdrive without having to download the whole thing. This saves on disk space and also allows you to quickly load just the files that you're interested in. Hyperdrives are mutable. You can add files, change files, remove files from a hyperdrive. And when peers load your drive using the very same link, they will be able to load the latest changes that you've published. Hyperdrives are versioned. This means that when you make changes to a file, the previous versions of those files are not lost. Peers can load the old versions of a file that was changed or deleted, for example, simply by specifying the version number of the hyperdrive when the file still existed or existed in a previous state.

[00:03:04.600] About USHIN and the contributors

I'm presenting this talk on behalf of USHIN. USHIN is a tiny nonprofit whose mission is to promote personal, community and global health through free and open universal shared information for everybody. USHIN was founded in the early 90s by Paula Maas, Steve Nash and others with the goal of creating a distributed health information network that would allow people to find, share, compare and deliberate health information from a variety of sources. Since then, the scope of the project has broadened to include all kinds of information. And in recent years, we've been focusing on building with peer-to-peer software. About three years ago, we started working with Mauve Signweaver, who has since then been our steadfast peer-to-peer explorer and guide. This year, we started working on this hyperdrive.el Emacs package, and Adam has been the powerhouse behind the Emacs Lisp development. About a month ago, Jonas Bernoulli started joining with us, and he has been offering his expertise in the realm of user interface design using his Transient library. And Protesilaos Stavrou has been not only valuable in terms of user design and feedback, but he created a wonderful basic introduction to Hyperdrive. Take a look.

[00:04:32.646] Basic introduction to Hyperdrive

My name is Protesilaos, also known as Prot. In this video, I want to show you the basics of hyperdrive.el. This is a package for Emacs that lets us connect to the Hyperdrive peer-to-peer network. We can browse existing drives on the network, meaning existing file systems, or create and maintain our own hyperdrive, to which we can add, remove or edit files. We will see this together. What I have here in front of me is a basic use-package declaration for hyperdrive.el. All I am doing is binding hyperdrive-menu to a key and also activating the menu bar entry of hyperdrive. Let me show you very quickly, hyperdrive-menu. And this is the sort of interface that it brings up. We will take a closer look at it. As for the menu bar, we have hyperdrive-menu over here, and we will take a look at this as well. Suffice to say that you can get the same functionality with the menu bar or with hyperdrive-menu, as well as by calling the commands directly with M-x or their respective key bindings. I won't cover everything in that regard, but please bear this fact in mind.

[00:06:06.133] Managing files with Hyperdrive.el

Let's start then with what I have here in this other tab, which is a set of files. I have prepared in my local file system - a hyperdrive. This contains a set of files that I ultimately want to share on the peer-to-peer network, meaning that I want this to eventually be reflected in my own hyperdrive. What I did to get started is I invoked hyperdrive-menu. The very first thing you need to do to get started with Hyperdrive, either to browse or to create, is to start the gateway, meaning to be able to connect to the peer-to-peer network. You see here in hyperdrive-menu that there is an indicator next to the "Gateway" heading telling me that the gateway is on. I started it with G and then s. Once you start it, you can type h to visit an existing drive, N to create a drive, or L to open a link to an existing hyperdrive. You may get this link via email, for example. In this case, let me be over here and let me just kill this buffer. What I want in this case is to select a hyperdrive. You see here I have my own hyperdrive and also the hyperdrive of USHIN, the developers of hyperdrive.el. What I want to do in this case, you see there are various options available. I want to type v to view a file. And in this case, I will just type the forward slash, which means to view the root directory of this hyperdrive. And I am here. It is an empty drive apart from my credentials over here. There is nothing more really to see. But what I want to do is to start adding files to this drive. Let me use the menu bar for this. I will hover over to the menu bar, and I will find the "Hyperdrive" submenu. Then I will go find where the drives are. And you see that there are writable drives, meaning drives that I manage, and read-only drives, meaning drives of other users. In this case, USHIN is read-only, of course, and Protesilaos is writable. So what I want to do is upload a file to Protesilaos. And it prompts me in the Minibuffer for a file. I will just add the README. It is asking me, "Where do you want to add it?" And in square brackets, the default is to add it to the root directory of the hyperdrive. So I will just type RET to select the default. And there it is. It is already in the drive. Let me add the COPYING file as well, using the same method. I will go find my drive. "Upload File." And let's add the COPYING. And I also want to add it to the root directory. So already I have two files. This is looking good. Now I am opening the file inside of hyperdrive. This is not the same file as I have in my local directory. What I can do here, for example, is I can... Let me open another file here, the COPYING. I can create an Org link. Notice that these are Org files. So I will use the standard org-store-link command. You can see the key binding I invoked and the command it calls on the top right corner of my screen. So what I did is I copied a link to this hyperdrive file. And in this other file over here, I want to say "Free/libre." So I am editing my hyperdrive terms: "Check the," and I will paste the link here, "for how to use my files." If we check what is here, you will see that this is a link inside of the hyperdrive. And it's pointing specifically to that heading over there, which has this unique identifier, as you can see over here. So this is wonderful. I want to rename it to "Check the COPYING for how to use my files." I will save this. And now what I want to do is, I want to invoke hyperdrive-menu. And you will see now the menu, unlike earlier, has more commands, more stuff we can do with it. For example, I can type w to copy a URL. And by typing w, you will notice... Let me go to the scratch buffer to paste this in. You will notice what the URL is. It is my hyperdrive, my unique identifier, and then forward slash, meaning the root directory, and then README.org. That is the file I was editing. Let's do hyperdrive-menu again. Let's remove the scratch buffer. And let's do hyperdrive-menu. And you will notice that there are options to download the file, for example. If you are reading somebody else's file, you can download it to your own file system. Let me do that. "Download." And it is asking me, "Where do you want to save this?" For now, I will save it in the /tmp/, like this. Let's call it test.org. Okay, /tmp/test.org. Let me go and visit test.org. And there it is. It downloaded it just like that. This is how you can, for example, download the pictures and videos that I will eventually share on my hyperdrive. Let's invoke hyperdrive-menu again. And let's go up to the parent, you see, with the caret (^) sign. This will take me to the parent directory, in this case, the root directory of my hyperdrive.

[00:12:56.573] Dired like interface

Let me do it a bit differently. The same idea, a bit differently. For those of you who are familiar with Dired and the dired-jump command, Dired is the standard file manager of Emacs. And dired-jump is a command that lets you jump from the current file to the directory that contains that file. So you see, I am here. The dired-jump command, by default, is bound to Ctrl-x, Ctrl-j (C-x C-j). So if I do C-x C-j, in this case, it invokes a command. You can see the name of it, hyperdrive-up, which is functionally equivalent to dired-jump. It does the same thing, meaning that it took me to the parent directory of this file. I think this is very helpful. This listing over here, in general, tries to mimic or to reuse the knowledge you already have of Dired. For example, if you type o, it will open the file at point in the other window, same as in Dired. Whereas if you type RET, it would open it in the current window. Again, same as what you will do in Dired. Let's see over here. You have options to jump with j, which is using Minibuffer completion to go to a file. Right now, I only have two files, but the idea is the same. It's, again, what you would do in Dired if you type j with the default key bindings, though, not with Evil mode or something else. Let's see again what we have over here. You can create a bookmark, and this will work, but no need to show you everything. The idea is that you create a bookmark the way you create any Emacs bookmark, to a file, to a directory. It doesn't matter. And then you can jump to it, the way bookmarks in Emacs always work.

[00:15:01.234] History in hyperdrive

What I want to show you now a little bit is the history. History in Hyperdrive has to do with the drive itself. Meaning that individual files do not have their own history, but the drive as such has a history. Whenever you add a file, you remove a file, or you edit a file, you are incrementing the versioning of the hyperdrive by one. So each action corresponds to one unit of history. If you add a file, remove a file, and edit a file, this means that you are up three versions. So whatever your version number is, plus three. I am on version 24 over here. [ latest, not version 24] Let me go to this file now, and let me do V h to see a history of it. You will notice that between versions 23 and 24, this file was constant. But in version 25, we have a change. When you are in this buffer over here, you can type the equals sign (=), which is a key binding that will bring up the diff. So the set of changes between the previous version and the current version. And you see here, between versions 23 and 25, I have this addition. I think this is wonderful because now you can always go and check what is the state of this file. What is this person up to? What have they been changing? And with Hyperdrive, you can also visit… Let me go to the parent here. You can always visit a previous history. So, for example, I will go to the previous history, and let me see this file again in its previous history. I am looking at the file, but notice that the file now is not editable because this is in the past. I cannot rewrite history. I can only go to the present and then modify it and then create a new history, a new version. But this helps me see the state of the file at that version of the hyperdrive. So this is the basic idea of it, folks.

[00:17:39.973] Use case of sharing large files

What I want to do then is continue with my process here. Let me actually do it like this so that you can see. Continue with the process of publishing all those files that I have on my hyperdrive. For example, this is a picture of a flower that I have taken. I think it's very nice. And this is a video of an eagle that was flying above me. And I will share this on the Hyperdrive network. From the network, by the way, you can also stream video as well. It is described in the hyperdrive.el manual, but I cannot show you everything right now. I think you get the idea. The gist is, you have a file system that you can share with the world using peer-to-peer technology. And for me, this is a powerful tool. This is a valuable proposition because I can share these large files I have, these pictures or videos, which I cannot post on my website due to bandwidth considerations. And this way, I can still share with the world something that I consider interesting. That's all for today, folks. Thank you very much for your attention. Remember that hyperdrive.el is still in development and things may change. But the fundamentals are in place and will remain constant. That's all for today. Take care. Goodbye.

[00:19:20.913] Drive creation with hyperdrive.el

Another fundamental feature of hyperdrive.el is drive creation. The first step, as always, is to make sure that the gateway is running. So I'll open up hyperdrive-menu with C-c h. By the way, my key presses, as well as the commands that they run, can be seen at the top right of my screen. Down here, I see that the gateway is off. So I'll start it with G s. Now, it takes a few moments for the gateway to spin up. So to refresh the menu, I will close it and open it again. And now we see that the gateway is on. I'll press N to create a new drive. Now it's prompting me for a new hyperdrive seed. A seed is a string of characters that will be used to generate, in tandem with my secret master key, a new public key that will globally, uniquely identify this drive. So I'll type in "emacsconf". And after a moment, we see the newly created, empty drive. So I'll open up hyperdrive-menu once more. And I'll press h to open the sub-menu that shows more information about this hyperdrive, as well as commands related to this drive. In blue here, we see the seed that I just entered as well as the public key that it generated. We also see that the petname is also set to emacsconf. The petname is different from the seed. The petname is my personal, local identifier for this drive. I can change it whenever I want. And while it's not a secret, it's not displayed to other users. So I will leave it for now. "emacsconf" is fine. But when I go to share this drive, I'll want to display something more memorable than this long public key. And that's what the nickname is for. I'll change that by pressing n, and I'll type in "Emacs Conference". Now, when other peers load this hyperdrive by its URL, which I can copy by pressing w, they will see the nickname in addition to the public key. So if hyperdrive.el is like a phone book, the public keys are akin to phone numbers, the nickname is like the name that your contacts give you when they introduce themselves, and the petname is the name that you actually write down in your phone book. I'll show you what nicknames and petnames look like for drives that are not writable to me. I'll press C-g to close this submenu, and now I'm back at the main menu. I'll press C-u h to choose a hyperdrive, and I'll look at Prot's hyperdrive here. And here we see that the nickname of Prot's hyperdrive is "Protesilaos". Now, it's grayed out, which means that I can't change it. I can't change it because it's not my hyperdrive. But I can change the petname if I want it to show up under a different name. So I'll press p, and I'll type in "Prot", and hit Enter. Now I'll open his hyperdrive by pressing f. And I'll pick a path, I'll just hit RET to open the root directory. And now, when Prot's hyperdrive shows up, at the top of the screen, I see that it's identified with the petname "Prot."

[00:23:26.199] hyperdrive-mirror

Now I'll show off hyperdrive-mirror. hyperdrive-mirror is like hyperdrive-upload-file, except that it allows you to upload an entire directory full of files recursively. For this example, I will upload the contents of the emacsconf-mirror directory. First step is to open the menu. I'll press H, and then I will choose the emacsconf drive. In the bottom here, we see the Mirror group. The first option that I can change is the source directory. The source directory is the directory on my local machine from which files will be uploaded. By default, the source directory is set to the current directory of the main buffer. This is good for now, so I'll leave it. The target directory is the directory in the hyperdrive where the files will end up. By default, it's the root directory, but for this example, I'll put these files in a subdirectory called notes. The filter is the rule that allows you to programmatically determine which files in the local directory will be uploaded into the hyperdrive, and which ones won't. By default, all files are mirrored, but in this case, let's say that I want to upload only the Org files, these first three, and I want to exclude the markdown file, solar-oven-notes.md. So I'll press m f, and I will choose the regular expression string option. If I wanted to, I could choose a named function or a lambda, but I won't demo that here. I'll type in org$, which will match against the files that end with "org." The filter is here, and I'll leave confirmation on. The confirmation step just allows me to review the list of files that are going to be uploaded into the drive before it happens. So I'll press m m to mirror them, and I see here that these three files are going to be uploaded. Looks good. I'll press C-c C-c to confirm the mirror. Now it says three files have been uploaded, and here they are in the drive. Good, so now I will modify the fermented-overnight-oats file in the hyperdrive. So I've loaded it, and I'll add here: "or other grains - cook them in advance if you want to," and I'll save it. Now this file, fermented-overnight-oats.org, has been modified on the hyperdrive more recently than on the file system. The file system file has not been modified. So I'll go back to my local directory, and I'll modify a different file. In this case, I'll add another hoedown to the fiddle-tunes.org file. "Tom and Jerry." That's a good hoedown. And now I'll mirror again. So I'll open the menu, and I'll press h, and then I'll open up the emacsconf demo drive. And now I've also decided that I want to include the solar-oven-notes.md file. So I'll remove the filter so that it's no longer excluded. I'll leave the rest of the settings the same, and I'll press m m again. And now we see that the hyperdrive-mirror buffer looks different. So, there are two main groups. These are the files that are going to be uploaded, and these are the files that are ignored. They're not going to be uploaded. The first subgroup is the files that are new locally. So the solar-oven-notes.md file is new on my machine, and it doesn't exist in the hyperdrive. So the mirror command is going to take that file and add it to the hyperdrive. This group contains the files that are newer locally. So the fiddle-tunes.org file has been modified on my local machine, but it hasn't been modified on the hyperdrive. So hyperdrive-mirror is going to take the updated version and put it on the hyperdrive. Now, the first group that's going to be ignored are the files that are older locally. So the fermented oats file has been modified on the hyperdrive more recently than on my local file system. So hyperdrive-mirror isn't going to overwrite the version of the file in my hyperdrive with the older local version. And finally, the emacsconf-preparation.org file hasn't been modified on either the hyperdrive or my local file system, and the timestamp is identical. So hyperdrive-mirror is going to ignore this file as well. So the only two files that are going to be uploaded now are the solar-oven-notes.md file and the fiddle-tunes.org file. I'll confirm that with C-c C-c. And now in my hyperdrive we see that there are four files. The solar-oven-notes.md file has been uploaded, and if I open the fiddle-tunes.org file, we see that it now contains the line "Tom and Jerry," which means that it was updated based on the change to the file on my local file system. This hyperdrive-mirror command is the command that we use to periodically update the USHIN hyperdrive with the contents of the USHIN website. There's going to be a link to the USHIN hyperdrive as well as the website at the end of the video.

[00:30:06.819] hyperdrive history

Now I'll go into a little more detail about the Hyperdrive history buffer by showing off the README file in Prot's hyperdrive. I'll press C-c h to open the menu, C-u h to be prompted for a drive. I'll select Prot's drive, then I'll press f to jump to a file inside of Prot's drive, and then RET to go to the root directory. From here, I'll press j to jump to an item in his directory. I'll press RET on the README to load it. And then finally, I'll open up C-c h to look at the menu. Here, I see that I'm looking at the latest version of Prot's README.org file. I also see that the previous version of README.org is unknown. It's unknown because hyperdrives are sparsely replicated. That means that when my node loaded this README.org file, it didn't bother to load anything else. It didn't load the previous history or any other file in his drive. But now that I want to check out whether there is a previous version, I'll press V p. And when it loads, I see in the mode line down here that I'm now looking at version 25 of this drive. That means that I'm looking at the README.org file at version 25 of this drive. I'll open the menu again, and I see that the same version number is displayed here. The previous version, before version 25, is also unknown because we haven't bothered to load anything before version 25. I also see that the next version, after version 25, is the latest version. So I'll open that up by pressing V n. And now we are back where we started, at the latest version. But now we see that the previous version, the version before the latest version, is version 25. Because now that our node has loaded the previous version, it can display that information to us. From here, I'll open up the history buffer. I'll press V h. And I can see that there are two known existent ranges in Prot's README.org history. This means that the latest version of README.org was modified at version 39. And that Prot made four changes to other files in his hyperdrive besides the README.org file since then. Before that, the time that Prot modified the README.org file was at version 25. And then he made 13 other changes to other files inside of this drive. Before that, we don't know. We haven't loaded the history. But since we're curious, I'll hit RET on the unknown line. And now we see that in Prot's hyperdrive, the README.org file didn't exist for the first 22 revisions of his drive. Then Prot created it at version 23. Then again, at 25, made a change. And then made another change at 39. For good measure, I'll show you the diffs. The first diff just contains the entire file because the file didn't exist before version 23. Then, at version 25, Prot added a link to the COPYING.org file. And then, at 39, Prot changed the link to his own hyperdrive to be a relative link.

[00:34:20.880] Streaming video from hyperdrive

Now, I'll stream a video from the USHIN hyperdrive that shows off a prototype interface we created for exploring networks of sources of information. I'll open up hyperdrive-menu. Press C-u h to be prompted for a drive. I'll select the USHIN drive. Press f to jump to a file in it. And then I'll jump to the media directory. Once it loads, I'll press RET on the video that I want to stream. of the subjective trust interface that the USHIN team built for the u4u.io web app. Streaming a video from Hyperdrive.

[00:35:08.746] hyperdrive.el under the hood

Here's how hyperdrive.el works under the hood. It uses plz [Please], the HTTP library that Adam Porter wrote, to send requests via curl to hyper-gateway. hyper-gateway is a program that Mauve Signweaver wrote, which runs a hyperdrive node under the hood. It also runs a local HTTP server, which accepts requests to control the node. For example, if hyperdrive.el wants to show a file from someone's hyperdrive, it sends the appropriate link via curl as a GET request to hyper-gateway. hyper-gateway then fetches the data from the network and returns it via curl, via plz, back to hyperdrive.el. If hyper-gateway already has a locally cached copy, it doesn't bother checking the network. It just sends it straight away. Likewise, if hyperdrive.el wants to add a file to a hyperdrive, it sends a PUT request. hyper-gateway is not installed as part of hyperdrive.el. It needs to be installed as a separate, executable program. We have plans to switch from using hyper-gateway to another program that Mauve Signweaver is working on, called hyper-sdk-rpc. hyper-sdk-rpc will give us more fine-grained control over the underlying Hyperdrive node, which will open up some new features. For example, we'll be able to rename files atomically, diff directories between versions, and list the peers that we're currently connected to. Here are some of the Emacs libraries that hyperdrive.el depends on. Hyperdrive.el uses plz to send HTTP requests to hyper-gateway. Check it out. It's on GNU ELPA. ewoc.el is a built-in library that's documented in the Emacs Lisp manual under the rather cryptic heading "Abstract Display Functions." Really, what it does is it allows you to map a data model to some display. What we do with it in hyperdrive.el is, in the directory view, we map directory entry items, files and directories, to display items. And ewoc.el makes it easy to update the display whenever the underlying data changes. Hyperdrive.el uses persist.el to store Hyperdrive metadata about known hyperdrives as well as version history between sessions, so that the data is cached when you log out and log back in. If you're going to be developing a program in Emacs for Emacs 29 or later, I recommend looking at multisession.el, because it's built-in, and it has some features that persist.el doesn't have.

[00:38:57.541] Next steps

Thank you for listening to this talk. I hope you get a chance to try out hyperdrive.el. Here is a link to the hyperdrive.el manual in the USHIN hyperdrive. When you go to paste this link in, you'll need to combine it back into one line. This link is available in the hyperdrive.el manual, which is available from within a browser, at this link on the USHIN website. We also have a public conference room that you can join. It's an XMPP multi-user chat that the Sopranica team graciously makes available for us to use. There's also a Matrix bridge, if that's more your cup of tea. Thank you to the EmacsConf organizers. It's been a pleasure to participate, and I look forward to hearing the rest of the talks. Have a good day.

Captioner: bhavin192

Q&A transcript (unedited)

So Joseph, thanks for being here. Thanks for talking to the hyperdrive. We already had some, or we already have a lot of questions here. And I guess I would start with, let's call it the difficult, the most difficult 1. So when you were developing hyperdrive for your colleague, what do you, or what have you learned the most? enjoyable the development of this project can be with talented people working by my side, like Jonas and Adam and Prat and Mo, it's been really a pleasure to work with these folks. then probably pushed it somewhere in open source or how did it develop, your development experience? using peer-to-peer technology for sharing all kinds of information. And we came across Move SignWeaver, who was recommended to us by a mutual friend. And we started working with Move, and then about a year ago, we started looking into using Emacs, the peer-to-peer software, so that we could make use of all of the powerful things that Emacs already does with org mode and other packages. And then we started working with Adam and Pratt and Jonas. So to read it out, I use multiple computers and my partner also would like access to my notes. So, 2 questions at first. First 1, how well would this work with using this to edit my Zettelkasten hyperdrive using multiple computers? So what that means is that if you have a hyperdrive that you've created, you're the only 1 who can make changes to it. And that's limited right now to editing 1 hyperdrive from 1 machine. In theory, you could use the same private key and write to it from multiple machines, but you would have to make sure that you sync it on both machines and didn't make concurrent writes because then you would fork the history of your hyperdrive, and that would be bad. But we've spent a lot of time making links to hyperdrives work well, relative links within hyperdrives to other files inside of your drive. So you should be able to, with some exceptions, just take your personal information management set of org files or whatever it is that you have, and upload them into a hyperdrive if all of that is publicly available or would be good to share publicly. And you can make that available for other people to link to. So you can have multiple different hyperdrives that link to 1 another. connected to each other in some way. There was a follow-up question or the second part of the question. Okay, then using the same hyperdrive is probably not possible, but interlinking would be the best way to do it. There was a question concerning how they should install it. So What would be a good way of getting hyperdrives if you do not want to install npm and have a binary? Could you compile it with denner or rusk or zig or go? CLI alternative tool, I would prefer to download a single binary. around with using Geeks to install Hyper Gateway. So the way that HyperDrive.el, the Emacs package, works right now is similar to the way that the transmission Emacs client for BitTorrent works, where you have a client in Emacs that connects to a daemon that is a separate process that's running on your machine, the transmission daemon. But in this case, we have HyperGateway, which is running as a daemon on your machine. And then hyperdrive.el connects to that daemon and sends requests, and all of the hyperdrive stuff under the hood happens with her gateway. But so that package can, or hypergateway, the program can be installed, The easiest way is to just download it from the GitHub releases. You could also use NPM to install it. And then the third option that we've been playing around with is Jonas was writing a little script to install it using Geeks, since Geeks now comes with Node 18. And so you should be able to install it using Geeks. joined with microphone. Do we have now any question to Joseph or just here for chilling out. I guess it's a no. Plasma, yeah. information being private in the hyperdrives. this point. At this point, what we've been working on is mainly using hyperdrives for a public forum type tool. But you could encrypt those files if you wanted to. You can also just, a poor man's security would just be to share your HyperDrive link only with those people that you want to have access to your drive. But the way that it works right now is anyone who has the link to a hyperdrive can access its content. So long as there are peers available on the network who can serve it to you. Plasma? interrupt me. before. What about, if I remember correctly, it doesn't do as well with large files, so if you're going to store 200 gigs of video files, stuff like IPFS works a lot better, or BitTorrent. This is, are you, were you using the, any way of using multiple protocols for stuff like that? Or what were you doing with, or were you just doing the small files with the same protocol? Or as well that could interface with Kubo or some other IPFS daemon and I think that those could work really well together. We mostly have been playing around with sharing relatively small files, up to hundreds of megabytes or maybe a gigabyte. We haven't played around yet with hyperdrive.el, the Emacs client, testing that with HyperGateway. But there may be other experiments that have been done that show that that works well. The main thing is that IPFS uses content addressability to reduce duplication of the content. Whereas in HyperDrive, if you upload the same file with the same contents twice, now you have double the content being stored in your HyperDrive. It's not deduplicated. You can always clear out part of the history of your hyperdrive But IPFS has really good built-in deduplication whereas hyperdrive does not you have a couple of different Hypercore blogs, what about like commenting between them? Like you have some people who have a commenting form on Reddit for their blog posts. work recently with the distributed press API to integrate ActivityPub with these peer-to-peer technologies. Move can give you more information about that. But there is another feature that we'd like to add to hyperdrive.el, which is peer discovery using the swarming feature that HyperCore, HyperSWARM offers, where you'd be able to say that my node, my peer-to-peer node is interested in Emacs and free software as topics. And those would be 2 different topics. I would advertise on the network that I'm interested in those topics. And I would be able to discover other peers on the network who have also advertised that they're interested in those same topics. And then they would tell me, hey, here's the public key of my hyperdrive. Come check it out. I have posted information about those topics. And so in that way, you'd be able to, in a distributed fashion, discover other peers on the network who are interested in topics that you're interested in. that idea is like if you had your emacs Zettelkasten Publish like let's say you have some private data You make sure that that's scrubbed out before it goes to your hyper core and then you have another part of it that gets turned into a website for it's also given to other hyper core clients but you'd rather get the emacs users the org documents then you also publish some of them on a website so everybody as much people can get it as possible. And then a way of figuring out who you'd want to do, or if you're an Emacs user, maybe figure out that they're all related to each other, but you want to get the art mode documents because you're using EMAX. Yeah. before we switch into the next track, just to let you know. that we added, would allow you to selectively choose which files you want to share in a hyperdrive. So, with Prot's denote file naming scheme or Carl Voigt's file tags naming scheme, you could just specify a regular expression. And you could say, I want to share out of my directory of org files, I want to share only those files that have been tagged as public, or only those files that have been tagged as emacs and then only those ones would get uploaded into your hyperdrive private So I just maybe I missed it and you haven't put a link. I have no idea what happened to my microphone, but now it's back. top of the screen. So hyperdrive. Is the hyperdrive a find on the hole punch point, point T O hole Or is it just another hyperdrive? So the HolePunch team has released hyperdrive and other hyper core libraries as free software libraries that you can use. And so MoV SignWeaver, the project that MoV is working on, HyperGateway, depends on those libraries and it makes it easy for you to build other clients like hyperdrive.el which connect to the hyperdrive network. I hope that answers your question. And what did make you choose hyperdrive for this Emacs project? which makes it distinct from IPFS or BitTorrent, where when you share some piece of content, you're stuck with that static piece of content, which works well for some cases, but if you say you have a Zettelkasten or you have a set of org files that you want to share with people, you want to be able to update those files and have other people pull those updates from you. And so HyperDrive allows you to have these mutable sets of files that you can share and use the same link for other peers to pull the latest changes from you. Also, it's versioned, as we showed in the video, which is really helpful for having community deliberations and community discussions where you want to be able to reference some something that somebody said in the past and not have it get deleted or changed or something. letting you know if you want to say something now. Too late. The BB room is still open, so you can still discuss. There's also a lot going on on the pad. But you can also discuss here inside and answer the pet questions maybe later. I'll go ahead, please. would it make sense in any sense to put a FUSE interface or put the POSIX semantics in front of this at some point? It's kind of a similar question to any plans for a Tramp interface. There was a project that the HyperCore HolePunch team was working on a year or more ago that provided a FUSE interface. And I think it didn't pan out. But it's a good idea. Same with the Tramp interface. It seems like a good idea that would make it possible to more easily hook into the built-in Emacs functionality for, for example, like incremental file name completion, which we don't currently support in Hyperdrive.el. So I'd love to have feedback and design ideas for those projects. sort of assumes the file system is there and usable in that way. That's all. Syncthing links. Because I think you can set up Syncthing in such a way that you have the private networks that other people can't actually get access to. Syncthing. I'll have to look into that. because yeah, there's ways you can explicitly authorize devices. Yeah, right. I think you could actually set it up in such a way that you can have private stuff and links, and then that might be a way that you can get a completely distributed Zettelcast and with private notes. pad about DATRS, a Rust version of HyperDrive. I had not heard of that, so I'll have to look into that. If you had your druthers, what would make your work on hyperdrive.dl easier? It's been a lot of fun. I would love to have more user feedback. That would be my wish. I tried putting a git repo in HyperDrive. Does it work well? I don't think that would work well because, as I mentioned a moment a few moments ago, the data that you put into a hyperdrive is duplicated. So if you had the whole work tree in hyperdrive every time you made a change and saved it, it would be duplicated. If you had just a bare repository, I don't know, try it. but 1 of the optimizations they have for being able to view a whole bunch of people's data is they made shallow clones a lot please? 1 of the things they do is they allow you to have a whole history of every single change for a dataset Zettelkasten project. But 1 of the optimizations Hypercore did to make it more network web friendly is they made the shallow clones work a lot better and a lot... Yeah, they made that work a lot better so you don't have to download every single thing for every single project. And because they both are implementing the delta upgrades, I don't see how they could work really well together. At least from what it looked like to me. It can't hurt to experiment. Is data transferred between nodes in the clear or encrypted? That's a good question. I don't know how it's encrypted. I don't, I wouldn't recommend sharing sensitive data with hyperdrive right now? I would recommend if you want to play with it, have it be something where you're expecting the data to be shared. Is there a searchable catalog? at rest. I'm pretty sure the data at rest would not be encrypted. Right. You can separate that into those 2 questions. of hyper drives? So that's a thing, an idea that we've been a distributed trust network for discovering peers that are trusted for a particular topic. And we actually made a demo video of a previous prototype that's available on the Ashen hyperdrive that you can watch that shows the basic idea. But the idea is just that you would have a list of peers that you think are worth listening to or worth reading for a particular topic. And those peers would have peers that they think are worth listening to for that same topic. And so you would say, if I'm interested in Emacs, I want to see all the peers that I trust for the topic Emacs. And if, say, Adam Porter shows up in my list and Adam Porter trusts Jonas and Jonas trusts Pratt, I would be able to read hyperdrive information from all of those people by looking at the indirect relationships that I have by following the chain of relationships, kind of like a web of trust. And so it would also allow you to have a network of peers that you trust to block other people on your behalf. So it would be useful for subjective moderation where you can remove spam and bad actors from the people that you follow without having to delegate that powerful responsibility to some third party in a permanent way where that third party might abuse that power. So it allows you to share your list of trusted peers and your list of blocked peers with other people in a peer-to-peer way. It kind of does some... It's trying to do something weird with the internet where it redesigns it from the ground up to be peer-to-peer, local first, or something like that. Yes. I have heard of it, but I haven't really researched it. If you edit a file on the hyperdrive, then edit the same file on the local mirror, how is the conflict handled when you sync the mirror again? So I think if I understand the question, the answer is that you can't edit the file in 2 different places, I think is the answer to the question. If you were to manually copy the private key from 1 machine onto another machine, then you could cause a conflict, like a merge conflict, but you would have to go out of your way to do that. And It's not handled. I think the Hypercore Hole Punch team has another project that they're working on that would, it's called AutoBase, that would merge those conflicts. But we're not using that right now. And I think it's in early development still. So there might be a solution in the future. what's the most interesting thing you weren't expecting to discover while developing this? Like change of thoughts on how you write or I don't know. and really to programming in general. And so it's been a fantastic learning experience. Adam, Alpha Papa, Adam and I have been doing a lot of pair programming sessions where we work together and I get to learn from him. And we've had meetings with Jonas and Prat and meetings with Mauve where it's a fantastic learning experience for me to discover how to build software in an efficient and intelligent way. It's a huge pleasure. If there are no more questions, I just wanted to encourage everyone to try it out and to let us know what you think. It would be really helpful to have some feedback from people who are using it in new and creative ways that we haven't anticipated. new thing called hyperdrive.el today, and I think it's pretty cool. Hey Joseph, how's it going? Oh, talk today. I'm going to say goodbye. Thank you. And thank you for your questions, Appreciate your questions, your thoughts. we have our, our first, I don't know if our first new user, but we have the first link being shared, to hyperdrive file in the chat and I loaded it and it works. And it's funny too. It's worth looking at. So. Oh, I think it's frozen. I don't know if anybody can hear me. It's it's not, okay. Just unfroze. Anyway. All right. Well, By the way, I enjoyed your talks about hyperbole. I'm going to rewatch those later when I get a chance. It was nice to meet you, too. Bob is a really great guy to work with. All right, you have a good day, first. Feels like it's a good dovetail with the hyper core yeah, I think there's a lot of a lot of interesting possibilities to build on this we have some plans that we Will get to you later this well in the coming year And we'll see where the hyperdrive people, you know, upstream how they develop it as well and yeah, so exciting times.

Questions or comments? Please e-mail joseph@ushin.org

Back to the talks Previous by track: emacs-gc-stats: Does garbage collection actually slow down Emacs? Next by track: Writing a language server in OCaml for Emacs, fun, and profit Track: Development