New in hyperdrive.el: quick install, peer graph, transclusion!
Joseph Turner - https://ushin.org xmpp:discuss@conference.ushin.org (XMPP MUC for USHIN discussion), contact@ushin.org
Format: 21-min talk ; Q&A: BigBlueButton conference room
Etherpad: https://pad.emacsconf.org/2024-hyperdrive
Status: TO_CAPTION_QA
Talk
Q&A
00:00.000 New version of hyperdrive.el 01:06.687 Q: Network effects are tricky - do you know of any public shares people can join to try this tool out properly? 07:31.064 Q: blocklist or whitelist so I can make them containing useful information for only me while also being useful with in a public sense 11:41.024 Q: Could you comment on the "visualization" thing, (org visualization), and your experience with this type of content in buffers and the various possibilities (svg, etc.)?
Description
hyperdrive.el is an Emacs interface to hyperdrive, a mutable, versioned, peer-to-peer shared filesystem. Among other things, hyperdrive.el has these features: Share unlimited files of unlimited size; Explore file history with built-in versioning; Stream video and audio; No signup or account creation; Free as in Freedom!
Since last year's EmacsConf talk, hyperdrive.el
grew some new features, including:
- Easily install the gateway program with
M-x hyperdrive-install
. - Visualize your network of sources as a graph or a list.
- Transclude snippets of hyperdrive files with hyperdrive-org-transclusion.
This talk will show off these new hyperdrive.el
features in action!
Feel free to join our public XMPP chat room!
- xmpp:discuss@conference.ushin.org (Join anonymously from your browser)
- #_bifrost_discuss_conference.ushin.org:aria-net.org (Matrix bridge)
Bugs can be submitted to the ushin issue tracker. Patches, comments or questions can be submitted to the ushin public inbox.
About the speaker:
I'm Joseph Turner. I enjoy fiddle, Aikido, peer-to-peer networks, Emacs, and swimming in cold water. I work with USHIN, a tiny educational US nonprofit whose mission is to promote personal, community, and global health through free and open universal shared information for everybody.
See also: EmacsConf - 2023 - talks - hyperdrive.el: Peer-to-peer filesystem in Emacs
Discussion
Questions and answers
- Q: Hi there, thank you for your talk - I enjoyed watching it! I
tried this tool last year, and it seemed to work well - but I don't
know anyone who actually uses it. Network effects are tricky - do
you know of any public shares people can join to try this tool out
properly? Thank you!
- A: Yes, network effects are indeed tricky. Hopefully, the peer graph can help with this. That said, here are some hyperdrives I know of:
- https://ushin.org - hyper://aaj45d88g4eenu76rpmwzjiabsof1w8u6fufq6oogyhjk1ubygxy/
- https://blog.mauve.moe - hyper://1m51x54k3fwbuec5z4edbathiq3aj6bew8b556ezrszizskogo3o/
- https://hypha.coop - hyper://zdouwaei9kc5zbk93w5dakbr8maayupojthf3eafo16s4d5pbsry/
- Q:One usecase for this is sharing and building upon second
brains/zettelkastens"denote or org-roam" but a blocker from me
wanting to make one public is wanting to use a blocklist or
whitelist so I can make them containing useful information for only
my while also being useful with in a public sense
- A: Rephrasing: how to keep the content of a Zettelkasten private, and only have parts of it be public?
- If the desire is only to share certain files in the
Zettelkasten, M-x hypedrive-mirror can do this
- Can specify either a regexp that matches some of the files that get uploaded to a directory of files on a machine (and only those files will be shared)
- It can also be a lambda, i.e. anything that can be formulated as a function
- E.g., with Karl Voit's filetags (or Prot's Denote filenaming scheme), you could share only those files which are tagged as "public".
- Q: idea: try hyperdrive to distribute WORG (and EmacsWiki)
- A: Great idea!
- Q: Could you comment on the "visualization" thing, (org
visualization), and your experience with this type of content in
buffers and the various possibilities (svg, etc.)?
- A: We submited a patch which was merged in Emacs 30 to make image maps (the "overlay" which makes images clickable and have hover-over help-echo descriptions) transform along with the image, e.g., if you zoom in on an image, the clickable map still matches the zoomed-in image.
- A: In hyperdrive-sbb-view.el, we build a Graphviz string based on the peer relations data, then call out to the external graphviz process twice, once for the svg and once for the cmapx. Then we build an image map based on the cmapx string. We put the image and image map together and render it in a buffer. I also adapted some code from image-mode.el to make the image resize whenever the buffer's window resizes. For details, see https://git.sr.ht/\~ushin/hyperdrive.el/tree/master/item/hyperdrive-sbb-view.el
- A: This approach is directly inspired by org-graph-view.el, by Adam Porter (github alphapapa).
Q:What is something surprising about how you or somebody else that you didn't expect from using the hyperdrive network?
- A: Don't quite understand the question. I think the surprising use case for it is to be able to share say a bunch of Org files, link between those Org files... You could use transclusion to to have a discussion and then you could collaborate on on projects using this sort of asynchronous communication using hyperdrive files. If there's a more specific question, I'm happy to answer it better.
Q: You mentioned streaming audio and video - could you confirm that this is in reference to consumption of media, and not broadcasting of it - right?
- A: Since the network is peer-to-peer, audio and video is streamed between peers.
- So this is all shared on a peer-to-peer network. So if I have a file on my machine that's a video or audio and I'm advertising on the network that people can download it from me, then if you want to download that video and I have a copy of it, when you click on that video in your hyperdrive, not only does it download it from me, but also you can stream it so you don't have to download the whole thing before watching it. So it is both in terms of serving the file and downloading the file.
Notes
The "sources, blockers, blocked" idea was influenced by:
- Alex Cobleigh : https://cblgh.org/trustnet
the safe hyperdrive feature was designed in part based on org-safe-remote-resources. I'd be curious to hear thoughts on how something like this could be improved or generalized.
- ooo nice graph
- Perhaps a function which accepts buffer-file-name and returns non-nil if it's safe to call set-auto-mode
- also, there was off list discussion related to CVE: untrusted-content
- untrusted-content is a variable
- we also discussed generalization of this idea with trusted/untrusted files to define them based on file location
- in a way it is similar to code blocks inside Org files
- not sure if all this is relevant to hyperdrive
- It definitely could be. It could be useful to enumerate the different potentially dangerous interactions we could have with files.
- (1) enabling a major mode (2) enabling file-local variables (3) including remote content (4) evaluating code blocks
- You might want a file to automatically set major mode but not evaluate code blocks, so more granular control would be useful.
- I think that allowing the user to set these safety settings based on buffer-file-name would benefit hyperdrive.el, eww, and any other remote protocol.
- nice!
- impressive!
- idea: try hyperdrive to distribute WORG
- better yet, those EmacsWiki "packages" that are getting rediscovered every decade or so, if they're not going to end up on a git host (as per its opinionated author's non-interest to do so if I'm not mistaken)
- putting EmacsWiki on hyperdrive would be useful too!
- YouTube comment: Awesome.
Transcript
Hello, welcome to this talk on hyperdrive.el. I'm Joseph Turner presenting on behalf of ushin, a tiny independent information freedom nonprofit. hyperdrive.el is an Emacs interface to the hyperdrive peer-to-peer file sharing system built by Holepunch. Like other peer-to-peer file sharing tools, such as BitTorrent, hyperdrive lets you share unlimited files directly with other users without having to go through a central hub. One thing that sets Hyperdrive apart is that the files you share can be modified. Since hyperdrive has built-in versioning, you can explore the history of changes that you make to your files or others make to theirs. You can stream video and audio. There's no sign-up or account creation process required. It's all free software. This means that the community has the legal right, the tools, and the information necessary to participate in the development process, or even fork the project and go cimarron. Since last year's talk, we've added a few new features. It's now much easier to get started using hyperdrive.el, since we added the hyperdrive-install command, which downloads the gateway program and installs it on your machine. We added a peer graph feature, which lets you visualize your network of sources of information. We also published a new package, hyperdrive-org-transclusion, which lets you display live updating snippets of hyperdrive files in Org documents. Let's get started. Let's say you get an email from your friend Alice inviting you to check out hyperdrive.el. Let's click on the link here to look at the manual. Let's go down to the installation section. It says you need to have GNU Emacs version 28.1 or later. I'll assume you have it installed, but if not, you can click this link. You need to have curl installed, which is likely already the case, but if not, hyperdrive.el will let you know. Now from within Emacs you can install hyperdrive.el. Run M-x package-refresh-contents. Then, M-x package-install, type in hyperdrive, and hit return. Next, we'll install hyper-gateway-ushin. Let's run M-x hyperdrive-install. It's prompting me to download and install the gateway, which is 51 megabytes. I'll press y. I will skip most of the download process. The download is wrapping up. Now, it's prompting me to start the gateway. I can either run M-x hyperdrive-start, or I can press this button. And after a few moments, we'll see that the gateway will start. It's ready. You can also click this button, "View hyperdrive User Info Manual", to view the same manual that we were viewing in the browser. The info manual comes with hyperdrive.el and is available offline. Now that everything's installed and the gateway is running, we're back at Alice's email. Let's click on the link to her hyperdrive. Here's Alice's hyperdrive. Let's open her hello.org file. I'm being prompted to mark Alice's hyperdrive. Currently, it's unknown. I'll press ? to see more details. "safe", "unsafe", "unknown". I'll press e for "explain". In the info manual, it says that if a hyperdrive is marked as safe, that means files in that hyperdrive will automatically have their major mode enabled based on their file extension. In this case, if I mark Alice's hyperdrive as safe and I click on this hello.org file, Emacs will automatically enable Org mode in that file. Since I know Alice directly, I'll mark her hyperdrive as safe. I'll click on the file hello.org again, and now I'll press Shift-s for safe. Now I'm going to set Alice's petname. hyperdrive.el has different ways to name a hyperdrive. The public key is the drive's unique identifier. You can also assign yourself a public nickname, which you announce to the world. Then users on their own machines can assign each hyperdrive that they are aware of a private petname. I'll assign Alice a petname, which is the way that I will identify Alice just on my own machine. I'll type in "My Friend Alice". Then when I refresh her drive, we see that it says petname in the top left. Now I'll do what she asks, which is to right click on her name and open her peer graph. In the context menu, I see a few different actions I can take, but I'll click on "Peer Graph". It's prompting me to pick the max hops for sources. Default 3. I'll choose the default. Same for blockers. I'll explain what that means later. This is Alice's peer graph. Here we can see that Alice has two direct sources, Bob and Eve. Both Bob and Eve have assigned themselves nicknames, and so we see Bob and Eve here. Eve has assigned this other hyperdrive to be a source. But this other hyperdrive, whose public key is "kb3zr6mq" and so on--if we hover over the hyperdrive, we'll see the full public key--has not assigned itself a nickname. So we just see the truncated short public key. Because we've assigned a petname for Alice, we see Alice's petname show up here. Now I'll open the peer list view to show the same information in a different way. Since I'm using a large font, I'll widen this window so we can see everything going on in the peer list. We see that the root hyperdrive is petname "My Friend Alice", nickname "alice", public key "dm1", and so on. We've set the sources max hops to three, which means that we will go out at most three hops from Alice in order to find sources. One hop from Alice are Bob and Eve. One hop from Eve is "kb3", which is two hops from Alice. Now we've set the max hops for sources to three, but in this case, the network is small, and "kb3" has no sources. So we can only go out two hops from Alice until we run out of sources. In the future, we plan to add a search feature based on this list of sources. You type in a query, and you get back a list of results published by Eve, Bob, or "kb3". It's also possible to block peers. I'll open the transient menu by pressing question mark. For now, let's turn on showing blocked peers. I'll press "s x", and now it says blocked sources. Now we can see Mallory also shows up. Mallory has been included as a source by Eve, but is blocked. So in the end result, she does not show up in the list of sources. Instead, she shows up as a blocked source. But how is it that Mallory ended up being blocked? Let's open up the transient menu once more and click "s b" to show blockers. Now we can see that Bob has blocked Mallory. This means that Bob is not interested in seeing search results from Mallory. Since Alice has included Bob as a blocker, that means that Alice trusts Bob to block people on her behalf. So since Bob has blocked Mallory, Mallory does not show up in Alice's peergraph as a source. Well, now I'm curious to see what it is Mallory published that drove Bob to block her. I'll right-click on Mallory, and in the context menu, "Open Hyperdrive" and... Wingsuit Flying Grand Canyon Point of View Not Clickbait? I gotta check this out! [playing: Rick Astley's "Never Gonna Give You Up"] No! Disgusting! No wonder Bob blocked Mallory. Let's go back to Alice's peer graph. Mallory is a blocked source. Are there any peers who are blocked but who aren't sources? I'll open the transient menu and I'll press "s x". Now we're looking at blocked non-sources. Darth, who's been blocked by Bob, has not been added by any of Alice's sources as a source. So Darth is a blocked non-source. I'll open the transient menu again and I'll press "s x". Now we're looking at all blocked peers. Both Mallory and Darth show up. If I scroll down in the peer list, we'll see that Mallory is listed under blocked sources, and Darth is listed under blocked non-sources. This view has gotten a little bit busy. Let's say that we're only interested in how Alice relates to Mallory. I'll open the transient menu, and I will filter the graph to show paths only to Mallory. I'll press "o a", type in Mallory, and hit enter. Now we see that Mallory has a bold border and "kb3" and Darth are missing from the graph. Eve shows up because there's a path from Alice to Mallory that goes through Eve. Same for Bob. We can also add more than one peer in this view. I'll press "o a" again, and this time I'll choose Darth. Now Darth and Mallory have bold borders, and "kb3" is still missing. We can remove Mallory and Darth one at a time with "o r", or we can press C-u, the universal prefix argument, and then "o r" to remove both Mallory and Darth from the list. Now we're back to seeing everybody. This view of the graph is somewhat simplified because we're showing only the shortest paths between peers. If I press Shift-s now we're looking at all paths. Here, we can see that in addition to going from Alice to Bob as a source directly, we also go from Alice to Bob through Eve as a source. Let's go back to showing only the shortest paths. Currently, the max hops for sources and for blockers is set to 3. Let's see what happens if we set the blockers max hops to 0. After the graph reloads, we now see that Mallory shows up as a source, not as a blocked source. This is because Bob, who is one hop away from Alice, is no longer included as a blocker, because we've set the blockers max hops to 0. And since Bob is not a blocker, the fact that Bob blocks Mallory doesn't factor in. While Alice is the only one who gets to decide who her direct sources, blockers, and blocked peers are, anyone can view Alice's peer graph, tinker with the sources or blockers max hops, and use her list of sources to do a search. Let's see what happens if we set sources max hops to 2. Well, the graph doesn't change, because we only ever went out 2 hops anyway. Let's set source's max hops to 1. Now we only see Bob and Eve, since "kb3" and Mallory are 2 hops out. Source hops 0. Now we're left with just Alice. Now I'd like to create a new hyperdrive and add Alice as a source. I'll run M-x hyperdrive-new. Now it's prompting me for a new hyperdrive seed. A seed is a string of characters that's combined with your secret key, which is generated for you by the gateway program, in order to create a new public key for each new hyperdrive. I'll type in "Joseph". A new drive. Now, I'll right-click on my own name, and in the context menu, I'll choose "Set Relation From". From Joseph... to Alice...as a source. Now, I'll right-click on my name again, and click on "Peer Graph". Sources max hops, let's say three and [three] for blockers as well, Here's the peer graph from my newly created hyperdrive's perspective. Mallory is included as a source, three hops from the root. Let's open the list view. If we were to mark Alice as a blocker by clicking on this button in the blocker column next to the petname for Alice, Then we see that Mallory shows up as a blocked source. Only now that Joseph includes Alice as a blocker does the fact that Bob blocked Mallory take effect. You can click on another peer to set them as the root hyperdrive. I'll click on Eve. Now we're looking at the peer graph from Eve's perspective. This peer graph view has a history. If I open the transient menu, you'll see here there's a back and a forward button. Back is bound to "l" for "left". If I press "l", we're looking at the graph from Joseph's perspective. If I press "l" again, now we're looking at Alice, as at the beginning. I'll press "r" to go forward, and "r" again, and now we're back at Eve. Now, I'd like to show you the hyperdrive-org-transclusion package we published. I'll open up the hyperdrive manual to see how to install it. Now, from here, I'll press "m" and type in transclusion to jump straight to the "Org-transclusion integration" section of the manual. To summarize, we can install hyperdrive-org-transclusion with M-x package-install RET hyperdrive-org-transclusion. Then, once this is done, we will copy the following snippet and add it either to our init.el file or, in this case, I'll just evaluate it. Now, hyperdrive-org-transclusion-mode is enabled. Let's go back to the hello.org file in Alice's hyperdrive. I'll right-click on Alice, then click "Open Hyperdrive", and hit RET on hello.org. This time, I'm not prompted to mark Alice's hyperdrive as safe, since we did it already. I'll navigate to the "Check out my peers" heading. Then I'll run M-x org-store-link. Now, I'll create a new file in my hyperdrive by running M-x hyperdrive-find-file. I'll select my hyperdrive and I'll call this new file response-to-alice.org. Now, I'll paste in a response that I've written. I'll go to the bottom and I will run M-x org-insert-link and I'll insert the link that I stored earlier. As a link description, I'll change it to "Snippet from Alice's hyperdrive file". I'll save my hyperdrive file. Now, to show that the link works, I'll just kill this buffer showing the hello.org file, and I'll click on the link. Now, in addition to linking from my file to the heading in Alice's file, I'd like to transclude this heading's content in my file. So I'll go to the link in my file and run M-x org-transclusion-make-from-link and I'll save my file again. All that's been added is another link, prefixed with this transclude keyword. To make the transclusion show up inline, I'll run M-x org-transclusion-add. The content of Alice's file hasn't been copied into mine. It's just being transcluded. If I run M-x org-transclusion-remove, you'll see that the underlying content is still just the transclude keyword and the link. Now some time has passed and Alice has updated her hyperdrive file. She wrote, "Please add me as a source." Since the transclude link that I added in my file does not specify a version, it will always transclude the latest version of Alice's hello.org file. If I run M-x org-transclusion-add once more, we'll see that the latest version of Alice's file gets transcluded into mine. Let's say that I want to transclude the previous version of Alice's hello.org file, and I don't want my transclusion to change when Alice updates her file. That's possible too. I'll go over to her file and run M-x hyperdrive-open-previous-version. Now when I widen the window a little, we can see in the mode line that this is version 57. For an explanation of what the version numbers mean, check out the last video or the info manual. For now, I'll go down to her "Check out my peers" heading and run M-x org-store-link. Now, in my response to Alice, I'll insert the link. I'll run org-transclusion-make-from-link. Now there are two transclusions. The first transclusion will update whenever Alice updates her file, and the second transclusion is pinned to version 57. That covers hyperdrive-install, the peer graph, and hyperdrive-org-transclusion. Feel free to check out the hyperdrive.el info manual in the ushin hyperdrive here, or on the ushin.org website. Thanks to the folks at Sopranica, we have a public XMPP group chat you can join, either using an XMPP client or anonymously from your browser. If you use Matrix, there's also a bridge. Thank you for watching, and Thank you to the EmacsConf organizers and all the presenters who made this wonderful event possible.Captioner: sachac
Q&A transcript
Questions or comments? Please e-mail contact@ushin.org