Putting Org Mode on the Indieweb

Michael Herstine (IRC: sp1ff)

In this talk, Michael Herstine shows how to set up Org Mode for sending and publishing Webmentions as part of the Indieweb. Afterwards, he will handle questions via BigBlueButton.

The following image shows where the talk is in the schedule for Sun 2022-12-04. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 17-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-indieweb
Discuss on IRC: #emacsconf-gen
Status: TO_INDEX_QA

Talk

00:00.000 Introduction
00:14.080 Maintaining a personal website using Org mode
01:13.880 Problems with comments and isolation
02:32.040 The Indieweb
03:17.800 Webmentions
05:00.600 Decentralized commenting
07:04.160 The publication framework
08:48.080 Sending Webmentions
09:54.000 The process of publication
11:36.840 Defining new link types
12:31.920 org-publish-to
14:07.349 Sending Webmentions with request.el
14:46.520 Summary
15:51.400 The future

Q&A

00:00.000 Introduction
01:05.500 Q0 - How did you create the drill down representation of the make call?
02:22.132 Q1 - Have you seen Reclaim ID?
02:22.840 Q2 - what happens when you republish or re-export the same post will the web mentions be sent out repeatedly
03:14.300 Q3 - Backend storage - alternative to database
04:01.107 Q4 - Any thoughts on using it with ox-hugo?
05:27.632 Q6 - web 1, web 2 vs web3 
05:57.207 Intermission - opening the Q&A message
08:12.207 Q7 - Cutting telegraph out of the equation?
09:08.715 Q8 - Web3 - indieweb vs(?) static site
09:55.232 Q9 - Do you have a process running on the webserver to receive requests?
10:44.857 Q8 (bis) - discussion about Web3 + having a demo
12:20.957 Q9 - How satisfied are you? - current vs alternative indieweb protocols
13:44.440 Q10 - Are you not doing too much inside emacs?
14:32.540 Q11 - If you start with Telegraph and then want to change the endpoint, what happens?
15:55.814 Q12 - Have you seen Agregore browser?
16:13.914 Q13 - Are you not doing too much inside emacs? (bis)
17:33.690 Closing and moving to the next speaker

Description

Table of Contents

Many of us maintain personal websites using Org Mode. While an Org-generated static site has advantages over full-blown Content Management Systems, its simplicity comes with costs such as fewer features. The first feature I missed was supporting comments on my site, but I quickly began to feel isolated on the web altogether.

Enter the Indieweb: the Indieweb is a collection of protocols for connecting to other independent sites, pushing your content to social media sites, collecting likes, comments & responses from other sites back to yours, and many other things as well.

In this talk, I'll briefly sketch out the dilemma of the independent web site & how the Indieweb tries to address it. The focus, however, will be on how Emacs, Org Mode, and a few Unix tools suffice to get your static Org Mode site onto the Indieweb.

(Update July 2023) I've created a simple site demonstrating how to use the package here. The source for the site is on Github for anyone to see or tinker with.

Michael is a developer and long-time Emacs user from the San Francisco Bay area. He hacks in C++, Lisp & Rust and thinks a lot about writing provably correct code. You can find him at:

• his home page
• on IRC: sp1ff on Libera.Chat
• through e-mail
• or on Github

Discussion

Notes

• Thanks for a great talk - this was the missing piece in the puzzle of the website I'm tinkering with!
  • I'm so happy to hear that!
• Similar: http://www.by-star.net/ and http://www.by-star.net/content/generated/doc.free/bystar/PLPC/180038/current/accessPage

Questions and answers

• Q: Have you wu. This allows you to make a login that you own or at least is more open souree and ownable. Seems to fit in with indie web
  • A: No but I will check it out later
• Q: How did you create the graphical drill-down representation of the make call? Is it hand-written and scanned in?
  • A: LOL... I started with a diagram I made at https://excalidraw.com/. The initial diagram was the complete flow. Exported that to Gimp & made the intermediate slides there
• Q: what happens when you re-publish or re-export the same post? i.e. will the webmention be sent out repeatedly?
  • A: You could do that but it would annoy the recipient. The system is smart enough not to re-send the mentions.
• Q: An advantage I see to using org mode for the indieweb  for this is you can use it for your notes "org roam for example" and org mode no-export for private data you don't want ot share. Your webmentions could be org files as notes.. Anything else good about using org mode for this.
  • A:
• Q:  Any thoughts on using with with Ox-Hugo?
  • A: No. So far, I was following an Emacs/Org mode OOTB (out of the box) approach and pushing emacs as far as I could take it
• Q: So, is this a Web3 approach? Web 1 = static sites Web 2 = interactive sites- but centralized
  • A: Indieweb is about reclaiming your data. It's a distibuted approach.
  • Annotation Karl: some people started the term Web0 for similar, decentralized approaches.
  • Annotation Karl: Web3 is supposed to be something really strange with Blockchains and this is definitely nothing like that Web3 (which will be a dead bubble in a few months IMHO)
  • Here's the link: Do you see this as a format for annotations specifically, or something more general that can be used for interlinear glosses, etc?
• Q: Is there a workflow to use emacs to publish and connect directly to target websites instead of telegraph?
  • A: If you want to cut telegraph cut out of the equation, you'll need more work on the client-side. "What if they're down?" ... 
• Q: Do you have to have a process running on the web server to recieve requests?
  • A: Nope-- just a cron job
• Q: I think perhaps you are doing too much inside of emacs?
  • A: When I started, I thought that I was so close with the OOTB (out of the box) features. 
• Q: Say you start out using webmentions-as-a-service (webmention.io, telegraph.p3k.io) and then you want to chnge endpoint, whether to a different service, or to your own new server with a CGI script or something. Will that work smoothly, do you think, or will there be mentions piling up at your old address?
  • A: Should be fine, unless your senders are doing something very odd
• Q:Have you seen https://github.com/AgregoreWeb/agregore-browser It is a decentrilized kiss browser. Using some of the peer to peer protocols used whithin this could be useful for propgating stuff like webmentions.
  • A: No, I haven't (but I will soon!).

Other discussions from IRC

• I think perhaps you are doing too much inside of emacs.
  • mohsen: Somethings are better done outside of emacs. I have built something similar at http://www.by-star.net/PLPC/180038 and also please see http://www.by-star.net

Transcript

[00:00:00.000] Hey everyone, I'm Michael, and I'm going to be talking about Org mode and the IndieWeb. I am located in the San Francisco Bay Area, where I'm a developer as well as a longtime Emacs user.

[00:00:14.080] So, I maintain a personal website using Org mode. If you're watching this talk, I'm going to guess that you probably are too. For anybody who isn't, let me explain exactly what I mean by that. I have a static website. I author the pages in Org mode's markup language, otherwise known as Orgdown. I use the Org export facility to transcode that markup to HTML. And then I just use rsync to push the resulting HTML pages up to a VPS. I like the workflow. It's familiar to me as a coder. I get to use familiar tools like Git and Make. Publishing and then pushing the site to a local test server is just make. Pushing it to the live site is just make prod.

[00:01:13.880] That said, certain problems made themselves apparent with this arrangement pretty quickly. Commenting was one. It's very difficult to support commenting on a static website. I've got no database. I have no real server, and so on. Yes, of course, there is Disqus and other third party services that will do this for you, but I was uncomfortable outsourcing that job. And it was more than just comments. There was a general sense of isolation. There's no connection to places like Reddit, or Mastodon, or Twitter, you know, where all the people are. I guess you can post, then Tweet a link to it. But suppose somebody responds to your Tweet. Now you've got a conversation going on on Twitter that you're a visitor, and that a non-Twitter-using visitor to your site would be completely disconnected from. I am seeing people using Reddit effectively as the comment section for their sites. But unless you've got an audience, you know, the size of Derek's or Amos's, I don't think that's really feasible either.

[00:02:32.040] Now, after casting about for some time, I stumbled upon something called the IndieWeb. In their own words, the IndieWeb is a community of independent and personal websites connected by simple standards based on the principles of owning your domain and using it as your primary identity, publishing on your own site, and optionally syndicating elsewhere, and owning your data. I would describe it as a collection of individuals who've chosen to own their own platforms, alongside a loosely specked set of standards that tie those sites together. And it's really those standards that make the IndieWeb more than just a call for everybody to go back to the arts and blog on on their own sites.

[00:03:17.800] Now, this presentation isn't going to focus on the IndieWeb as such. It's more about using Org mode to put your site on the IndieWeb. I'm a little limited by time here, so I'm going to focus on just one of those protocols, and that's Webmentions. So, what's a Webmention? Let's begin with the inveterate Alice, who has a website and has posted content to that site. Her old friend Bob comes along, notices that content and wishes to say something about it. He posts to his site and his publication software will, if it supports Webmentions, will notice that he's mentioned Alice's post. At that point, his publication software reaches out to Alice's site, asks for the mentioned document, and will examine it to see if Alice advertises an endpoint at her site capable of receiving Webmentions. In this case, it does. So, Bob's publishing software does it. At the end of the day, a Webmention is really just an HTTP post request with two parameters, a source and a target. On receipt, Alice's server will reach out to Bob's site, request the document that contains the mention, and validate it, decide whether or not she wants to accept the Webmention. In this case, it's legit, it's accepted, and Alice chooses to make a note, to update her content, to make a note of the fact that it was mentioned by Bob.

[00:05:00.600] Now a couple of things to note here. The first is that this is effectively decentralized commenting. Both parties own their content, and there's no third party involved, trusted or otherwise. Now at this point, you might object that, well, as the owner of a statically generated site, I have essentially none of the infrastructure I need to implement this. I have no server above and beyond Apache. I can't really…, I have no database. I guess you could send Webmentions with curl, but now to do Webmention endpoint discovery, you're going to be parsing arbitrary HTML. It's a lot of work. That's okay. There are sites out there that offer Webmentions as a service. So, let's take a look at how that goes. We return to our original example. Alice continues to advertise an endpoint capable of receiving Webmentions, but it's no longer on her site. It's at webmention.io. Bob is essentially in the same position, but he now has in his world a third party site called Telegraph. When Bob wants to publish, he no longer needs to go through all the work of sending Webmention, and of carrying out Webmention endpoint discovery. He sends one API request to Telegraph, effectively saying, please send a Webmention for me. Asynchronously, telegraph.io is going to retrieve Alice's post, do Webmention endpoint discovery, and send the Webmention on Bob's behalf. Except this time he's sending it to webmention.io. Now, at some arbitrary point in the future, Alice can ask webmention.io, "Hey, do I have any new Webmentions?" And if she does, she may choose to update her content and publish.

[00:07:04.160] Okay, so let's code this up. Now, I'm recording this talk about a month ahead of time and I haven't been able to put together a little demo project. Hopefully, I can hack something together before this video streams. But in the meantime, I'd like to imagine a little test site. It has a single page, maybe we'll call it index.org, and we're going to publish it. We're going to export it to a subdirectory of our project directory, simply called www. So, the entry point to the Org export system is the function org-publish, whose docstring helpfully says, it publishes all projects. The set of all projects is defined by the variable org-publish-project-alist, which is a very flexible association list that lets you define what files are in your project, how you wish to export them, where they're going to go, etc, etc, etc. So great. This is actually pretty straightforward. We just give ourselves a little Elisp file and a single function. I'll call it publish, and all it will do is define org-publish-project-alist and invoke org-publish-all. At that point, exporting is a one-liner. We just invoke Emacs, load up our site.el file, and invoke the publish function. And if we want to publish to the live server, that's just another one-liner of ours. So, that's the publication framework.

[00:08:48.080] Now, let's take a look at sending Webmentions. The idea is that we're going to get our fingers into the publication process. Note when we see a Webmention in the process of exporting our Orgdown, and write it out to disk for sending later on. Now, I didn't want to send a Webmention for every single link in the post. I wanted this to be an intentional choice. And it turns out there are different sorts of Webmentions you can make on a page. For instance, if you add the CSS class u-in-reply-to to your link, the recipient will interpret this Webmention as a reply. There are similar CSS classes for likes, reposts, and generalized mentions. When the recipient gets your Webmention, if they want to know who's talking to them, they need to parse your page and look for DOM elements with certain CSS classes defined by the protocol as well.

[00:09:54.000] I'm a visual thinker, so I kind of drew out the process of publication, and exactly where we're going to get our fingers into this. So, this is me invoking make, which of course fires up Emacs. Just as before, my publish function will define org-publish-project-alist with one difference. There is an attribute, a property in the list called publishing-function. We're going to need to customize that. As usual, we then call…, we kick off the process by calling org-publish-all. org-publish-all will invoke your publishing function for each page, and it will hand to your publishing function for each page. The name of the file you're publishing, where it's going, and a parameter entitled plist. This is not super well documented. There are points in the docs that refer to this as a communication channel. What I discovered by simply reading the code was that it's a property list that is initialized for each invocation of your publication function. The initial properties are inherited from your project, but you are free to add properties as you go to communicate between different portions of the publication process. My publication function really does one thing, and that's simply swap out the HTML template that's passed to org-publish-to.

[00:11:36.840] Now, in order to take note of each Webmention that I send, I took advantage of another Org mode extension point called defining new link types. Here you can see I've created a new link type called reply. One of the attributes that you can attach to this is the function that is used to export your link. I've elided the code for mentions, likes, and reposts. If you look at my export function, you can see that it ultimately yields the appropriate HTML for this link. Before that, it calls a little helper function that will pull out the actual target URL of the link and shove it into this communication channel under the property name sp1ff/mentions.

[00:12:31.920] So, org-publish-to is really the workhorse of the publication process. The first thing it's going to do is transcode from the parsed Orgdown, which is an intermediate representation known as Org elements, to HTML. In particular, for every one of my new links that I'm using to mark Webmentions, it's going to invoke my little export function. And so as we work our way through the post, I'm going to accumulate all the Webmentions that I've made in the property list. The second step is to actually render the final HTML document, and that's where my specialized template comes in. All I do there is, use it to get my authorship information into the page. And then the last step is called finalization. At this point, we have the rendered HTML document, and Org mode gives you an extension point here, where you can do arbitrary post-processing on that document. I arguably abuse it to retrieve all the Webmentions I've made out of the communication channel and write them to disk. At this point, when we type make, we wind up with the rendered HTML for our Orgdown document, along with a little text file in which we've recorded all the Webmentions that need to be sent.

[00:14:07.349] The next step is to send said Webmentions. This is surprisingly easy in Emacs Lisp. This is my actual implementation. I use the request.el package to talk to Telegraph. And at this point, we really don't need to add a lot to our little site Elisp file. I sketched out a send-webmentions implementation that just goes through in a loop and calls my send-webmention function. And now publication becomes a two-step process. First, the org-publish, then sending Webmentions.

[00:14:46.520] Okay, so I realize this has been a bit of a whirlwind. So, where are we now? We have a sample site that we can publish and have sent Webmentions. And we've done it with just Emacs, Org mode, a little Lisp, and a make file. If you'd like to see more, I've put my library up on GitHub. It has logic for both sending and receiving Webmentions as well as something that on the IndieWeb is called POSSE, which is an acronym standing for Publish (on your) Own Site, Syndicate Elsewhere. What that means is that you turn the publication step from merely publishing new content to your site to also replicating it to places like Twitter and Facebook and so forth. And also when people like, comment, and retweet your content, that gets fed back to your site, where you can display it as comments.

[00:15:51.400] In terms of the future, I feel like I'm at a decision point. Org mode is admirably flexible, and I'm confident that I can continue to add support for IndieWeb protocols. On the other hand, it is so flexible that the process of figuring out which extension points to use in any situation is very challenging. When I started down this path, my mindset was keep it simple and let's just see how far I could get with Org mode. And I feel like I might be bumping up against the limitations of that approach now. Thank you very much.

Captioner: bhavin192

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

CategoryOrgMode