Back to the schedule
Previous: Extend Emacs to Modern GUI Applications with EAF
Next: Day 2 closing remarks

WAVEing at Repetitive Repetitive Repetitive Music

Zachary Kanfer

Download compressed .webm video (20.7M)
Download compressed .webm video (12.7M, highly compressed) View transcript

Download compressed Q&A .webm video (15.2M)
Download compressed Q&A .webm video (6.9M, highly compressed) View transcript for Q&A

During quarantine, I found myself spending time with an Android app. One of the features this app has is composing music that loops endlessly. As with many things, I wondered how much better this tool would be, if only it was inside Emacs.

This talk will explain how I made this tool inside Emacs, with detours through Emacs text properties, font rendering, the .WAVE file format, and music theory. And hopefully at the end, we'll have something worth listening to.

There are extended notes, references, and links at https://zck.org/emacsconf2020.

The source can be found at https://hg.sr.ht/~zck/zmusic/.

• Actual start and end time (EST): Start 2020-11-29T16:29; Stop 2020-11-29T16:46

Questions

Q9: What were some of the challenges with writing a special-mode for Emacs? I'm interested in getting into this in the future, but I'm not really sure where to start.

I used define-derived-mode (https://www.gnu.org/software/emacs/manual/html_node/elisp/Derived-Modes.html) to make this mode. It's really useful! For more information, I recorded a talk about making major modes (https://www.youtube.com/watch?v=gk39mp8Vy4M) a few years ago, at an EmacsNYC (https://emacsnyc.org/) meeting.

• That'd be awesome, thanks! Will do.

Q8: Any MIDI mapping possibilities?

Should be! Would just need to change the low-level.

Q7: Do you think would be possible to add a set of recorded sounds in order to use those?

Yes! Part of zmusic is tooling to make wave files, so it should be possible to slice-and-dice input data, and output valid wave files.

Q6: Have you written any actual songs (in RRRM/WAVEing)? Can you play one?

Nothing super or well put together as of now.

Q5: Are there any open source musical instrument sample libraries that could be used? E.g. "play A 440 on Piano sample 1" to provide better quality notes than built-in tones

Experimentation would be fun. However, the nice part about Emacs is that it doesn't have any external dependencies, you only need a way to play WAVes.

Q4: What is your musical background? Do you play any instruments?

Random instruments, started with recorder, played cello for a long time, now playing guitar.

Q3: Any chance for an Emacs tracker/mod player? (plays several samples arranged in the same top-down fashion with effects applied to them for chiptune and keygen music)

I don't really know what a tracker/mod player is.

Q2: Will you play us another song? (RIP ears — who needs 'em, this is awesome! it is!)

UPDATE: can confirm, it was easy to play a song myself Very nice!

However git clone https://hg.sr.ht/~zck/zmusic didn't work, I wonder if I'm doing it wrong

• Had to browse to https://hg.sr.ht/~zck/zmusic/browse/zmusic.el and copy/paste.
• It's mercurial! (I have Opinions about version control systems). Try hg clone instead, or copy/paste from the link directly.
• BAM! hg clone works fine.
  • Hooray!

Q1: Why do you go top-to-bottom for time progression and left-to-right for low-to-high in stead of doing it pivoted? (e.g. higher is higher tone, left-to-right is time progression). This is awesome by the way!

The initial app (the inspiration) worked this way. It is definitely something worth looking into.

Notes

Notes, references, and links at https://zck.org/emacsconf2020

Transcript

(00:04) Hi. I'm Zachary Kanfer, and this is waving at repetitive repetitive repetitive music. Over quarantine, I've been bored, and I found this Android app that has a bunch of mini-games, one of which lets you compose music. And it works, but I want a little bit more functionality than it offers, it's not very flexible. So, I thought what if I made this, and what programs could I make this in that are really flexible, are really customizable. Emacs. So, I looked into it, and Emacs can play sounds, right? If you hit control g a couple of times, you'll hear like an error tone, and it turns out that, that is actually playing a WAVE file, but what's a WAVE file?

(00:58) Well, it turns out that WAVE is a musical file format, or really an annoying file format. So, data in it can be an unsigned integer or a signed integer, it's not consistent, and it's little-endian by default which is not the way I like to think about it. Now, you can set a WAVE file to be big-endian, but if you do that, Emacs can't play it. So, little-endian it is. There's also duplicate data fields. Here are some fields that are fine, but then there's a fourth field that's calculated based on multiplying two of the other ones together, and then there's another data field that's… you multiply those three ones together. So, it's just repetitive and unnecessary, but you have to do it, or it's not a valid WAVE file. Also, the last part of the file is described as data or as one website I found said, the actual sound data.

(02:04) Now, I don't know about you but when I see that, I think, what is data? It turns out that sound is just a wave, and the data is just a bunch of measurements of the height of that wave forming each sample. So, this wave starts at 8 goes 9, 11, 13, 14, 15, and then back down. If you just take those measurements, those numbers, put them in a file, that's all your data is.

(02:35) All right. Let's go to a demo of my program. So, this is what zmusic looks like. The blue highlighted row is a single beat, there's 16 of them in this zmusic file, and each dash in the row is a single note increasing…, starting really low all the way on the left and going up as we go to the right. So, if we started playing, we won't hear anything, but we'll see the highlighted beat is the currently playing one, and we see that it loops. So, we can stop it, and now we can click to add some notes. [Music] Even more than one note at the same time works. [Music] And we can even add notes while it's playing. [Music]

(04:08) Okay, here are some other features that I didn't have time to demo. So, you can save the music to a file, and this is interesting because normally if there's no note in a beat, we just don't play that beat, but if you're writing to a file you have to put something in, so when it's playing it knows to not make a sound there. We can also use different scales. We're using the minor pentatonic in the demo, but you can use the major scale the minor scale or anything else. And there's also keyboard support, but it sounds really bad, and I'll explain why later.

(04:48) Here are some things I learned while writing zmusic. Emacs has buttons which are great as long as you don't put two of them right next to each other. So, if you do that, mousing over one of them highlights both of them. Now, that's because a button is really just a series of characters with a text property to highlight them. So, the fix is, you put another character between the two buttons, then mousing over one of them only highlights the one you want, but even this doesn't work really great for zmusic, because zmusic has a lot of very small buttons in a row. So, it's really easy to accidentally put your cursor over the space and click on that instead of the button. So, I looked into unicode, and I found this character called a zero-width space. So, we should be able to put that between buttons and not be able to accidentally click on it. Unfortunately, a zero-width space isn't actually zero width. If we put a hundred of them between two other characters, you can see there's space there, and I think what's happening is, the space is zero width but then Emacs put uses one pixel between each pair of characters for the cursor, so it's almost zero width. Some ways to play sound that don't quite work! play-sound plays music, but it blocks, you can't do things like, set other notes or even pause the music. And if you throw it into async.el, it's silent, and I don't know why. So, the solution I went with is taking that WAVE file, ran into the file system, and then shelling out to a native executable to play the sound. And that works fine as long as you only do it once, because if you do it a couple of times at the same time like if you have a chord, and you want to play three notes simultaneously, you get this weird interference, and that's actually why the keyboard from before didn't work. Also, side effects have this unexpected impact, when you saw the demo it was running pretty smoothly, but if I just add one message statement every beat for debugging purposes, I was getting lag and jitter.

(07:03) Here's the one thing I learned about music theory, music theory is not easy to program. I was looking around to see what concepts we can use to code the scales, to code the notes, the first thing that I saw is scale degrees, and this when I looked into it, you don't want to program in scale degrees. So, you see we have the first, second, third, fourth, fifth, sixth, seventh but then it wraps around. That octave up is also a first, and that's because both of those notes are C, so that didn't work, and also you couldn't really easily specify a flat or sharp. You could say a sharp third or you know, a flat seventh or whatever, but then you kind of have these two pieces of data that indicate the note, and I didn't love that.

(07:55) So, I looked again, and I found intervals, and then I thought about it, and you don't really want to program in intervals either. It fixes some of the problems with scale degrees, you see, all the way on the right you have an octave, so you wrap to 8, and you go 9, 10 and that works. But you solve the same problem, you see you have a major third but below we also have a minor third, so you saw that problem of having two pieces of information.

(08:20) So, I thought about it. Music is really frequencies. Like an A is 440 hertz. So, at a low level that's what we're going to do, we're just going to use frequencies. And then at the one level above that, that's a little bit easier for humans to think about, we're going to use semitones up from the root, which is kind of like scale degrees, but instead of just counting each note as one more, we're going to say how many semitones up it is. So, if there's a sharp between two notes, that's going to be two steps up instead of just one. And then we translate those two frequencies, so your A is 440 hertz, another note might be 613.5, or whatever, and that's we use the low level to play.

(09:02) Some future work I have, I want to add some drums. I want to make that keyboard actually work, and computers and synthesizers are the only place you hear a pure sine wave like the one we have here, so I want to add overtones or other octaves above it just to make it sound a little bit more realistic. I've put notes references and the source code up at https://zck.org/emacsconf2020 [updated]. I'm one of the organizers of EmacsNYC check that out. And if you take a look or have any thoughts, I'd love to hear them, and thanks so much for coming to my talk.

Transcript: Q&A

(00:09) Okay, yeah, so I'm Zachary Kanfer, let's go to the questions. The first question, "Why do we go top to bottom for time progression, and left to right for low to high?" Interesting, I think…, so the initial thing I was copying, that initial app work this way, and yeah, I mean, certainly traditional music, you know, on a staff does go left to right like this. I mean, going top to bottom does make it easier to add more beats without having to wrap, but certainly that could be managed. Yeah, I had not really thought about it, but it is definitely something worth looking into.

(01:21) Two, "Will you play us another song?" Not now, I can make some recordings of it, or certainly you can try it. I couldn't quite get the microphone and the webcam and everything to work with the sound playing now. So, I can record some. Also, I have put a link in the Etherpad, https://zck.org/emacsconf2020 [updated], where you can go and get the source, and you can try it yourself. There's no dependencies needed, so it's just all in Emacs. So, please, you know, try it yourself.

(01:57) "Any chance for an Emacs tracker or mod player?" I don't really know what a mod player or tracker are, but I mean, I'm sure that would be cool, maybe there's one on now, but I don't know.

(02:11) My musical background. So, I've played various instruments since about the third grade. Started recorder, play cello, I play guitar now. But yeah, so just kind of random instruments, and I guess kind of some of those things influence how I think about music.

(02:33) "Are there any open source musical management sample libraries that could be used?" Good question, I'm sure there are, I don't know any of that integrate really well with Emacs. One of the cool things that I liked about this is that there are no dependencies, you know, you don't need any external program to generate the music. I mean, it does shell out to to play, but that should be able to be done on any operating system, as always, you have something that can play WAVE files, but yeah, it is interesting to kind of try the different sounds and different tones that you could get with different instruments.

(03:14) Have I written any actual songs? Nothing super well put together, I kind of just been playing around with this. It's kind of… making this was one of those things where once I made it, I was like, okay, now I can play with it, and I did a little bit, and was like, I don't know if I feel like it right now. You know, which I've found that to be the case with some things that I've implemented in Emacs where it's… I make it, and then it's the kind of some of the desire to use it all the time goes away, but I'm sure I'll circle back around at some point especially kind of maybe once I add in different tones or something.

(03:54) I guess a similar question for pre-recorded sounds. Yeah, I mean, if it's… part of what I did, what I wrote was a WAVE generation library, so, if you kind of have the data, you could use those and chop them up and take certain lengths of them and make a WAVE file, so it's not plug and play right now, but you could certainly add those notes to do it.

(04:19) "Any MIDI mapping possibilities?" I haven't looked into it, but I'm sure you definitely could output to MIDI which is another benefit of having that multiple layers with the top layer is just, you know, if the root note is this, we're just two semitones up or seven semitones up or whatever it is. It should be relatively simple to kind of switch out that layer underneath from WAVE to MIDI or other things.

(04:44) "What were some of the challenges with writing a special mode for Emacs?" Interested in getting into this, not sure where to start. This isn't the first mode I've written, so that's right…, certainly that helps. I actually… I have a video that we recorded it as part of EmacsNYC on making a major mode. That's basically starts from nothing, and kind of builds up to an implementation of tic-tac-toe, but so it kind of goes into printing things out and buttons and making the mode. I mean, one of the best parts about Emacs is, because it's so configurable and so introspectible, you can start pretty simply, and just kind of ask Emacs about things, and then make one little change. It's really… it's not that bad, so, I'll try to throw a link up on that page I put up, or please email me for whoever asked this question to get a link to that video, or just look at the source code of this or any other major mode. Emacs makes it pretty easy to extend major modes.

(05:54) And I think that's the last question in the Etherpad, so, thanks so much everybody for coming. (Amin: Thank you so much to Zachary for your awesome talk, and for doing live questions. Thank you.) Thank you. (Amin: Cheers.)

Sunday, Nov 29 2020, ~ 4:33 PM - 4:43 PM EST
Sunday, Nov 29 2020, ~ 1:33 PM - 1:43 PM PST
Sunday, Nov 29 2020, ~ 9:33 PM - 9:43 PM UTC
Sunday, Nov 29 2020, ~10:33 PM - 10:43 PM CET
Monday, Nov 30 2020, ~ 5:33 AM - 5:43 AM +08

Back to the schedule
Previous: Extend Emacs to Modern GUI Applications with EAF
Next: Day 2 closing remarks