Health data journaling and visualization with Org Mode and gnuplot

David O'Toole (he/him, https://emacs.ch/@dto)

In this talk, David O'Toole shares how he tracked and graphed his personal data using Org mode and Gnuplot, and how you can use a template generator to do the same. Afterwards, he will handle questions over BigBlueButton.

The following image shows where the talk is in the schedule for Sat 2022-12-03. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 25-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-health
Discuss on IRC: #emacsconf-gen
Status: TO_CAPTION_QA

Talk

00:00.000 Introduction
00:33.640 How to take daily health journal items
01:59.440 How to set up your org templates
03:38.320 How to do it in GNU Emacs
04:16.840 Overview of the presentation
04:51.960 The journal
05:52.800 The capture buffer
06:51.320 The columnview table
08:03.480 Gnuplot
09:03.320 Output parameters
10:15.480 Time series data
13:05.920 Health variables
14:22.680 Goal lines
15:12.000 The Gnuplot command
17:35.560 The template generator
19:11.480 The code that creates a template
21:41.000 The power of the chart
24:09.920 Thanks

Q&A

00:50.800 Do you use this just for yourself? Or do you use this to discuss/show with doctors/health professionals?
02:38.157 How do you input the health data?
03:53.121 How do you track the various health statistics that you are gathering?
04:50.422 It's possible to download data from the Apple watch's health app. Is it easy enough to incorporate those .csv files into your implementation of Gnuplot?
06:48.800 Regarding the medication tracking you only have option to record missed or not. If one needs to take multiple medication throughout the day, how would you propose to track that? Within gnuplot or separate?
08:27.720 How's the workflow when working on the gnuplot code?
09:07.680 How much time does it take to process the amount of data that you add inside GNU Emacs?
09:38.190 Will indent-guide behave well with yaml files for helm?

Description

Note: The correct name of gnuplot is gnuplot, not GNUplot. It is not GNU software, nor is it licensed under a GNU license. Details

"GNU Emacs with Org Mode and gnuplot offer a complete Free Software solution for tracking health information based on a daily data journal. With Org capture templates and data tables, you can easily enter facts for each day such as hours of sleep, minutes of exercise, doses of medication, weight, or even add freeform notes. Through the use of subjective numeric scores you can track symptoms like stress, anxiety, or pain. Org Mode's gnuplot support enables you to visualize the data in order to spot correlations and evaluate trends. The use of detailed record-keeping and graphing can help you communicate with health care providers."

A reusable org template will be provided at: https://gitlab.com/dto/health-template

Discussion

Notes

• I agree that just tracking is a great thing. I heard of some people starting to lose weight, just because they started recording it daily.
• just Thank you very much
• I like this demo of plotting use
• Damn, this template looks awesome, nice work
• (incf eieio-reputation)
• huh, I could probably use that thing that shows a line for other sexps at the same level...
  • Yeah, that looks super handy!
  • A: yeah it's cool. i think it's called indent-guide
• very good talk!!!
• emacs is like willy wonkas candy factory
• thanks for the talk; was thinking that you could use a mobile app like BeOrg and cloud file sync to help automate your data entry
  • hmm i haven't heard of BeOrg i'll have to look it up!

Questions and answers

• Q: do you use this just for yourself? Or do you use this to discuss/show with doctors/health professionals? 
  • A:
• Q:How do you input the health data? semi automated with org mode capture templets?, copy paste, Automated with a smart watch and ifttt or tasker in an org mode document to automatily add stuff like sleeping data? and which parts are and are not automated, semi automated or manual.
  • A:All manually
• Q:How do you track the various health statistics that you are gathering?
  • A:Manaully for example for sleeping I look at the clock before and after I go to sleep.
• Q: It's possible to download data from the apple watch's health app. Is it easy enough to incorporate those .csv files into your implementation of Gnuplot? 
  • A:You can import csv to org. Just copy paste the csv data to org, highlight it and press C-c | (the function is org-table-create-or-convert-from-region)
• Q: Regarding the medication tracking you only have option to record missed or not. If one needs to take multiple medication throughout the day, how would you propose to track that? Within gnuplot or separate?
  • A:
• Q:How's the workflow when working on the gnuplot code -- can you e.g. C-c C-c and the svg output on the right is updated automatically?
  • A:Use auto-revert-tail-mode in the buffer that displays the svg, then it will update automatically
• Q: How much time does it take to process the amount of data that you add inside GNU Emacs?
  • A: some seconds (no issues to manage that inside the editor)
• Q: will indent-guide behave well with yaml files for helm?
  • A: 
• Q: Have you noticed your behaviour changing as a result of tracking your data?
  • A: Yes, definitively and in a good way
• Q: did you not have exercise data for the final period of bad sleep, or was it the case that you did not exercise? This question tells me that you want to symbolically represent missing data differently from "0". And, if the answer is "no exercise", then it is possible that this is a factor in your lack of sleep, it's not necessarily the withdrawal from nicotine. As always, use caution when drawing up causal links from correlation. Recommended in this context - "The book of why", by Judea Pearl.
  • A: 

Transcript

[00:00:00.000] Hi, this is Dave O'Toole, and today I'll be giving a presentation on tracking health data with Emacs, Org Mode, and Gnuplot. So Gnuplot is the well-known scientific and mathematical plotting application. You feed it text files full of names, dates, numbers, data points, and you get out a nice graph. You can spit out SVG. You can spit out PNG graphics. In this case, we're using an SVG.

[00:00:33.640] What I'm going to show you today is how to take daily health journal items: in other words, things like I exercised such and such number of minutes today, I got X hours of sleep last night, I used such and such number of pieces of nicotine gum, say five pieces. So let's see, we've got this whole picture here, all right, and I've tracked here... This is a month of data from my life. This is... I'm not showing all the variables, but this is what I felt comfortable sharing in order to help people who might have a need to track, either because of a chronic condition, or just because of a health improvement goal or what have you, people who might need to track health data in a way that's a little bit more robust than just one or two variables and just weight or just blood pressure. So in this case, I've got exercise, I've got the number of hours of sleep, the number of doses of nicotine, (that's the yellow line here), and this is referring to nicotine gum. What we're going to be talking about is looking at connections, the idea that plotting your data can actually help you figure out what's going on. This is just one month. I've been doing this for a couple of months now, but I felt comfortable showing one month with a limited subset of the variables.

[00:01:59.440] What I'm going to be doing in this presentation is showing you how to set up your org templates so that you can, you know, hit a hotkey to capture today's data with an org template-- or in this case yesterday's. Usually I'm saying, okay, yesterday this happened, because you don't know until the day's over how many pieces of nicotine gum you ate or how many hours you slept. So usually we're recording data for the previous day. We can set up a capture template so that it fills a little org entry. One for exercise, one for sleep, one for nicotine, one for distress. Here distress is just 1 to 10: how bad do you feel today? It's not a scientific measure, but you know, many, many things ask you to rate on a scale of 1 to 10, how bad is the anxiety, how bad is the general level of stress, and so without a lot of complication, I just rate that one to ten. Pain, okay, we won't have to get into any details, but if there is a level of chronic pain, well, I put that between 1 and 10. As we can see here, during the period that I've shown you, it's pretty low. There's some. If you miss a dose of medication, you can track that, in this case with a big ugly red triangle, you know. You can see, I can see here that in mid-, in late September, sorry, in early to mid-October, I stopped using the nicotine gum and probably should have cut down more gradually because my sleep suffered. Look at this. The sleep line is down here, okay?

[00:03:38.320] What I'm going to do now, now that I've shown you the graph and some of the things that are useful about it, I'm going to actually take a step back and show you from start to finish how you can do this in GNU Emacs, and I have a little template generator that you can use if you'd like. All right, so let's go back. Let's step back from this file. We're going to split the screen, and on the left side, I'm going to put the underlying Org file that generates this graph. Let me shrink that a little bit.

[00:04:16.840] All right, I'm going to work my way backwards from the template to the template generator, meaning that you'll be able to spit out, given your own specification of health variables, you'll be able to have it spit out a custom Gnuplot script like this that's preset up with the definitions for the column view in Org mode. I'm assuming a little bit of familiarity with Org mode and Gnuplotting, but I'll try to explain as much as I can as I go along.

[00:04:51.960] The journal here is where... okay, okay, one moment. So as you can see, there's a sub-entry here for each day that I've included from my data set starting on September 13th of this year and ending on October 17th. And there's an Org property drawer with the corresponding names of each field and the value. Now the idea here is that the columns specify... if you know a little bit about Org mode, what happens is that you... let's say that I hit the key for my journal template, which... Mine is very similar.

[00:05:52.800] This is the capture buffer for today's date, and if you're recording yesterday's date, you can just flip it like that if you need to. Then I say, yesterday, I remember I went for about a one-mile walk, so that's probably about 20 minutes, and that I had such and such, I had eight and a half hours of sleep, let's say. I estimate how many pieces of nicotine gum I have. I try to count as closely as I can, how much distress, you know what I mean, whether or not I missed a dose of medication. Then when you hit C-c C-c, it captures that to the end of your Org file. Now what this shows is that... I cut and paste it in. I've been keeping these entries every day for months, and that I cut and pasted in a month of data.

[00:06:51.320] Now I'm going to dig in a little bit to the Gnuplot script. This here, all this stuff, is one component of the graph, and I'll go over how it works. First, the items through this column declaration here, and the id:myid, this columnview table here, #+BEGIN: columnview, this whole bit here, is going to get filled in with the corresponding columns, exercise minutes, sleep hours, nicotine doses. And then it gets pumped out by Org mode into a file that looks like this: tab-separated values with an ISO-style date at the beginning.

[00:08:03.480] So what we're going to do is we're going to go through the Gnuplot portion of this, and I'm going to enlarge the font a little. I'm going to go line by line through the Gnuplot portion. Now, my template generator will give you one like this. You don't have to write this from scratch. But I'm going to go through it line by line because if you do use the template, then it'll help to have gone through it line by line, because you're probably going to have to modify it. So first, we're going to clear the graphics from any previous runs so that if we reuse the same Gnuplot process, we're not overwriting the old-- that we are completely overwriting the old image. So that's the purpose of this line here.

[00:09:03.320] The output parameters: we want to put out an SVG file. Font Arial, that's funny, but I don't know what font it's actually ending up choosing, but it looks fine. Then we want it to be square, so I'm giving it 900 by 900 pixels, even though it is a scalable vector graphic. We're putting it in the same folder as the org file, example.svg. These lines here set it up to use the Org mode format that we showed in the other file over here. The time format is four-digit year, two-digit month, two-digit day. The time format doesn't specify here the time, but that doesn't seem to mess it up. This line "set datafile separator" means that the separators between that and between all the other fields are tabs, which is what Org mode does when it spits out a table by default. Okay, along to the next lines.

[00:10:15.480] We're going to set up for time series data, meaning that the x-axis is going to be time, x2tics 1 format. I believe this means that every day has one tick and that this tells it that the first-- unfortunately, I forget the exact meaning of this one line. I'm just going to move on. We want one X tick per day, and because X is in seconds, it's 24 hours times 60 minutes times 60 seconds. This line "set grid xtics" gives us a vertical line on each day of the graph. I'll pull up the graph just so that it's a little easier to see. All these vertical lines, one on each day, that's given to you by "set grid xtics". One Y tick every five points. So here at five pieces of nicotine, we've got a five, at ten pieces – well, we don't want to eat ten pieces, but ten, fifteen, twenty. Rotating the labels to make them fit a little bit better, that's this part here where the labels are sideways, and even with just one month of data, they're getting a little crowded. This "set key box lc" just makes the line around the key, the legend here, a little bit less severe. set xtics format: this makes it so that, for example, I've done a United-States-style date here with the month and then the day. You don't necessarily have to do that. You can have whatever you want. This xtics format, that relates to how the dates are printed. Remember that over here, this set timefmt, that relates to how the dates are formatted in the Org mode output. So remember, those are two... You don't want to mix those up. All right, "yrange [0:40]". Thus far, my exercise sessions have all been less than 30 minutes, and nothing's gone over 30. If you have a health variable that is in a significantly different range, you may need to get a slightly more complicated Gnuplot script because it is possible to plot multiple yranges in one plot if you have a variable that uses a different range. It's just a little trickier. These parts here, aside from the fact that you might make some changes that relate to the date and your country format, are going to be the same. This is like boilerplate for almost anything.

[00:13:05.920] Now here are the parts that are going to vary depending on what health variables you want to store. There are three main sections here. One is setting the different line types that are used. Setting linetype 1 with line width 2, line color RGB. Unfortunately, Gnuplot is a little bit cryptic, which is why I've made this template generator that I'll show you in a moment. I pick a color. So this is exercise, forest green. Point size 1, meaning you get these little green triangles about that size. But the point type 9 is the pointing up triangle. Line type 2, purple. So that's the sleep line. So we're just establishing these different line types that we've given arbitrary numbers. Now onto the next section. Oh, before I move on here, you can see point type 11 for line 5, which is red. And that's the missed medications line, so you get a triangle that's upside down because that's point shape 11.

[00:14:22.680] All right. The next section here is the goal lines. There are horizontal dashed lines here at 8 purple hours of sleep, because 8 hours is the goal. So there's a horizontal line at Y = 8. For pieces of nicotine gum, I'm trying to keep it to around 5 right now. So my goal line is at 5. So these... Here, a goal of at least 20 minutes of exercise. Sometimes I get more, sometimes I get less. There's a green line and a 20, showing that that's the goal. These lines here are actually the goal lines. You can specify the goal for each one in the template generator that I'll show you.

[00:15:12.000] The last part is the actual plot command. So the dependent... So okay, these all start with 1, "using 1" against this variable. So $2... This is a ternary operator here that says if the value of the second column is zero, then don't plot a point. In other words, not a number means it won't plot a point. The template generator lets you skip over the details of that. It sticks this in there. I'll show you. So we only want to plot a point when the value is non-zero. If there was no exercise, we're not plotting a point. The with construct means we'll plot data using date against exercise with points, the title is "exercise (minutes)", line type 1. Remember, we established line type 1 up here as being forest green, point style 1, point type 9, green triangles. Now I'm going to show 1 against column 3, which is "hours of sleep". This one is plotted with lines, so we don't specify a point type or point size, just a line type 2. And remember, you can see that line type 2 is defined as purple with point type 1, point size 1. Okay, so I did specify point size and point type, but because I'm not plotting with points, those are ignored. Here we come to the line with nicotine. The fourth column is the nicotine number, the fourth column from the Org mode file. So here you can see how we're telling Gnuplot to take each column of the tab-separated Org mode file and put it into the graph. The line types are set up here. The goal lines are set up here. And then the actual plot command is set up here.

[00:17:35.560] So now we're going to work further backwards from this Gnuplot template to the template generator that I used to make it. Now I'm not going to go into all of the details of the code, but what I am going to show you is that there's a variable called health-factors. And what this does, this health-factors-from-list lets you specify, with a property list of keyword and value pairs (here's the keyword name and the value is exercise), the goal that I want 20 minutes of exercise, that the unit is minutes, that the color is forest green, and so on. The aspects of the Gnuplot setup have been abstracted here. Eight hours of sleep is the goal here. The hours are units. What color, what thickness of the line. Here we specify the number of points. There's references online that show you what point types are what shapes in Gnuplot, and so on and so forth.

[00:19:11.480] I'll walk through the code a little bit that does this, that actually takes these pieces, that takes this specification of what your variables are and turns it into a template. First, I'm using EIEIO, the object system that's included with GNU Emacs. It's a reasonable facsimile of the Common Lisp Object System. What I'm going to be doing here is defining a class with each of those items, those properties that we talked about in that list that lets you specify name, what the goal is, what the units are, and the Gnuplot things (the Gnuplot parameters like thickness, plot type, and all that) into a class that will then spit out the template once you feed it some of these health factor objects. So just a moment. For example, you can see that this template originally came from being generated by this code here. To use the template, to use this little template generator... See, here's where it spits out the line type given the pieces. This is all just text formatting. This is one of the things that Emacs Lisp just really excels at. I need to take a piece of data like a list of health information, a list of health variables, what their units are, and how they're supposed to be formatted in Gnuplot, and go from that to the nice template. So that's pretty much the whole thing. I want to see if there's anything I missed.

[00:21:41.000] Bring up the chart. This has been really useful for communicating with healthcare professionals because you are both on the same page about exactly what is happening, what's been happening because if... Let's say that you're tired when you talk to your care provider. Well, if you have objective information that you've been recording every day, that you're ahead of the game, really, because you don't need, necessarily, the presence of mind to be able to give your care provider a complete picture of what's going on in your world. If you can find those few minutes a day to enter-- not even a few minutes, really just a minute to enter the data and say what happened yesterday... I'm finding over these months that I've been more in touch with my health when I can-- not forced, but when I have the habit, the consistent habit every single day of recording that data--I'm accountable to myself. It's interesting. I guess it gets into a little bit of ideas about the Quantified Self and how holding yourself accountable can change what you do and what the outcomes are. Just look at this here. Without getting into too much detail, one of the reasons I track my sleep is because, as you can see, my sleep is not as well-regulated as most people, and that's why I need to do that. This was a time... 10, 12, here's 14 hours of sleep, that's depression. It oscillates a little bit. But then below the goal line, the things are a little more normal here. This is a little more normal. But then, really, without thinking about it too much, I cut out the nicotine, and my sleep suffered. Just the fact that I'm able to look and see that connection is really amazing to me. Maybe I would have anyway, but looking at the whole months of data, there have been many things to discuss and many things to think about.

[00:24:09.920] Because this is a short presentation, I probably should wrap up. I just want to thank the whole Emacs community for being there and for including me in the conference and I hope to participate next year as well. Thank you so much.

Captioner: sachac

Questions or comments? Please e-mail emacsconf-org-private@gnu.org

CategoryOrgMode