How to incorporate handwritten notes into Emacs Orgmode

Bala Ramadurai (his/him, bala@balaramadurai.net)

In this talk, Bala Ramadurai shares how he takes handwritten notes and includes them in his Org Mode files. Afterwards, he will handle questions over BigBlueButton.

The following image shows where the talk is in the schedule for Sat 2022-12-03. Solid lines show talks with Q&A via BigBlueButton. Dashed lines show talks with Q&A via IRC or Etherpad.

Format: 8-min talk followed by live Q&A (done)
Etherpad: https://pad.emacsconf.org/2022-handwritten
Discuss on IRC: #emacsconf-gen
Status: TO_CAPTION_QA

Talk

00:02.200 Introduction
01:41.080 Org Mode
02:29.520 Handwriting
03:42.167 Combining Org Mode and handwriting
03:59.720 Step 1: Write the notes by hand
04:14.420 Step 2: Scan them
04:23.640 Step 3: Store the document
04:42.300 (Optional) Step 4: Convert the notes
05:02.280 Using touch devices
06:30.920 Options

Q&A

00:30.160 How do you link the notes together so that you could search through them in the future?
01:51.649 Is it necessary to OCR your handwriting?
03:59.351 What about searching notes? Notes to text while being offline?
06:00.080 Have you looked at taking handwritten notes on a tablet like Xournal++?
07:32.160 Have you tried out the reMarkable device and figured out how to link the files back into Org mode constructivley yet?
09:26.299 Handwritten and org transcribed notes de-duplication for searching: do you want one or the other, both?
12:01.280 How often do you instead type in and summarize your notes?
15:14.164 How fancy has your handwritten notes import been?
21:58.411 Do you actually have a device of your own that allows you to take notes like this? Or is it just written on paper?
24:36.320 Mindmaps

Description

Research suggests that note taking is most effective when done by hand. Yes, handwritten notes. Certainly, typewritten or typed out notes are more efficient, however notes written by hand are retained by the human brain much longer.

Integrating handwritten notes into a computer leaned productivity workflow is tricky. Also, in terms of hardware, if all you have is a smartphone, then we need to deal with the situation a bit differently.

This talk will introduce to you a simple system to integrate handwritten notes into your org-mode (Emacs) based productivity workflow.

• Gist of the problem of dealing with handwritten notes
• Advantages of handwritten notes
• Emacs org mode workflow
  • Option 1: Hardware for handwriting not available
  • Option 2: Hardware for handwriting available

Discussion

Notes

• Tesseract is a free and open source tool to do OCR (may or may not convert handwriting)
  • https://github.com/tesseract-ocr/tesseract
• Karl Voit: I'm not practicing handwritten OCR at the moment. From what I've read, offline handwriting OCR (offline OCR = not recognizing while writing but form a (scanned) document) is only implemented in a good way in highly proprietary services such as Microsoft OneNote and Evernote and Apple Notes. Since I don't use those services on purpose (https://karl-voit.at/2018/04/21/end-of-OneNote/), I don't have personal experience. I'm still hoping that there will be a FOSS solution one fine day.
  • I'm (rarely) taking notes on my BOOX Note Air e-ink tablet (which I love). It has decent on-device offline-OCR but it's really tedious to use it. So usually, I keep the notes as PDF file and don't care to OCR it. Not great but at the moment I can live with it.
  • After all: I'm much faster when typing on my keyboard (https://karl-voit.at/2021/03/21/advantage2-plans/) - so if there is no graphical information to be captured, I'm faster typing in Org mode than writing by hand.
  • Do you have a blog article about that? I own a BOOX Air myself but I can't think of a viable workflow using it with handwriting + OCR on-device.
• https://capture2text.sourceforge.net/ -- another potential OCR solution that is FOSS, but supported only on Windows OS
• mathpix seems to support OCR now as well:
  • https://mathpix.com/blog/mathpix-text-ocr
  • https://mathpix.com/handwriting-recognition
• Leo Vivier owns an e-ink tablet: Boox Max Lumi (almost A4-size, bigger brother of Boox Note Air)

• Digital pens and digitizing platforms help to get your notes from analogue into digital form: 

  • https://www.wired.com/gallery/best-smart-pens/
  • Also, livescript uses special pens and tablets (erasable and correctable), transcribes your handwriting

• LiveScribe

• I always love the way Bala records his talks

• i read every word
• Apple notes scanning is working very well these days.
• OneNote is super locked-in these days.
• Thank you Prof!
• Lol @ just type out the damn note XD. It's a good point, but I have over a year of handwritten notes that I haven't bothered typing out
• I have had some success with tesseract and my handwritten notes
• has anyone training such a OCR system to make it more accurate for your handwriting or additional symbols.
• shh I've been using Google Cloud Vision to get the text out of my sketches =)
• wow what a refined video. really solid editing. this is enjoyable to watch
• would it be better to use OCR or to read the notes and use speech-to-text?
• Does it work well with free software, klavul
• hmh, there seems to be plugin to integrate orgmode with xournalpp ( https://gitlab.com/vherrmann/org-xournalpp )
• I use capture2text , made a small wrapper
• Would it be possible to make a libre software which could detect handwritten notes as does the Boogie Board and run Emacs on it?
  • other: people with short-term requirements and who don't care are using great OCR with cloud-based lock-in services. Therefore, there isn't much incentive to create FOSS.

Questions and answers

• Q: How do you link the notes together so that you could search through them in the future?
  • A: I'm still in the process of trying to link these topics. Once you convert these notes from handwritten text --or if you have a handwriting device and you can convert it into text-- import it into org-roam and link it the way that you'd normally do it. I started a week ago linking these texts. That would be my response-- which leads into the next question...
• Q: Is it necessary to OCR your handwriting or change the way you write to use your workflow? 
  • A: Yes, this is a necessary step. You have to conduct optical character recognition. Because otherwise, indexing and linking becomes a problem later on. If it's just for you and reading your own text and you browse it and read it when you want, then that's different. You don't have to worry about the OCR-- just import the jpeg or png or whatever and put it under the org-mode or org-roam headline and you should be good to go. However, if you want to search and link then OCR becomes necessary. I'm unfamiliar with anything inside the emacs ecosystem that does OCR. I use tools like OneNote and I think Google Keep does that. I use an external tool to convert if you want linking and indexing. For me sometimes screenshots and handwritten notes have to go together. The key is to have those together.
• Q: What about Searching notes, notes to text while being offline
  • A: Yes, there are no two ways about it. Only proprietary software like Google Keep and Microsoft, perhaps Dropbox does it. I would say it's absolutely mandatory for you to convert this... unless the emacs org mode community puts their heads together. If you want me to help by reviewing a package I am keen on doing that and trying it out.
    OneNote can search with OCR, as far as Bala is aware.
• Q: Those articles on notetaking seem interesting. Could we get a link for them?
  • A: I will definitely drop a link in the etherpad after the talk to make it available. A good friend in France shared this with me. He had a whole bunch of articles that supported my thesis. (an email was sent with a follow-up request on 10 December)
• Q: Have you looked at taking handwritten notes on a tablet like Xournal++

  -   A: [Not yet] The only tools I've used have been OneNote and Google keep. For now, the dropbox method for me is to write the notes, then take a photo with a mobile phone camera, upload it to dropbox-- why dropbox-- because it allows jpeg to be uploaded. I find pdf very cumbersome. I then import it into OneNote which converts it into digitized text. Then I take and move it from the inbox into the org mode system. That's how my workflow is. I have started to experiment with other devices and workflows, but not enough to give you intelligent advice.
  

  Q: Have you tried out the "Remarkable" device and figured out how to link the files back into org mode constructively yet?

  -   A: Yes, I've heard of that device as well as Book and Amazon has come out with the writing Kindle. These devices do exist, but I'm not sure if they convert to text and put it in a format you can import into org mode. This would be nice, if you can integrate it into emacs. I use dropbox to do all the good stuff. I can certainly do that.
  

• Q: Something to think about is handwritten and org transcribed notes de-duplication for searching, do you want one or the other, both?
  • A: Thank you for asking this. Transcription has become very important not only transcribing written but voice notes. In spite of so many tools out there, I find it extremely cumbersome to transcribe voice notes. It would be nice if we had voice, and handwritten transcription helps. Actually that's a great idea. I would be very interested in voice and handwriting. I've seen this in software in OneNote and transcription packages. This is really important and a great idea. I volunteer to test this out. Right now I find it very difficult to transcribe, so if this can do it for us that would be amazing.
  • side note: org-remark could be useful for this https://github.com/nobiot/org-remark
• Q: How fancy have your imports of handwritten notes been for org mode? (Bullets, TODO , org tables etc...)
  • A: Bala uses a star on paper when writing tasks and makes them into org tasks during manual transcription. He also uses dates for this. I know then that I need to keep track of those things in a task. It would be really cool to OCR org constructs from paper. TODO items etc. Doing away with all of the intermediate steps would be nice. A writing org-mode as if it was a human language instead of something that is restricted to a computer.
  • Just to glorify handwriting. I decided that I would hand write what I was going to talk about. The handwritten notes turned out much better than my plain vanilla talk. I even wrote some jokes this way. Handwriting helps thinking and speaking with an audience. Computers have led to handwriting being less important, but Bala thinks they're very efficient and important
• Q: How often do you instead use keyboard entry to type in and summarize your notes? Would you consider that a suitable approach for yourself, e.g. at the end of the day?
  • A: When in a hurry, Bala types notes directly into Org Mode.
  • He has not yet established a habit of daily journaling but would like to do so in the future. I have a shortcut for org-roam dailies that shows up in blue text on the calendar. Daily writing would be helpful.
  • Handwriting can be easier depending on language/script. Perhaps handwriting is best for non-Roman script. My mother tongue has a script that is not English.
• Q: Do you have any ideas about mindmapping and incorporating those notes into your process? I've used nebo and other tools, but wish I had a better way of extracting key words.
  • A: I have used mindmapping which is hierarchical, in fact I use something called function maps. Just like org-roam and interacting relationships. All of this I've been drawing on my laptop screen. All of this helps me. I don't digitally convert it. It helps me and is a lot easier. When I need to add to it I can do so. I find that doing it on a device and retaining it without conversion is best. Converting does not help me at all. YOu can select it, with a lasso select. Those things can be done. I like it to be that way. Then I link it into org mode.
• Q: Bala, sorry to join late to the party, you might have answer. Do you train your own OCR?

Transcript

Can you recognize this building?

[00:00:08.080] Some of you may have recognized this. This building is called the John Hancock building. This is in Chicago. I recently bought this building. Isn't it nice? Heavens no! I am in my home in Pune in India. I am Bala Ramadurai, an author, professor, and an innovation coach. Hello and welcome to my talk on “How to incorporate handwritten notes into Emacs Org Mode” Now, why did I show you this building? The name is of interest for this talk. In the US, someone's signature is also referred to as their Hancock. Your handwriting is pretty much part of your identity. It is as fundamental as that. Of course, there is a movie by that name too, Hancock. I could have started with a clip from that movie, but in spite of Will Smith and Charlize Theron, both my favorite movie stars in the movie, I hated it. [Clip from Hancock (2008)] But I digress. Handwriting has been a fascinating topic for me.

[00:01:44.200] However, note-taking has always been on my computer, in particular, in the Emacs Org Mode system. It is so easy to note down anything, add a schedule, add a deadline, search anything you want, link anything you want, export it to any format, track what you've been doing, clock your tasks, and on and on and on. It's such a squeaky-clean system to track everything and link it to anything from the digital world. Sharing the original notes is still a pain in the rear, but for personal stuff, it's awesome. But then what about handwriting?

[00:02:32.640] Research seems to suggest that handwritten notes can enhance clarity of thought, retention, sometimes even getting rid of your worries.

[Fried, C. B. (2008). In-class laptop use and its effects on student learning]

[00:03:45.680] the efficient Org Mode system with the effective handwritten note-taking system? Merge the systems together. Absolutely. How do you do that? Step 1: Write the notes by hand

[00:04:03.680] on a notebook. Pen, pencil. Keep some convention for yourselves for tracking tasks like a star or an asterisk. Step 2: Scan them using

[00:04:16.400] your favourite mobile app. I recommend Adobe Scan or Dropbox. Step 3: store the document as a JPG file

[00:04:30.000] into a folder called Inbox. Make sure this syncs into a cloud storage folder and your Org Mode system has access to this folder. (Optional) Step 4: convert the notes into text

[00:04:45.920] using Google Keep or just type the damn thing one more time. Then you can process that in your Org Mode system as you always do in whatever is there in your inbox. But, three steps or four before I get access

[00:05:06.120] to my notes and into my Org Mode? What a precious waste of time. I'd rather be tinkering with my config file in that time, correct? Boy... Get or buy or gift or convince your partner, parent, or anybody else to gift yourself a touch-enabled large device. Then use an app like OneNote to write notes on the device and link the handwritten note directly into Org Mode by copying the link. You can use a shortcut like Ctrl-1 to mark the todos, but that means it remains only on OneNote ecosystem, the todos. You can use the same app to also convert the handwritten note into text just by the click of a button. As a bonus, you can even include screenshots from your online meetings.

[00:06:30.920] I like both my options. Option 1: use a regular notebook, scan and process them into my inbox. Option 2: write the notes in a digital device and convert them into text. Or, third option, I appeal to thee, oh great community, can you please build a package inside Org Mode that recognises handwriting to export it into our Org Mode, Emacs Org Mode. Thank you so much for your attention. Bye.

Captioner: jai

Questions or comments? Please e-mail bala@balaramadurai.net

<a href="../../../CategoryOrgMode/">CategoryOrgMode</a>