Indexing handwritten notes in DEVONthink

DEVONthink
1

I just posted this as a response under DTTG, but realize that it probably belongs here. The question at hand is how to bring the wonderful world of handwritten notes in DEVONthink. It is possible that I am behind the curve here, and others have found better solutions. If so, I very much look forward to hearing about them. Here is the post:

I’m not sure where this issue is at the moment, and since I’m still in the process of building a new information infrastructure for my work (following Evernote’s decision a while ago to drop scripting support) I can’t swear as to how well it will work out. But with those caveats aside…

I use Notability for taking handwritten notes. Notability has a feature that allows you to automatically back up PDFs of the latest copy of all of your notes to DropBox. The new copy of a note overwrites the previous copy. So if you index that folder into DEVONthink you automatically have PDFs of all of your handwritten notes.

Again, I am not certain how well this is going to work in practice, but the tests I’ve run so far work well. There are some things that I’ve not played with:

  • How does OCR work in this case? Is it lost when notes upgrade? My guess is, “of course it is,” but perhaps you can tell me what to expect.
  • I am assuming that as long as the name of the DropBox folder is the same on all machines that notes will be indexed and accessible for every machine sharing that database. I have not played with indexing in DTTG, but anticipate it will fail there. Although past handwritten notes can easily be imported rather than just indexed.

Another solution would be to index the .note files themselves. Since Notability has a desktop app, that would be beautiful. The .note files are stored on iCloud Drive. There are two problems, however. The first is that you can access the relevant folder by going to System Preferences → iCloud → Manage Storage → Notability → Open in Finder. When you do so the folder opens and claims to live under iCloud Drive. But when you look at iCloud Drive, no Notability folder appears, even as a hidden file. Hard to index that.

The other problem is that while Notability is perfectly happy to read in one of those files as a NEW note, what it will not do is simply open the existing note in the app. Notability desperately needs a custom URL scheme, preferably with deep linking.

As much as I hate to say it, I may have to start using OneNote instead of Notability. (Argh). OneNote DOES support deep linking, and these links can be accessed and opened locally from inside DEVONthink:

  • Inside OneNote, copy a note or paragraph within a note as a link.
  • The link that you get will be to the web location of the note. When linking from DEVONthink (or any other app), preface the web URL with onenote:// (i.e., enter the URL as onenote://https://onedrive.live.com/view… )
  • When you open the link, specify “Open link in MicroSoft Onenote”.

Bingo. You have the current version of the deep linked location in the note sitting in front of you in OneNote, ready to go. All of the above can of course be easily automated.

I truly loathe the idea of moving my note taking to OneNote for a number of reasons. Apart from the fact that it is MicroSoft, they have made the kind of absurd decision not to support OCR on the Mac version. On the other hand, if I need OCR I can always save it as a PDF and do the OCR there. I’m wondering if there might also be a way to use the Macs native OCR capability to grab images in OneNote, do the OCR, and write it out.

Another huge problem with OneNote is that it does a poor job importing PDFs for markup.

But while I would prefer not to use OneNote, I absolutely cannot live without being able to access research materials and notes with URLs. I’m trying to make myself believe that accessing Notability’s PDFs will do the trick, but I may not succeed. I may wind up going with Notability for marking up PDFs, but using OneNote for just handwritten notes. Ugly…

My world would be WILDLY better if LiquidText had a custom deep linking URL scheme.

Relevant to all of this, I wonder if people have seen the Manifesto for Ubiquitous Linking that appeared recently, spearheaded by Angel Vu (PDFpen) together with a note of names from a bunch of different companies and universities.

2 Likes
2

I use GoodNotes this way - their backup synced to my Devonthink. The OCR works, in so far that OCR on my awful handwriting ever works. No deep linking, however. But it’s good enough for me.

1 Like
3

ABBYY OCR doesn’t support handwritten text, and if you OCR your hand written notes it will generate a new PDF with an inaccurate text layer. As Notability supports handwriting recognition I would assume that the exported PDF would already contains a text layer so no further OCR would be necessary for its contents to be searchable in DEVONthink. If you provide a sample of an export PDF I could check.

4

@aedwards: Thanks for the reply. Notability does a remarkably good job with handwriting recognition when you explicitly tell it to within the app. Notability clearly uses stroke information as well as straight OCR, and can usually read handwriting that it frankly has no business being able to read. It takes it a noticeable amount of time to do the conversion. You can then convert the block of handwriting itself into text, or copy the converted text to the clipboard.

When sitting and laying down with iPad propped on knee, it is most comfortable for me to write text that cuts across the page at an angle. I can select that handwriting, rotate the selection, and then convert to text and bingo. I have a little script on the Mac side that takes the handwriting and removes the extraneous newlines when you copy the text and paste it back in, so the results are quite nice.

But that is only when you select a section and tell it to do the conversion. The standard OCR in the files that it saves, which is searchable in DEVONthink, is not as high quality.

That is useful and important, but back to the original point, even if the OCR were perfect what you clearly want to be able to do is reference and automatically open the note so that it is ready to work on.

Even this is not real deep linking. The idea of deep linking is to be able to say “open this document to this point on this page.” The utility of that is obvious, and is becoming more common. In html you do it just by using “#” in an anchor. As more and more applications provide custom URL schemes allowing deep linking I think it likely this will simply become an expected feature.

2 Likes
5

A workaround for deep linking into an archived PDF is to embed a unique bit of text, say “link_01.” Then use devonthink://…?search=link_01 to open the file and search for that marker.

But again, here we are talking about PDF tricks. While these are well and good they still fall far short of being able to link directly into a note.

6

As more and more applications provide custom URL schemes allowing deep linking I think it likely this will simply become an expected feature.

You are still referring to hacks or workarounds here.

And also, instead of an increasing number of URL schemes, a reduction and standardization would be a much better and more preferable option.

2 Likes
7

@BLUEFROG: I know that I am still referring to hacks and workarounds. That is my frustration!

I am impressed by what DEVONthink does. The deeper that I get into it the more useful it is becoming – assuming of course that AppleScript doesn’t go away any time soon. I am impressed enough that I am seeing how nice a (highly idiosyncratic) PKM I can build around those capabilities.

BUT… and it is a big but. In the present context, there exists no good overall solution allowing me to work with documents in as powerful and fluid a way as should be possible given available technology.

DEVONthink’s PDF markup is OK as far as it goes, but it falls far short of the active reading capabilities of LiquidText. Notability handles handwritten notes and markups very well, including both OCR and information on the strokes used to generate handwriting. It even does a passable if not great job using OCR to turn handwriting into searchable text. But like LiquidText it lacks any deep linking. OneNote can handle deep linking, but is not a very good app in many other respects. I could continue for some time. Having tried out a very long list of note taking options, none bring it all together.

An absolutely killer solution in DEVONthink would be to have the active reading capabilities of LiquidText, with that sort of linking available throughout documents in a DEVONthink database. Couple that with a workspace that has the feel and functionality of Notability (including that level of handwriting to text conversion) and it would make a qualitative change in how people are able to work with documents.

Eventually someone will implement a better mousetrap along those lines, and the world will beat a path to their door. It would be great if that someone turned out to be DEVONthink.

But in the mean time there is functionality that I need. Since that functionality is not available in a straightforward fashion, I will do what I can to get it by hook or by crook. At the moment that means trying to piece together hacks and workarounds that let me interact with diverse information in the ways that I need to, as efficiently as possible.

As I play with schemes to minimize the friction of moving around among various tools to take advantage of what they offer, I can imagine that there might be people on forums such as this who would find some of those hacks and workarounds useful. I hope that such discussions might also give DT an opportunity for a fresh look at use cases. PKM has been crucial to my professional life for over 40 years, and I am enjoying seeing the recent explosion of thinking about these problems. But I’m just as happy that I’m not the one saddled with writing and maintaining packages that try to ride that wave! The one thing that I think we can say with some degree of certainty is that a few years from now PKM will look very different from what it looks like today. And as is always the case, out of a growing crowd of players a few will emerge as dominant in that field. I’m hoping that DT emerges a leader in what is to come. From where I sit you have a very good head start. But you have your work cut out for you.

I don’t entirely understand the comment about URL schemes, though. One of the things that makes DT so powerful is its diversity of ways to link information. That includes presenting the world with a custom URL scheme and the sort of API needed to make it dance to the tunes that individual users need it to. And since from the outside custom URL schemes are reasonably consistent, apps like DT readily make use of the schemes offered by others.

Anyway… I am a fan, but a fan with a lot of experience and a big picture view. I should also say that while I’m not a software developer per se, I have written a few hundred thousand lines of code in my life. I have a great deal of appreciation for the effort that goes into a package like DEVONthink, and also appreciate your willingness to engage.

4 Likes
8

+1 for this feature. It looks like Notability runs on MyScript which can be integrated into other apps.

9

fwiw I’m a fan of using Notability for handwriting
When completed, I export the note in pdf format for storage in Devonthink
The pdf is ocr’d by Notability; the text is indexed for searching in Devonthink

4 Likes
10

I don’t know if you have tried Zoomnotes, but it is a pdf notes app like Notability that allows deep linking. It does not have a pretty UI or a shallow learning curve, but the ability to link pages of pdfs with the app as well as other apps is unbeatable.

11

Welcome @Yazmsw

Interesting app. Thanks for sharing :slight_smile:

12

I‘m also very interested in this topic. Currently I‘m experimenting with several apps, like Notability and Nebo on the iPad. I like Nebo very much, mostly due its great handwriting recognition and converting features. The app is published by MyScript, so I guess they know the magic. Still it’s missing even rudimentary encryption features, so I’m hesitating to put all my thoughts into this great app.

DEVONthink is much better concerning security, file-handling and search, but the cooperation between DEVONthink and Nebo/Notability is less than optimal: DEVONthink cannot read Nebo nor Notability files. I could use PDF files instead, but each round trip duplicates the corresponding file.

I’m also hoping for an integration of MyScript technology into DEVONthink. Bringing both technologies together would be a great benefit for me.
Links between documents would be the cherry on the top.

13

They should use an open file format or document their own one. Proprietary, closed stuff makes everything harder for the users.

2 Likes
14

I’ve started archiving handwritten notes from GoodNotes. (This is a recent change to my workflow.) I’m not talking about notes you might want to re-edit at a later date, just stuff that needs retaining and filing. If you export the notes as a pdf from GoodNotes the OCR is good (don’t re-run OCR in DT though - the GoodNotes OCR is much better at handwriting recognition).

For me, it moves the notes to a secure location (my database), and means I can search their content in my database whenever needed.

2 Likes