Possible to index iCloud-synchronized Apple Books's .epub and .pdfs w/ highlights & notes?

DEVONthink
1

Hi, I would like to use DEVONthink to index my Apple Books app directory, however it’s proving difficult. I’ve searched for a solution here and elsewhere online, but haven’t figured out how, yet. I did find several helpful articles, however I think a recent macOS upgrade may have changed how an iCloud-synced Books directory functions.

I did notice quite a few warnings against storing dtBase files on shared filesystems, but I’m hoping that my use case - indexing a shared folder - is supported. If it is possible, then the main issue that I’m having right now is pointing DEVONthink to the proper folder, as it appears macOS is doing some magic under the hood to make it appear like a standard folder in some contexts.

Additionally, if it’s possible to index these files, I’m really hoping that DEVONthink will also index and display highlights and notes made with the Books app (as it’s currently my primary way of conducting and synchronizing research notes, etc.)

Currently, the directory (ostensibly ~/iCloud Drive/Books) does not appear in the DEVONthink file selector or Finder. Using a file seach, I’m able to navigate directly to .epub and .pdf files within the directory. Using ‘Copy file path’, I get the path:

~/Library/Mobile Documents/iCloud~com~apple~iBooks/Documents

However, if I hit Shift-Cmd-G from DEVONthink’s file requester dialogue and paste that path in, it immediately switches to:

~/Library/Mobile Documents/iCloud~com~apple~iBooks/Documents

Clicking Open after that just drops the requester into the root iCloud Drive folder. If I instead use Terminal to issue the command:

cd ~/Library/Mobile Documents/iCloud~com~apple~iBooks/Documents

then I am able to navigate to the folder with all of the .epub files. I’ve tried giving DEVONthink full drive access, but that does not appear to change the above behavior.

Any ideas how I can get this to work, or is this just not going to work the way I’d like it to? Also will DEVONthink index Apple Books’ highlights and notes?

2

Welcome @jmcmichael

Indexing Apple’s Books is not supported and the way Apple handles the iCloud data in the filesystem inhibits this too.

3

OK, makes sense - the more I’m reading about Books, the less I’m liking the way that Apple has made it difficult to interoperate with other tools.

So now I’m looking around for an alternate to Books. I’m really used to the polished reader experience of Kindle and Apple Books, so I’m looking for a reader with similar user experience plus transparent sync of activity, highlights, notes across a number of mac/iOS devices - and ideally will allow me to use DEVONthink to also point at the same files to perform searching, concordance, of file content and highlights/notes.

I really wanted to like the DEVONthink epub reader but it’s a very different UI from the more skeuomorphic ‘looks kinda like a book page’ Books/Kindle layout and style.

I was glad to see that my highlights that I’ve made in PDFExpert are displayed in DEVONthink’s reader - are there any officially endorsed apps for epub files that work similarly well with the DEVON suite?

4

Then thing you are looking for does not exists in iOS (or even in macOS). The nearest ebook reader to your desire is FBReader (from FBReader.org, not others), that allows you to sync all (even ebooks), but has a “private” file system. Nearest thing is, once you’ve finished with a book, export annotations in HTML, as you can do with iBooks itself.

What I use to do for books that I densely annotate, is generate a PDF with an easy to read page and font size, and put in a folder. Annotate it with any PDF program (even under Windows, Linux, Android…) and have it in PDF with standard annotation format (or flattened once finished). That way I avoid the non-standard ePub/mobi annotation format, that each program has its own.

5

@jmcmichael:

This used to work perfectly for me.

One needs to index the following folder into DT (I also make it searchable in Alfred)

~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books

DT was then able to search inside all these books and show the search terms.

I just did a test and it looks like now only the names of the books are found but not the content, not sure if there was any change on Apple side in the last Catalina, my version is 10.15.4.

image

image
image1154×650 115 KB

Well, at least, this is something.

6

I had run across that directory in researching various ways to reliably reference Books’ library folder. However when I execute ls ~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books, only a single file is listed: Books.plist.

Are you running the latest version of macOS and/or Books? Perhaps this difference is due to a recent OS or app update. I’m on macOS 10.15.5 and Books v2.4. Also I have iCloud synchronization turned on.

I was unable to locate an ebook reader that works with my use case and appears to be supported by a reliable developer. So, I am going to try a kludge where I keep a Dropbox mirror directory of all my ebooks, to be indexed by DEVONthink. Then I think I can write a shell script to find a book title in Books’ directory (which I can reference by path from the command line) by a substring and then open it in Books, which I will call with an Automator script. Hopefully this will give me the ability to open an epub in Books from a list in DEVONthink. If I can figure out how to reliably export highlights, notes etc. from Books in a way that will easily allow me to view it in Books, I’ll be set.

7

Did you get it to work, @jmcmichael ?

8

On Big Sur 11.4 and Books 3.2 after opening that path:
open ~/Library/Mobile\ Documents/iCloud~com~apple~iBooks/Documents/
in Terminal, you can then use the Finder – file/add to sidebar. If clicking on the sidebar link brings you back to the iCloud Drive instead, you can right click it to open in a new Finder tab and you’ll see all of the epubs and pdfs in the Books iCloud. But…as of today, I’ve lost all faith in the skittish Books app. While editing titles, dates, years in the app, I suddenly saw all of my files disappear, over 1000 of them one by one in quick succession. It was as if a hacker came in and grabbed them. I was able to retrieve most of them from an external drive backup but as far as Apple’s iCloud storage…not a trace of them were able to be recovered. Books is also a disaster with audiobooks–they don’t sync to mobile devices via the cloud. (Actually I found a workaround to sync them but the files would not play on iPhone or iPad Books). So…I am basically going to rely on Devonthink for now as a main point of access to epubs, pdf books, and audiobooks. I will index the files stored in iCloud Drive. Certain genres of books I put directly into DT because I wouldn’t need to access them on mobile devices. For my use case, I can implement any of the options DT offers for annotating, etc. If I want a more streamlined desktop reading experience, I can still open the indexed file with Books but I’ll keep the desktop and mobile versions of Books out of the cloud.

Keeping all the original files in iCloud Drive or Google Drive gives the option of playing the audiobooks with a mobile app called Book Player. I’m trying to avoid duplicating the book files in a cloud which would take up twice as much space.