I am working on a branch of code that is failing when processing PDF records with no OCR layer.
That itself I can manage, but in the process of setting up a testbed, I came to question how I was determining that a PDF in fact has no searchable text, and therefore should be OCR’d by my script before proceeding.
I found a reference “PDF with no searchable text” file online, which is attached.
I ran my JXA code checking for r.wordCount() === 0 to initiate the branch of code that I want to work on. To my surprise, instead of returning 0, it returned 66.
The file correctly shows as kind: PDF document not PDF + text in the DEVONthink info panel.
I have combed the forum and can’t find any more robust way to check for the presence of an OCR layer.
I ran the file though a PDF analyzer and I don’t see a searchable text layer and, as you can see, the PDF describes itself as “Non-text-searchable PDF”.
Perhaps Apple’s Vision framework is used here? There’s a global preference with which you can turn that off, iirc.
And what does r.plainText() give you in this case?
Oh, and preview on iPad does indeed allow the selection of text – Apple’s Vision at work
plainText() does contain the text of the PDF and I can manually select text in the DEVONthink window, copy and paste it. However I can’t find text via search.
So the file is in a hybrid state where it is:
a. not searchable and is indicated as such in DEVONthink with a kind of PDF document, but
b. the document is “editable” within DEVONthink because of some magic - Apple Visual framework?
If I do ocr the document, that works as expected and changes the kind to PDF+Text and I can then search it within DEVONthink.
“How do I detect that a record is lacking a text layer for searching so that I can run ocr against it?”
I don’t see an enum type in r.recordType() that differentiates between PDF document and PDF+Text like r.kind() which I could use, but we have been warned not to use r.kind() because of localization issues IIRC.
Just to reiterate, unfortunately I can’t use r.wordCount() or r.plainText() for this anymore with the advent of Apple’s Vision if I am understanding this correctly.
Did you check your preference settings as I suggested before? I can’t access my DT right now, but there’s a setting concerning Vision, something like dynamic text perhaps. Maybe turning that off helps?
Transcribe PDF documents: PDFs without a text layer are automatically processed via the Vision framework.
I just turned that off here, imported your file and checked r.wordCount() – was 0, as it should be.
Text is still selectable in DT, though. But that’s probably the part of Vision that can’t be turned off in DT because it Is Good For You™
Because System Settings > General > Language & Region > Live Text is enabled.
Kind is PDF/PS and Word Count is 0 is the correct criteria for identifying PDF documents needing OCR.
If you have enabled, Settings > Files > Import > Recognize: Make text in PDF documents searchable, it indexes the content of the file via Vision and relates that text to the document. So converting it to plain text uses that indexed text. And if you’re using e.g., a Word Count column in the item list, it shows the words in the indexed text.
Also, that setting is not doing traditional OCR so there’s no text layer added to the document. This means it has no words. It also means, if you send that document to someone else, even yourself, it is not searchable by its text (ignoring the possibility of Live Text on another Apple device).
Thanks for explaining what is happening behind the scenes. I continue to learn, … but I would second the request for a new recordType to identify a PDF that has no text layer and would benefit from ocr.
Since the wordCount being 0 in such cases completely depends on this setting, it has become a much more fragile test than it used to be before Apple Vision.
In the interim, I will disable the setting on both my Macs.
A new record type would break many existing scripts which are not prepared for multiple PDF record types, therefore this is not an option. We might add a new property instead.
If a script checks for type == "PDF document" and wordCount == 0, it fails if the global preference Transcribe PDF documents is set. Therefore, I’d suggest to make the property wordCount reflect the same value as the search term word count does. A new property would still require rewriting scripts. As @pfarelle says:
Only those scripts that need to process this special case. But a new record type would break every script that just wants to process PDF documents in some way (no matter whether OCRed, indexed or not).
True. That’s why I proposed to make wordCount behave as it has previously for PDF: return only words in a text layer, which makes it independent of the Transcribe PDF documents setting.
I don’t remember actively selecting Apple Vision in the DEVONthink settings, so in a sense, this new default behavior broke my script, and presumably a lot more.
Maybe having wordCount only refer to the text layer would be the best solution?
While that’s possible, this would require changes to existing scripts. And doing a search is not quite the same as iterating over a list of selected records to filter out files with(out) a text layer.
I will just pipe up here (without being able to comment on your script) to say that I think word count = 0 is a bad criterion. There was a thread about this in the forum a few years ago that I took part in but I’m on my phone and on a quick search I haven’t found it.
Some PDFs do have a word count, it’s just that for mysterious reasons only the title was OCR’d or something, not the main body. They wouldn’t get picked up by your script but still need to be OCR’d. Some PDFs have a text layer that’s full of characters, not the text that’s in the document, and they also won’t get picked up by your script despite needing OCR.
In the end, I settled on word count <200 as a smart rule. It occasionally picks up a short PDF, but otherwise it’s usually catching PDFs with bad OCR layers.
(Not really my place to ask why you’re scripting this, but just in case: did you know a smart rule can do this for you? No script necessary!)
