A new tool for Zotero users

mhucka · December 6, 2020, 1:18am

When using Zotero for reference management, you may on occasion want to work with the PDF files from outside of Zotero. For example, if you’re a DEVONthink user, you will at some point discover the power of indexing your local Zotero database from DEVONthink. However, when viewing or manipulating the PDF files from outside of Zotero, you may run into the following problem: when looking at a given PDF file, how do you find out which Zotero entry it belongs to?

I developed Zowie (a loose acronym for “Zotero link writer”, and pronounced like the interjection), an open-source program to help deal with this problem. Zowie scans through the files in a local Zotero database, looks up the Zotero bibliographic record corresponding to each PDF file found, and writes a Zotero select link into the PDF file and/or certain macOS Finder/Spotlight metadata fields (depending on the user’s choice). A Zotero select link has the form zotero://select/... and when opened on macOS, causes the Zotero desktop application to open that item in your database. Zowie thus makes it possible to go from a PDF file opened in an application other than Zotero (e.g., DEVONthink, Adobe Acrobat), to the Zotero record corresponding to that PDF file.

In DEVONthink, I index the folder containing my local Zotero database of PDF files. Using Zowie to write the Zotero select link into the Finder comments of every PDF file means that within DEVONthink, the Zotero link will appear in the Annotations & Reminders inspector, in the Finder Comments box. For example:

The nice thing about having the Zotero select link accessible from DEVONthink is that you can create smart rules to manipulate it further. For example, I use the following smart rule to copy the Zotero select link from the Finder Comments to the “URL” field of each PDF file indexed in DEVONthink:

where the following is the simple embedded script:

Zowie is written in Python. A ready-to-run executable is available for macOS 10.13–10.14; for macOS versions after 10.14, Zowie needs the user to have a Python 3 environment on their computer. Zowie can be downloaded from the project home on GitHub, and is also is available from PyPI for installation via pip.

As with all such open-source user-contributed software, there are no guarantees about safety, functionality, or support. I hope it works for you, but I can’t promise that something terrible won’t happen to any of your files, or your computer.

Update 2020-12-29: Zowie version 1.1 now processes all files it finds, not just PDF files (because people put more than just PDF files in their Zotero databases). There is a new flag, -f, that lets you control which files it acts on. In addition, a self-contained ready-to-run binary is now available for macOS versions 10.13 (High Sierra) and 10.14 (Mojave) – no Python needed! (The binary may or may not run on 10.15 or later, depending on your macOS security settings. I hope to work around that limitation eventually.)

rmschne · December 6, 2020, 7:42am

Happy to give this a try as coincidently, my wife (from son’s input) is interested now in Zotero for her next round of academic paper-writing, and so me (local user support) needs to be interested also. It might be there, but I don’t notice a link to get the code? Github or something?

mhucka · December 6, 2020, 3:17pm

Of all the things to forget … yes, you’re right – I forgot to provide a link :-). Will fix right now.

rpallred · January 10, 2021, 2:06am

I had missed this part of the zowie work flow–this smart rule is groundbreaking for me. Thanks!

mhucka · January 10, 2021, 5:06am

For people who are using the Smart Rule above: if you don’t have a scheme for running Zowie on your Zotero files when you add new items to your Zotero database, then new items won’t trigger the Smart Rule simply because they won’t have the Zotero link in the Finder comments. One way to solve this is to use another Smart Rule in DEVONthink that is triggered when new items are imported, and if they lack the Zotero select link, run Zowie on them.

Here’s the Smart Rule I use for that:

and here’s the embedded AppleScript used in the rule:

on performSmartRule(selectedRecords)
	repeat with _record in selectedRecords
		set p to the path of the _record
		set result to do shell script "/usr/local/bin/zowie -q '" & p & "'"
		if result is not equal to "" then
			display notification result
		end if
	end repeat
end performSmartRule

Make sure to update the path to Zowie in the script above if it is not /usr/local/bin/zowie.

The latest version of this script will be available on GitHub.

rpallred · January 10, 2021, 6:24pm

I’m still getting used to the power of smart rules… I had set up a Hazel rule to do something similar:

mhucka · January 10, 2021, 6:55pm

Oh, that’s a good idea too!

lutefish · January 11, 2021, 7:05pm

This looks great, though I have a question about how it handles Zotero libraries that link to files stored elsewhere? Everything in my Zotero library is stored on cloud-synced Box.com finder folders, which are then indexed in DT. As there are no PDF files in the ~/Zotero/storage folders, only .zotero-ft-cache and .zotero-ft-info files, how does the script handle this?

rpallred · January 11, 2021, 7:39pm

So I’m not familiar with how Box.com works, but if it has a Finder location, you should be just fine. If I’m reading the this help article correctly, simply point zowie to the correct location:

zowie -f pdf /Users/GWashington/Box/FolderWhereIKeepTheFiles

That’s actually how the Hazel rule works–Hazel is watching the entire folder and all subfolders. When it detects a match (say, because I just added a new journal article to by Zotero database), it passes the full URL of that PDF in the variable $1 to zowie, which then runs just on that one file (it would be really inefficient to do it this way if it’s the first time ever that zowie has run on a folder!).

lutefish · January 11, 2021, 7:58pm

Thanks. I’m hitting some other errors with Zowie (tuple index out of range), but can take it up on GitHub.

npfet · January 26, 2021, 5:13pm

A quick question on DT <-> Zotero integration: Is there a way for DT to ignore the subfolder layout when indexing Zotero’s storage folder, and to ignore file types that aren’t desired (e.g. only index PDF’s)?

If I index the storage folder directly, I get the Zotero link identifier folders and lots of .js files stored as part of online web abstracts. I could use ZotFile, but that has some additional complexity with storing attachments outside of the storage directory, since then I will not be able to sync with my work laptop.

rpallred · January 26, 2021, 5:28pm

I am wondering the same thing–I created a smart group of my Zotero PDFs as a stopgap.

mhucka · January 27, 2021, 3:29am

That is a good question! To my knowledge, there is no way to directly control DEVONthink’s indexing mechanism and get it to ignore certain file types. However, one can resort to using a smart folder. Here’s how I do it, as an example to get you started.

I have a top-level folder in my database called Sources. Within that, I have a folder called Zotero and within that, I put the actual indexed folder. (The reason for this indirection is purely aesthetic: I index the storage folder inside my Zotero database folder, since that’s where Zotero stores PDFs, but I find the name storage confusing in the context of the rest of my folder structure in DEVONthink. The indexed folder cannot be renamed in DEVONthink because it matches whatever’s on disk, so the only option left is to create another folder in DEVONthink and call that one whatever I want.)

WIthin the folder Sources, I put a smart folder that searches inside the storage folder. Here’s its definition:

I’m not convinced this particular smart folder definition is ideal; for one thing, if I ever put another file type into my Zotero database, I have to remember to update the smart folder definition to include it. But it’s good enough for now, and it avoids all those css, js, and other files that may end up in a Zotero folder. It produces a very clean list of articles:

Hope this helps!

BLUEFROG · January 27, 2021, 6:36am

No you can’t control DEVONthink’s indexing and exclude items from being indexed.

npfet · January 28, 2021, 4:15am

Thanks! Good to understand how you have that setup.

mhucka · June 26, 2021, 6:06pm

For anyone using this approach, I wanted to alert you that I recently updated the embedded AppleScript mentioned above in order to deal with some problems with special characters in file names. The latest version is available on GitHub and here it is for convenience:

# The following function is based on code posted by user "mb21" on
# 2016-06-26 at https://stackoverflow.com/a/38042023/743730

on substituted(search_string, replacement_string, this_text)
	set AppleScript's text item delimiters to the search_string
	set the item_list to every text item of this_text
	set AppleScript's text item delimiters to the replacement_string
	set this_text to the item_list as string
	set AppleScript's text item delimiters to ""
	return this_text
end substituted

on performSmartRule(selectedRecords)
	repeat with _record in selectedRecords
		# In my environment, Zotero takes time to upload a newly-added
		# PDF to the cloud. The following delay is needed to give time
		# for the upload to take place, so that when Zowie runs and
		# queries Zotero via the network API, the data will be there.
		delay 30

		set raw_path to the path of the _record

		# Some chars in file names are problematic due to having special
		# meanings in shell command strings.  Need to quote them with 2
		# blackslashes, b/c the 1st backslash will be removed when the
		# shell command string is handed to the shell.
		set sanitized_path to substituted("&", "\\\\&", raw_path)

		# Another problem with file names is embedded single quotes. The
		# combination of changing the text delimiter and using the
		# AppleScript "quoted form of" below, seems to do the trick.
		set AppleScript's text item delimiters to "\\\\"
		set result to do shell script ¬
			"/usr/local/bin/zowie -q " & (quoted form of sanitized_path)

		# Display a DEVONthink notification if an error occurred.
		if result is not equal to "" then
			display notification result
		end if
	end repeat
end performSmartRule

NationalInterest · September 24, 2021, 2:58pm

Thanks for this application! Works great (once I downgraded sidetrack as you suggested on Github). I’m trying to make Zotero my central source of wisdom for information sources I find, be it websites, articles or books. This lets it be so!

BLUEFROG · September 24, 2021, 3:20pm

Welcome @NationalInterest

I’m sure @mhucka appreciates the pat on the back!

mhucka · October 25, 2021, 1:06am

In case anyone is interested, I’ve released a new version of Zowie (1.2.0) that fixes a software compatibility issue, adds a new feature that will be of interest to DEVONthink users, has faster startup time, and comes with single-file self-contained runnable binaries for macOS 10.15 and higher. See https://mhucka.github.io/zowie/.

mhucka · October 29, 2021, 4:35pm

Another note for anyone using Zowie for Zotero: I wrote an explanation of my smart rule scheme in a wiki page associated with the GitHub repository.