(DT3 Bibliography) Formatted reference from Custom MD

I came to the conclusion that I prefer DT3 over Bookends to manage my bibliography. So I made this script to:

  • Take custom metadata (type (article / book), authors, title, translator, editor, local, publisher, date, journal, issue, volume and pages) to create a formatted reference (to be added to the field reference).
  • Use the reference as record name.
  • Take bibkey and set it as record alias.
  • Convert keywords into tags.

I tried to create a property APA to trigger a different formatting for the reference (right now it is formatted as ABNT), but I could not get it to work yet. As it is, I doubt this could be useful to anyone right out of the box, but perhaps it could be helpful as starting point. Once I have updated versions of it, I will post them here.

UPDATED (July 12, 2019):
The script is now on GitHub to make it easier for me to keep updating it.


@Bernardo_V Do you have a screenshot of this script showing sample output?

I’m very interested in seeing how you develop this project. I’ve thinking about how to shift some of my workflow from Zotero to DT to minimize switching programs during writing.
On a related topic: I wanted to have DT index my notes contained within Zotero, but I couldn’t seem to find a way without actually dragging the note from Zotero into DT. Is Bookends the same?

1 Like

Export from Zotero as a CSV, then import that into DT3.

@rkaplan I agree that exporting as CSV is also another option. I should have been more specific in what I’m looking for - which is a way to get DT to read the data from Zotero notes so that I don’t have to drag and drop the file or import the CSV data manually every time I make some kind of modification to the Zotero database.
As I understand it, when files are indexed by DT the data will dynamically update or the user can requested the Update Indexed Files, which saves having to do drag-and-drop or export from one application to the other.

I’m interested in @Bernardo_V solution because it seems like a way to skip the reference manager altogether and create bibliographic data and annotations directly in DT. But it was difficult for me to tell exactly what is going on without the screenshot.

Of course! I should have posted it with the script.

In this cap you can see the custom metadata and the reference generated with the script. I also decided to use the reference as the record name.

Here you’ll see that the keywords became tags and that the bibkey became an alias.

The idea is to have more than one reference formatting option, but I am still to implement this.

I actually don’t have many notes on bookends. They are all on their respective pdfs as comments so they all show up on DT3 without any problems.

It would be terrific if it were possible to use DT as a Zotero replacement. But I think each app has its strengths and it would take way too much to duplicate those strengths in the other app.

Zotero’s tremendous strength lies in its translators and their ability to collect metadata from both raw PDF files and from sources all over the web, as well as its ablity to create a bibliography from that data in virtually any academic format desired. DT’s tremendous strength is the capability and speed of its search and inference engines in locating a specific item.

Finding a way to get these two to work together would be terrific; I don’t think either can replace the other in an academic workflow.

@Bernardo_V This is fantastic!! I now see clearly what the output of the script action is. This is a very good way to generate bibliographic data directly in DT.

What kind of setup/modification needs to take place in DT so that this script will run?

@rkaplan I totally agree with your comments here. This is exactly why I am currently implementing a two program+ solution.

With all of this enthusiasm around the release of DT3 I have been re-examining my workflows - trying to think different or at least more efficiently. I was always so impressed with DT2 how I could find new ways to leverage the program to expand or modify my workflow. DT3 offers many new avenues of exploration.
Perhaps it’s still a dream to have all of the Zotero work folded into DT3, but when I saw this topic, I had renewed hope!

The script uses the current selection, instead it should use the “theRecords” parameter provided by the smart rule. Beta 4 will also include a new “get custom meta data for … from … default value …” command to make retrieveal of certain custom metadata values much easier. Then your script could look like this:

on performSmartRule(theItems)
	tell application id "DNtp"
			repeat with this_item in theItems
				-- Supported by "Download Bibliographic Metadata" script
				set the_doi to get custom meta data for "doi" from this_item default value ""
				set the_date to get custom meta data for "date" from this_item
				set the_type to get custom meta data for "type" from this_item default value ""
				set the_authors to get custom meta data for "authors" from this_item default value ""
				set the_authors to my replaceText(the_authors, return, ", ")
				set the_publisher to get custom meta data for "publisher" from this_item default value ""
				set the_page to get custom meta data for "page" from this_item default value ""
				set the_journal to get custom meta data for "journal" from this_item default value ""
				set the_volume to get custom meta data for "volume" from this_item default value ""
				set the_issue to get custom meta data for "issue" from this_item default value ""
				-- Not (yet) supported by the "Download Bibliographic Metadata" script, has to be defined by user
				set the_bibkey to get custom meta data for "bibkey" from this_item default value ""
				set the_title to get custom meta data for "title" from this_item default value ""
				set the_editor to get custom meta data for "editor" from this_item default value ""
				set the_translator to get custom meta data for "translator" from this_item default value ""
				set the_place to get custom meta data for "place" from this_item default value ""
				-- Create reference & rename item
				set the_ref to ""
				if the_volume is not "" and (the_type begins with "Article" or the_type begins with "article") then
					set ref_volume to " (" & the_volume & ")"
					set the_ref to the_authors & "; " & the_title & ". In: " & the_journal & " " & the_issue & ref_volume & ", " & the_date & "." & " Pp." & the_page
				else if (the_translator is not "" and the_editor is not "") and (the_type begins with "Book" or the_type begins with "book") then
					set ref_translator to "Trad.: " & the_translator & ". "
					set ref_editor to "Ed.: " & the_editor & ". "
					set the_ref to the_authors & "; " & the_title & ". " & ref_editor & ref_translator & the_place & ": " & the_publisher & ", " & the_date
				end if
				if the_ref is not "" then
					add custom meta data the_ref for "reference" to this_item
					set name of this_item to the_ref as text
				end if
				-- Set aliases & tags
				set the aliases of this_item to the_bibkey
				set the_keywords to get custom meta data for "keywords" from this_item default value {}
				if the_keywords is not "" then
					set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, ","}
					set theTags to text items of the_keywords
					set AppleScript's text item delimiters to od
					set theTags to (parents of this_item) & theTags
					set the tags of this_item to theTags
				end if
			end repeat
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		end try
	end tell
end performSmartRule

on replaceText(theString, find, replace)
	if theString contains find then
		local od
		set od to text item delimiters of AppleScript
		set text item delimiters of AppleScript to find
		set theString to text items of theString
		set text item delimiters of AppleScript to replace
		set theString to "" & theString
		set text item delimiters of AppleScript to od
	end if
	return theString
end replaceText

Thinking about this a bit more…

Zotero stores its data in SQLite

Is it possible to write an Applescript script which reads the Zotero SQLite database and imports it to Devonthink?

At least theoretically by using shell scripts inside the AppleScript. But it sounds like a huge amount of work :slight_smile:

1 Like

You don’t want to read the SQLite. For one, the database is heavily normalized and it’s a bit of a pain to use – searches for “a reference with this author and that in the title” span at least for tables. For two, Zotero cannot be running when you read the database, because Zotero takes an exclusive lock on the database.

Zotero however has two things that make it fairly easy to get at its data (much more so than any other reference manager in existence), provided it is running: it has a very extensive plugin system, and has a built-in web server. It’s very easy to create a HTTP-JSON API to reach into Zotero. I’ve done it frequently, and if you don’t need anything to abstract away the Zotero internals for convenience, there’s one ready-to-go at https:// bit . ly / 2Sdg567 (I’m not allowed to include links in a post - super annoying). Using such a system you can reach into Zotero to grab data, but also for example to render bibliographies on your behalf. You can script Zotero almost in any way using this.

As for generating bibliographies – this is something that’s easy to underestimate, but Zotero (and Mendeley and a few others, who use the same engine that Zotero does) can format your items in 9397 different bibliographic formats at current count (with a lot of expertise behind it – who among us actually has the APA or Chicago manual, much less understands all their intricacies?) without any effort on your part besides choosing one of them. If you’ve ever had to take an existing article and publish it at a different outlet, you know how much of a life-saver this can be. I reckon DT can replace Zotero as a bibliography manager about as well as Zotero can replace DT as a research notebook.


Thanks retorquere

Your link is here:

Agreed that DT and Zotero each have their strengths -which is why a link between them has long been a holy grail discussed on this group.

A working example of a DT script to import Zotero data would be quite helpful.

If @Bernardo_V can pull JSON from an HTTP service and do something sensible with it, I can provide the HTTP-JSON service.

Can you elaborate on that? You mean that we would be able to get the reference in JSON format via download JSON from command?

It can do pretty much anything you can do by hand in Zotero. The simplest case would be to export a fixed collection into CSL-JSON format, but you could for example have multiple endpoints, e.g:

  • one that lists all groups
  • one that lists all collections within a group
  • one that gets all items from a group in any format (although I’d recommend CSL-JSON)
  • one that gets all items from a collection
  • one that renders selected items into a bibliography (which would be a little more work, but it’s possible)
  • one that pops up the Zotero citation picker as you would in Word and gives you back the items the user picked
  • … pretty much you name it. If it can be done in Zotero, it can almost always be done using scripting, because by Zotero’s nature, all the internals used to drive the Zotero UI are exposed to scripters.

The only hard requirement is that it speaks HTTP. I picked JSON to be spoken on top of that because you need a way to structure the data being sent back and forth, and JSON is the lingua franca for that these days.

As a simple example, if you use debug-bridge to send this

const DT = Zotero.Collections.getByLibrary(Zotero.Libraries.userLibraryID).find(coll => coll.name == 'DevonThink')
return DT && DT.getChildItems().map(item => Zotero.Utilities.itemToCSLJSON(item))

you get the items in the collection named DevonThink as CSL-JSON.

I confess I was thinking of something else, namely, the possibility of downloading BibTex references from the web directly in JSON.

If the reference is already in Zotero, why not simply use Applescript to get the data into variables and then apply it to the DT3 record?

You tell me – I don’t know anything about applescript, but AFAIK, Zotero does not support applescript, so you’ll need something like this bridge to reach inside it.

WRT downloading bibtex refs from the web – Zotero currently has some 500 scrapers to deal with the variety of sites out there. That’s not easy to replicate unless you principally just have a few sites you take refs from, and they have publicly-scrapable bibtex.

In any case, I have no skin in this game – if your way works for you, I’m not arguing you should change. I was told there was interest in coupling Zotero and DT, and I can offer expertise for the Zotero side of things. I don’t know DT at all well, and I don’t know applescript at all.