Integration of DTP3 with Bookends

I’m not complaining. I’ve only asked two or three questions and quickly received helpful responses at my level
I also read other peoples discussions in the hope of learning something, and that’s often too advanced for me. It’s usually helpful, though, and I eventually learn something by osmosis.

That’s what I meant with “you can always ask”. I’m sure nobody will kill you if you ask for an explanation or a link that explains something. Worst case would be that you get no answer, I think :slight_smile:

Thank you. I won’t be shy.

1 Like

As JPC said, DT3 for Historians is what it says. I looked at it a while back and my opinion is that if you are a historian and are setting up a new system to handle archival material, it is well worth investigating. It would have saved me a lot of hassle had it been available ten years ago when I began to use DTP and a reference manager to handle the thousands of documents I was accumulating. I thought the value of DTP for historians lay not so much in the macros (I cannot comment on how similar they are to the inbuilt ones) but in the guidance it provides on how to organise archival material using different apps. Had I been starting from scratch, I would have seriously considered it. However, I decided that, for me, the cost of switching from my existing, inferior but workable, system was too high.

I echo the view of several people here that following the forum is a great way to learn what DT3 can do, even though, like others, it makes me realise that I just don’t use its capability as fully as I could. But I am still sufficiently hooked to be dependent on it!

same issue - I want to see the pdfs of my BE folder that is on iCloud - DTTG doesn’t access.
Why?

I have a similar problem. I have used the Import References from Bookends function in Devonthink, and I have indexed my Bookends folder where my PDFs live. I’d like an easy way to link the two Devonthink records that refer to the same Bookends reference to each other. The easiest way I can see is the Wiki Linking function, if it were possible to include the Bookends attachment PDF’s name in the record created by the Import References from Bookends function. (It would be only one way, but I could add a link the other way manually when the need arises).

Is that possible? Or is the name of the PDF hiding somewhere already? I tried making a manual link from the bibliographical record in Devonthink to the PDF record, but when ran the Import References from Bookends function again to update things, as a test, it overwrote the original record and the inserted link was gone.

Any help would be welcome.

Welcome @Zyblorg

Importing references from Bookends isn’t an updating process between the two apps. It will reimport the data from Bookends.

Why are you rerunning the import?

Thanks for responding.

I’m re-running the import, because my Bookends references are an ongoing research repository. When I add references to Bookends, or in some cases modify references, I’d like that to be reflected in the Devonthink records.

What I have discovered is that the custom metadata that I have attached to a given reference is preserved if I reimport the reference, so that solves most of my problem.

The whole problem would be solved if I could figure out exactly how the import process works. What I want to include in the import is the name of the attachment for a given reference. That way, the wikilinks function will make an automatic link to the Deveonthink record of the attachment pdf in the indexed Bookends attachment folder. I can include the attachment name in the Summary in Bookends, and I know the import function makes use of the Summary in some manner. But it doesn’t seem to make use of all of it, and in particular it doesn’t include the attachment name.

I could write an Applescript that would put the name of the attachment in the Notes field in Bookends, which is included in the import. But unfortunately my knowledge of Applescript is too rudimentary to do that.

Here is a teaching edition script that adds attachment names to the note for items selected in Bookends…

tell application "Bookends"
	if ((count library windows) = 0) then
		display alert "Open a Bookends library."
		return
	else
		set selectedItems to (selected publication items of library window 1)
		if selectedItems = {} then
			display alert "Please select an item"
			return
		else
			repeat with thisReference in selectedItems -- Process the selected items
				
				set theAttachments to (attachment items of thisReference) -- Check if there are attachments
				if theAttachments ≠ {} then -- If so, process them
					
					set referenceNote to (notes of thisReference) -- Cache any existing note content
					
					repeat with theAttachment in (theAttachments) -- Process the attachments
						
						set attachmentName to (name of theAttachment) -- Get the attachment's name
						
						if referenceNote does not contain attachmentName then -- Check if the name is in the existing note already
							set notes of thisReference to (referenceNote & linefeed & attachmentName & "  Notes") -- Add the name to the notes, preserving existing notes
						end if
						
					end repeat
					
				end if
				
			end repeat
			
		end if
	end if
end tell

… and yes, it’s double-spaced to aid reading the comments.

Thanks Bluefrog. I was actually able to figure out a script that did what I want. It’s at the end of this post. Yours might work better (and thank you! for posting it), but what I have did the job for me.

I made a post on the Bookends forums with the Bookends/Devonthink/Highlights app workflow, once I figured it out (Bookends Devonthink Highlights app workflow - Sonny Software). I’ll post it here too in case it’s of any use to anyone:

—>

I thought I would share my workflow with Bookends, Devonthink, and the Highlights app. The aim is to use Devonthink as a sort of ‘hub’ to store my notes on journal articles and books, and to be able to do powerful searches through my Bookends reference data, including the attachments’ text.

You will end up with three linked records in Devonthink, for each Bookends reference:

(1) One with the bibliographical information imported from Bookends for a given reference
(2) One with the pdf attachment for that reference
(3) One with any annotations you have made to the pdf attachment for that reference

Part 1: Bookends – put ‘attachment name’ and ‘annotations record name’ into notes field

The key to linking the three records in Devonthink for each reference in Bookends is to put the name of the attachment record and the name of the annotations record (which you will make later) into the notes field of your Bookends reference. When you import (or re-import) the reference into Devonthink, these two names will be in the rtf section of the record, because the import includes the notes field.

I have a script at the end of this post to put those two names in the notes field. I name my annotations record “[pdf name] Notes”. Obviously, you can use whatever convention you want, but modify the Applescript below to reflect that change.

(Note that if you already have your Bookends references in Devonthink, you can still do this, as long as you haven’t added stuff to the rtf main ‘window’, so to speak. You would have to reimport your references, and when you re-import, the rtf is rewritten, so that added stuff will be lost. However, any custom metadata you added to the reference will be preserved. For this reason, it’s best to set up in Bookends what you want to go in that Devonthink rtf, rather than fiddling with it later. And in any case, most people will want to do reimports from time to time, as they add new reference to their Bookends library).

One you have the record of type (1) in Devonthink, those two names you added to the notes field will act as automatic links to any Devonthink record with the same names, if you have WikiLinks turned on. So turn it on, if you haven’t already.

Part 2: Devonthink – index your Bookends attachments folder; add custom metadata fields

In Devonthink, perform an Import/References from Bookends and choose all references (or whichever subset you want). This creates the references of type (1). (Note that if you are good with Applescript and want to write your own instead of using Devonthink’s own, or find another script does a better import, you can import more metadata into your record (1) than Devonthink currently does. Devonthink doesn’t get it all. Just make sure it includes the notes field for the autolinking to work).

Then also index your Bookends attachment folder. This creates the references of type (2). Don’t add the pdfs to Devonthink, just index the folder. The pdfs are parsed by Devonthink, so they are searchable by Devonthink (assuming you’ve OCRd any that need it). The Devonthink record type (1) with the bibliographical information automatically has a link to the pdf record of type (2), because the pdf record’s name is in the rtf of record (1). And because Devonthink detects links and puts them in the ‘links’ tab of the Document tab of the Inspector, the link is effectively two-way.

In Devonthink I have also made custom metadata for my articles and books. I’ve found the best place to fill all that out is in the bibliographical record of type (1), because its name is the most informative. So I can make smart groups based on the metadata, and the items show up as author and title, rather than just lastname and year.

Part 3: Highlights app – markup your PDF and export the annotations to Devonthink

Finally, when I mark up a PDF I use the Highlights app, because it has a good integration with Devonthink. I use yellow highlights for things I notice while reading but don’t want to save. I use orange highlights for things I want to quote into an annotation summary. I use orange anchored notes for my comments about those quotes or about the text in general. Then I sort the annotations by colour, putting orange first. Next, I export the annotations to Devonthink to produce a record of type (3). (The export to Devonthink directly requires a pro subscription to Highlights). I convert the record (3) to rtf format and change its name from the pdf name to “[pdf name] Notes”.

So now Devonthink record type (3) is my annotations of the pdf, and record (1) automatically links to it, just as it automatically links to record (2), because I’ve put its name there already when I ran the applescript on my Bookends references.

The result:

– Bookends keeps all the references with their bibliographical data.
– I can markup the PDFs in Highlight, and my annotations stay in the PDF itself, so are accessible in any PDF reader
– The PDFs themselves stay in the Bookends pdf folder, so are available to any program, in the cloud, and any modifications are made to the original file
– Devonthink has (1) a record of the bibliographical reference (to which I have added a lot of metadata) so I can do more powerful/tailored searches than I can in Bookends
– Devonthink has (2) a record of the pdf (which I also OCRd), so searches also take the text of the article or book into account
– As I work through the sources and make annotations, Devonthink has (3) a record that contains my own notes on the article or book
– And all three Devonthink records are linked together, and are linked back to the Bookends reference through (1) the record with the bibliographical information

The Applescript:

I am a total neophyte with Applescript, so if someone would like to take a look at this and suggest a more elegant way to do the same, please feel free. Also, please try this with only a few of your references first. It worked with my library, but I’d hate for you to mess up yours, just in case there is something in the script that I don’t understand that would affect your references but not mine.

The script works on references that you have selected rather than on the whole library. Obviously you can select the whole library if you wish.

The script modifies the ‘notes’ field. It sets the notes field to whatever was already in it, plus the name of the attachment of the reference minus the .pdf extension (if there is one), plus the name of the attachment followed by the word Notes.

For example, I have a record whose notes field was:


1942
Nicholas P. White.
23 cm.
Includes indexes.

After running the script it is:


1942
Nicholas P. White.
23 cm.
Includes indexes.

White 1976

White 1976 Notes

Anyway, here is the script. Please use caution in using it, as I am a total neophyte in scripting. [Or use Bluefrog’s script from the post just above, which only puts the name of the attachment in the Notes field, not also the name of the Attachment + Notes]

(And if you are total neophyte like me: Open Script editor, paste this in, save it. Then select a reference from Bookends. Go back to Script Editor and press the ‘play’ icon. Check to see if it worked).

tell application "Bookends"
set theIDs to «event ToySRUID» "Selection"
repeat with theID in paragraphs of theIDs
tell front library window
try
set myRefs to (publication items whose id is theID)
set myItem to first item of myRefs
set thePath to path of attachment items of myItem
set theName to attachments of myItem
set theName to characters 1 thru -5 of theName as string
set theName2 to theName & " Notes"
set theNotes to notes of myItem
if thePath is not {} then
set notes of myItem to theNotes & return & return & theName & return & return & theName2
end if

on error errorMessage

end try

end tell
end repeat
end tell
1 Like

You’re welcome :slight_smile:

As an improvement, I suggest my script - LOL :stuck_out_tongue: :wink:

Actually, what version of Bookends are you running?
You’re using outdated terminology with the set theIDs to «event ToySRUID» "Selection" stuff.

I’m using the latest version of Bookends, but the script is a modification of one that was made a long time ago that I found in a forum. I really didn’t understand the ToySRUID stuff, so I didn’t mess with it, because it worked as is.

Seriously, if you could modify your script to add “[attachment name] Notes” as the second thing put into the Notes record, I would be grateful, and would replace the one I have in the post with that one (and give credit).

Why are you getting characters 1 thru 5 ?

I amended my script above to use the attachment’s name and “Notes”.

In my script, without that constraint, the name of the pdf includes the .pdf extension at the end, and because Devonthink doesn’t include that when indexing the Bookends folder of pdfs, the Wikilinks won’t work. So that bit tells it to ignore the final four characters. Again, I found this online in a forum on how to get rid of the last bit of a string. I assume it essentially says to keep all the string up to the 5th last character.

Yes, that’s a fine AppleScript method to truncate a file’s extension.

Here is the post from above with the explanation, but using the better/newer script that Bluefrog posted, with a few of my own modifications. Thank you for that! The new script is at the end of the post.

I made a post on the Bookends forums with the Bookends/Devonthink/Highlights app workflow, once I figured it out (Bookends Devonthink Highlights app workflow - Sonny Software). I’ll post it here too in case it’s of any use to anyone:

—>

I thought I would share my workflow with Bookends, Devonthink, and the Highlights app. The aim is to use Devonthink as a sort of ‘hub’ to store my notes on journal articles and books, and to be able to do powerful searches through my Bookends reference data, including the attachments’ text.

You will end up with three linked records in Devonthink, for each Bookends reference:

(1) One with the bibliographical information imported from Bookends for a given reference
(2) One with the pdf attachment for that reference
(3) One with any annotations you have made to the pdf attachment for that reference

Part 1: Bookends – put ‘attachment name’, and ‘annotations record name’, and ‘temporary citation’ into notes field

The key to linking the three records in Devonthink for each reference in Bookends is to put the name of the attachment record and the name of the annotations record (which you will make later) into the notes field of your Bookends reference. When you import (or re-import) the reference into Devonthink, these two names will be in the rtf section of the record, because the import includes the notes field.

I have a script at the end of this post to put those two names in the notes field. I name my annotations record “[pdf name] Annotations”. Obviously, you can use whatever convention you want, but modify the Applescript below to reflect that change.

For good measure, I have included as well the Bookends temporary citation format so that, when it gets into Devonthink you can cut and paste this into your document without having to go back to Bookends (note that this format doesn’t have to be exact, it just has to have enough for Bookends to recognise it in a scan – so I’ve included author, publication year, and reference id).

(Note that if you already have your Bookends references in Devonthink, you can still do this, as long as you haven’t added stuff to the rtf main ‘window’, so to speak. You would have to reimport your references, and when you re-import, the rtf is rewritten, so that added stuff will be lost. However, any custom metadata you added to the reference will be preserved. For this reason, it’s best to set up in Bookends what you want to go in that Devonthink rtf, rather than fiddling with it later. And in any case, most people will want to do reimports from time to time, as they add new reference to their Bookends library).

One you have the record of type (1) in Devonthink, those two names you added to the notes field will act as automatic links to any Devonthink record with the same names, if you have WikiLinks turned on. So turn it on, if you haven’t already.

Part 2: Devonthink – index your Bookends attachments folder; add custom metadata fields

In Devonthink, perform an Import/References from Bookends and choose all references (or whichever subset you want). This creates the references of type (1). (Note that if you are good with Applescript and want to write your own instead of using Devonthink’s own, or find another script does a better import, you can import more metadata into your record (1) than Devonthink currently does. Devonthink doesn’t get it all. Just make sure it includes the notes field for the autolinking to work).

Then also index your Bookends attachment folder. This creates the references of type (2). Don’t add the pdfs to Devonthink, just index the folder. The pdfs are parsed by Devonthink, so they are searchable by Devonthink (assuming you’ve OCRd any that need it). The Devonthink record type (1) with the bibliographical information automatically has a link to the pdf record of type (2), because the pdf record’s name is in the rtf of record (1). And because Devonthink detects links and puts them in the ‘links’ tab of the Document tab of the Inspector, the link is effectively two-way.

In Devonthink I have also made custom metadata for my articles and books. I’ve found the best place to fill all that out is in the bibliographical record of type (1), because its name is the most informative. So I can make smart groups based on the metadata, and the items show up as author and title, rather than just lastname and year.

Part 3: Highlights app – markup your PDF and export the annotations to Devonthink

Finally, when I mark up a PDF I use the Highlights app, because it has a good integration with Devonthink. I use yellow highlights for things I notice while reading but don’t want to save. I use orange highlights for things I want to quote into an annotation summary. I use orange anchored notes for my comments about those quotes or about the text in general. Then I sort the annotations by colour, putting orange first. Next, I export the annotations to Devonthink to produce a record of type (3). (The export to Devonthink directly requires a pro subscription to Highlights). I convert the record (3) to rtf format and change its name from the pdf name to “[pdf name] Annotations”.

So now Devonthink record type (3) is my annotations of the pdf, and record (1) automatically links to it, just as it automatically links to record (2), because I’ve put its name there already when I ran the applescript on my Bookends references.

The result:

– Bookends keeps all the references with their bibliographical data.
– I can markup the PDFs in Highlight, and my annotations stay in the PDF itself, so are accessible in any PDF reader
– The PDFs themselves stay in the Bookends pdf folder, so are available to any program, in the cloud, and any modifications are made to the original file
– Devonthink has (1) a record of the bibliographical reference (to which I have added a lot of metadata) so I can do more powerful/tailored searches than I can in Bookends
– Devonthink has (2) a record of the pdf (which I also OCRd), so searches also take the text of the article or book into account
– As I work through the sources and make annotations, Devonthink has (3) a record that contains my own notes on the article or book
– And all three Devonthink records are linked together, and are linked back to the Bookends reference through (1) the record with the bibliographical information
– Devonthink record (1) also has a copy of the temporary citation, in case I want paste that directly into my document

The Applescript:

This script is based on the one provided by Bluefrog, which is much better than the original one I made myself. I’ve modified it to do separate checks for each of the three things that are to go into the Notes field, and only add those that are missing.

The script works on references that you have selected rather than on the whole library. Obviously you can select the whole library if you wish.

The script modifies the ‘notes’ field. It sets the notes field to whatever was already in it, plus the name of the attachment of the reference minus the .pdf extension (if there is one), plus the name of the attachment followed by the word Annotations.

For example, I have a record whose notes field was

1942
Nicholas P. White.
23 cm.
Includes indexes.

After running the script it is:

1942
Nicholas P. White.
23 cm.
Includes indexes.

White 1976

White 1976 Annotations

{White, F. C., 1976, #54213}

(And if you are total neophyte like me: Open Script editor, paste this in, save it. Then select a reference from Bookends. Go back to Script Editor and press the ‘play’ icon. Check to see if it worked).

tell application "Bookends"
	if ((count library windows) = 0) then
		display alert "Open a Bookends library."
		return
	else
		set selectedItems to (selected publication items of library window 1)
		if selectedItems = {} then
			display alert "Please select an item"
			return
		else
			repeat with thisReference in selectedItems -- Process the selected items
				set theAttachments to (attachment items of thisReference) -- Check if there are attachments
				if theAttachments ≠ {} then -- If so, process them
					set referenceCitation to ("{" & author names of thisReference & ", " & publication date string of thisReference & ", #" & id of thisReference & "}") -- Get the reference's citekey
					repeat with theAttachment in (theAttachments) -- Process the attachments
						set attachmentName to (name of theAttachment) -- Get the attachment's name
						set attachmentName to characters 1 thru -5 of attachmentName as string -- strip off the .pdf extension
						set referenceNote to (notes of thisReference) -- Cache any existing note content
						if referenceNote does not contain (attachmentName & linefeed) then -- Check if the attachment name is in the existing note already -- the linefeed check is to avoid it being triggered by the Annotations name, which includes the attachment name
							set notes of thisReference to (referenceNote & linefeed & linefeed & attachmentName) -- Add the attachment name to the notes, preserving existing notes
						end if
						set referenceNote to (notes of thisReference) -- Cache any existing note content
						if referenceNote does not contain (attachmentName & " Annotations") then -- Check if the Annotations name is in the existing note already
							set notes of thisReference to (referenceNote & linefeed & linefeed & attachmentName & " Annotations") -- Add the Annotations name to the annotations, preserving existing notes
						end if
						set referenceNote to (notes of thisReference) -- Cache any existing note content
						if referenceNote does not contain referenceCitation then -- Check if the Citation is in the existing note already
							set notes of thisReference to (referenceNote & linefeed & linefeed & referenceCitation) -- Add the Citation to the notes, preserving existing notes
						end if
					end repeat
				end if
			end repeat
		end if
	end if
end tell
2 Likes

Any messages in the DT log window?

Probably not, as the script is intended to work with Bookends. But then you’d likely be better off asking in their support channels.