Yes
Guess what? Now it compiles on my computer without error.
Something I am doing is going back and forth between it works and does not work - no clue what that may be.
** And even though it compiles with Script Editor, I still get a similar error if I try to compile it with Script Debugger
Do you have both DTPO and DT3 installed? Perhaps the script compiler sometimes switches which app it’s using? Definitely odd!
(I have not attempted to use this script at all, sorry.)
First of all, thank you so much for doing this, this makes a lot of things easier. I have adapted your code to do what you said you wanted to do with dates: I get a timestamp at the beginning of the run, retrieve a lastRun date from the database and then compare the modifiedDate from zotero for every item to the lastRun date and only process the reference if the modifiedDate is younger than the last run.
A couple of notes: First, the timestamps aren’t all that precise. Zotero specifies the last modified date with UTC, and the only method to simply get UTC time in AppleScript I found is to subtract “time to GMT” from the date object. The problem with “time to GMT” is that it will be an hour off if your timezone switches to/from daylight savings time at a different day than GMT. But it’s an hour, so who cares. I also figured I’d build in a 1-week grace period to make sure I don’t miss any entries, so the script processes every entry that has been modified later than 1 week before the last run. That way we do a bit of redundant iterating, but don’t miss anything. Probably could also just be a day but I switch timezones a lot, and just wanted to be sure.
The other thing to mention is that this script is quite vulnerable: if something goes wrong during a (sometimes quite long run), the UUID database will never be written and that might cause duplicates and other problems down the line. I wonder if it’s worth writing it every time a reference is processed, even though that might mean quite a long time for the first run at least.
Anyway, here’s my version of your code with annotations:
use AppleScript version "2.4"
use scripting additions
use framework "Foundation"
-- all the settings should be here
set bibJSONFile to "/Users/lyndon/repo/workflow/zot-export-bbt.json"
--set thePListFile to "/Users/lyndon/repo/workflow/zotero-to-devonthink.plist"
set theTemplateFile to "/Users/lyndon/Library/Application Support/DEVONthink 3/Templates.noindex/Education/Reference LD.md"
set theDTDBFile to "/Users/lyndon/DevonThink/Research.dtBase2"
set theDTLocation to "/Library"
set theModName to "_mod_datetime_zotero.txt"
set theDictionaryName to "_dictionary_zoteroid_uuid.txt"
--
set ca to current application
property NSJSONSerialization : a reference to current application's NSJSONSerialization
property NSJSONWritingPrettyPrinted : a reference to 1
property NSData : a reference to current application's NSData
property NSString : a reference to current application's NSString
-- a helper method to replace any substring with a space
on remove:remove_string fromString:source_string
set s_String to NSString's stringWithString:source_string
set r_String to NSString's stringWithString:remove_string
return s_String's stringByReplacingOccurrencesOfString:r_String withString:" "
end remove:fromString:
set theJSONData to NSData's dataWithContentsOfFile:(bibJSONFile)
set theJSON to NSJSONSerialization's JSONObjectWithData:theJSONData options:0 |error|:(missing value)
set theCurrentTime to (current date) - (time to GMT) -- Zotero uses UTC, so should we.
set theDateModifiedObject to current date -- Quickest way to get date object.
set bibjson to theJSON as record
set therefs to |items| of bibjson
set mainDict to missing value
set theDictRecord to missing value
-- load up the previous data
tell application id "DNtp"
set theDatabase to open database theDTDBFile
set theLocation to create location theDTLocation in theDatabase
-- set mainDict to current application's NSMutableDictionary's dictionaryWithContentsOfFile:thePListFile
if mainDict is missing value then
set mainDict to current application's NSMutableDictionary's new()
end if
tell theLocation
set theDictPath to theDTLocation & "/" & theDictionaryName
set theDictRecord to get record at theDictPath
if theDictRecord is missing value then
set theDictRecord to create record with {name:theDictionaryName, type:"txt"} in theLocation
end if
if theDictRecord is not missing value then
set mainDictContents to plain text of theDictRecord
set mdStr to (ca's NSString's stringWithString:mainDictContents)
set mdDataDict to item 1 of (ca's NSJSONSerialization's JSONObjectWithData:(mdStr's dataUsingEncoding:(ca's NSUTF8StringEncoding)) options:0 |error|:(reference))
mainDict's setDictionary:mdDataDict
end if
end tell
-- check last run of script.
tell theLocation
set theModPath to theDTLocation & "/" & theModName
set theModRecord to get record at theModPath
if theModRecord is missing value then
set theModRecord to create record with {name:theModName, type:"txt"} in theLocation
end if
if theModRecord is not missing value then
set theLastRun to plain text of theModRecord
end if
end tell
-- set theChildGroups to {}
-- try
-- set theChildGroups to children of theLocation
-- end try
-- repeat with theChild in theChildGroups
-- if type of theChild is group then
-- set theZoteroIDString to get custom meta data for "zoteroid" from theChild
-- set theUUID to uuid of theChild
-- (mainDict's setObject:theUUID forKey:theZoteroIDString)
-- end if
-- end repeat
end tell
set limitCounter to 0
repeat with theRef in therefs
set limitCounter to limitCounter + 1
-- if (limitCounter > 20) then exit repeat
-- Seems like Zotero stores modified date in ISO8061 format for UTC time: YYYY-MM-DDTHH:MM:SSZ. That's easy to convert reliably I'd think.
tell theDateModifiedObject to set {its year, its month, its day, its hours, its minutes, its seconds} to {text 1 thru 4, text 6 thru 7, text 9 thru 10, text 12 thru 13, text 15 thru 16, text 18 thru 19} of dateModified of theRef
-- Let's check the last modified date against the last run. Since there could be race conditions, let's give it a one-week grace period, that is, process all references that have been modified later than 1 week before the last run.
if theLastRun is equal to "" or theDateModifiedObject > (date (theLastRun as string)) - 1 * weeks then
set {theKey, theTitle, theURI, theZoteroID, theDateModified} to {citationKey, title, uri, itemID, dateModified} of theRef
set theSelectURI to ""
try
set theSelectURI to |select| of theRef
end try
try
set theTitle to |shortTitle| of theRef
end try
set theZoteroIDString to theZoteroID as string
set theDOI to ""
try
set theDOI to DOI of theRef
end try
set theAbstract to ""
try
set theAbstract to abstractNote of theRef
end try
set theDate to ""
try
set theDate to |date| of theRef
set theReferenceYear to theDate
end try
set theURL to ""
try
set theURL to |url| of theRef
end try
set theCreators to ""
set multipleCreators to ""
try
repeat with theCreator in creators of theRef
try
set theCreators to theCreators & multipleCreators & |firstName| of theCreator & " " & |lastName| of theCreator & " (" & |creatorType| of theCreator & ")"
set multipleCreators to " and "
end try
end repeat
end try
set theTags to {}
try
repeat with theTagItem in tags of theRef
set theTags to theTags & tag of theTagItem
end repeat
end try
-- construct group name and look for a previous group in the dictionary
set theGroupFile to theKey & " " & theTitle
set theGroupFile to (my remove:"/" fromString:theGroupFile)
set theUUID to missing value
set theUUID to (mainDict's objectForKey:theZoteroIDString)
-- create or update the group
if theUUID is missing value then
tell application id "DNtp"
set theGroup to create location theDTLocation & "/" & theGroupFile
set theUUID to uuid of theGroup
end tell
else
set theUUID to theUUID as string
tell application id "DNtp"
set theGroup to get record with uuid theUUID
if theGroup is missing value then set theGroup to create location theDTLocation & "/" & theGroupFile
set the name of theGroup to ("" & theGroupFile)
set theUUID to uuid of theGroup
end tell
end if
-- create or update a summary file
set theSummaryName to ("___" & theGroupFile & ".md") as text
set theSummaryDate to theDate
set theSummaryPlaceholders to {|%reference%|:theTitle, |%authors%|:theCreators, |%date%|:theSummaryDate, |%citation%|:theKey, |%doi%|:theDOI, |%abstract%|:theAbstract, |%zoteroselect%|:theSelectURI}
tell application id "DNtp"
set theTempRecord to import theTemplateFile placeholders theSummaryPlaceholders to theGroup
set thePrevSummaryName to get custom meta data for "referencesummaryfile" from theGroup
if thePrevSummaryName is missing value or thePrevSummaryName = "" then
set theSummaryRecord to get record at (the location of theTempRecord) & theSummaryName
else
set theSummaryRecord to get record at (the location of theTempRecord) & thePrevSummaryName
end if
if theSummaryRecord is missing value then
set the name of theTempRecord to theSummaryName
set theSummaryRecord to theTempRecord
else
set theTempContent to the plain text of theTempRecord
set the plain text of theSummaryRecord to theTempContent
delete record theTempRecord
set the name of theSummaryRecord to theSummaryName
end if
end tell
(mainDict's setObject:theUUID forKey:theZoteroIDString)
tell application id "DNtp"
tell theGroup
set aliases to theKey
set tags to theTags
set URL to theSelectURI
set custom meta data to {referencesummaryfile:theSummaryName, DOI:theDOI, abstract:theAbstract, citekey:theKey, zoteroid:theZoteroID}
end tell
end tell
-- add attachments and bookmarks to the group
if theURL ≠ "" then
tell application id "DNtp"
set theBookmarkRecord to lookup records with URL theURL
if theBookmarkRecord is missing value or (count of theBookmarkRecord) is less than 1 then
create record with {name:theURL, type:bookmark, URL:theURL} in theGroup
end if
end tell
end if
set theAttachmentPath to ""
set attachmentList to {}
try
set attachmentList to attachments of theRef
end try
repeat with theAttachment in attachmentList
set theAttachmentPath to ""
set theAttachmentURI to ""
set theAttachmentURL to ""
set theAttachmentLinkMode to ""
try
set theAttachmentPath to |path| of theAttachment
end try
try
set theAttachmentURI to uri of theAttachment
end try
try
set theAttachmentURL to |url| of theAttachment
end try
try
set theAttachmentLinkMode to linkMode of theAttachment
end try
if theAttachmentPath ≠ "" then
tell application id "DNtp"
tell theGroup
set theAttachmentRecord to lookup records with path theAttachmentPath
if theAttachmentRecord is missing value or (count of theAttachmentRecord) is less than 1 then
set theAttachmentRecord to indicate theAttachmentPath to theGroup
else
set theAttachmentRecord to item 1 of theAttachmentRecord
end if
if theAttachmentRecord is not missing value then
set custom meta data of theAttachmentRecord to {DOI:theDOI, abstract:theAbstract, citekey:theKey, zoteroid:theZoteroID}
end if
end tell
end tell
else if theAttachmentLinkMode = "linked_url" then
tell application id "DNtp"
set theBookmarkRecord to lookup records with URL theAttachmentURL
if theBookmarkRecord is missing value or (count of theBookmarkRecord) is less than 1 then
create record with {name:theAttachmentURL, type:bookmark, URL:theAttachmentURL} in theGroup
end if
end tell
end if
end repeat
end if
end repeat
set theMainDictJSONData to (NSJSONSerialization's dataWithJSONObject:mainDict options:NSJSONWritingPrettyPrinted |error|:(missing value))
set theMainDictJSONString to (ca's NSString's alloc()'s initWithData:theMainDictJSONData encoding:(ca's NSUTF8StringEncoding))
set theMainDictJSONStringAS to (theMainDictJSONString as text)
tell application id "DNtp"
tell theGroup
set plain text of theDictRecord to theMainDictJSONStringAS
-- (mainDict's writeToFile:thePListFile atomically:true)
end tell
tell theGroup
set plain text of theModRecord to theCurrentTime as text
end tell
end tell
1 Like
Sorry for the long delay in replying - yes, I ran into a few little bumps with that so I’ve changed it to write the map out for each item. It slows things down but the main speed issue is iteration over the AppleScript records. I need to figure out a faster iteration method. There are a few articles online about this kind of thing but I haven’t had a moment to sort it all out.
Thank you for the timestamp checking! Much appreciated and I’ll drop that in too.
1 Like
Do you mind sharing the updated version or just the relevant code that does this? That way we can divide our efforts! I have also considered just passing the processing stuff to a script in a slightly more efficient language and only use Applescript to read/write from Devonthink. Not sure I’ll get around to implementing and testing this too soon though.
Here’s an updated version that uses a script object reference for iteration. Gets through my over 2,000 record BibTeX file in about 15 minutes now (vs a few hours before), and includes the date check modification. It has left a duplicate map database between Zotero ID and DT UUID but I can’t figure out what that’s about and I’m going to watch a movie 
use AppleScript version "2.4"
use scripting additions
use framework "Foundation"
-- all the settings should be here
set bibJSONFile to "/Users/lyndon/repo/workflow/zot-export-bbt.json"
--set thePListFile to "/Users/lyndon/repo/workflow/zotero-to-devonthink.plist"
set theTemplateFile to "/Users/lyndon/Library/Application Support/DEVONthink 3/Templates.noindex/Education/Reference LD.md"
set theDTDBFile to "/Users/lyndon/DevonThink/Research.dtBase2"
set theDTLocation to "/Library"
set theModName to "_mod_datetime_zotero.txt"
set theDictionaryName to "_dictionary_zoteroid_uuid.txt"
--
set ca to current application
property NSJSONSerialization : a reference to current application's NSJSONSerialization
property NSJSONWritingPrettyPrinted : a reference to 1
property NSData : a reference to current application's NSData
property NSString : a reference to current application's NSString
script V
property vTheRefs : missing value
end script
-- a helper method to replace any substring with a space
on remove:remove_string fromString:source_string
set s_String to NSString's stringWithString:source_string
set r_String to NSString's stringWithString:remove_string
return s_String's stringByReplacingOccurrencesOfString:r_String withString:" "
end remove:fromString:
set theJSONData to NSData's dataWithContentsOfFile:(bibJSONFile)
set theJSON to NSJSONSerialization's JSONObjectWithData:theJSONData options:0 |error|:(missing value)
set theCurrentTime to (current date) - (time to GMT) -- Zotero uses UTC, so should we.
set theDateModifiedObject to current date -- Quickest way to get date object.
set bibjson to theJSON as record
set therefs to |items| of bibjson
set V's vTheRefs to |items| of bibjson
set therefscount to (count therefs)
set mainDict to missing value
set theDictRecord to missing value
-- load up the previous data
tell application id "DNtp"
set theDatabase to open database theDTDBFile
set theLocation to create location theDTLocation in theDatabase
-- set mainDict to current application's NSMutableDictionary's dictionaryWithContentsOfFile:thePListFile
if mainDict is missing value then
set mainDict to current application's NSMutableDictionary's new()
end if
tell theLocation
set theDictPath to theDTLocation & "/" & theDictionaryName
set theDictRecord to get record at theDictPath
if theDictRecord is missing value then
set theDictRecord to create record with {name:theDictionaryName, type:"txt"} in theLocation
end if
if theDictRecord is not missing value then
set mainDictContents to plain text of theDictRecord
set mdStr to (ca's NSString's stringWithString:mainDictContents)
set mdDataDict to item 1 of (ca's NSJSONSerialization's JSONObjectWithData:(mdStr's dataUsingEncoding:(ca's NSUTF8StringEncoding)) options:0 |error|:(reference))
mainDict's setDictionary:mdDataDict
end if
end tell
-- check last run of script.
tell theLocation
set theModPath to theDTLocation & "/" & theModName
set theModRecord to get record at theModPath
if theModRecord is missing value then
set theModRecord to create record with {name:theModName, type:"txt"} in theLocation
end if
if theModRecord is not missing value then
set theLastRun to plain text of theModRecord
end if
end tell
-- set theChildGroups to {}
-- try
-- set theChildGroups to children of theLocation
-- end try
-- repeat with theChild in theChildGroups
-- if type of theChild is group then
-- set theZoteroIDString to get custom meta data for "zoteroid" from theChild
-- set theUUID to uuid of theChild
-- (mainDict's setObject:theUUID forKey:theZoteroIDString)
-- end if
-- end repeat
end tell
set limitCounter to 0
--repeat with theRef in therefs
repeat with i from 1 to therefscount
set theRef to item i of V's vTheRefs
set limitCounter to limitCounter + 1
-- if (limitCounter > 20) then exit repeat
repeat 1 times -- dummy loop to allow skip to next item on error
-- Seems like Zotero stores modified date in ISO8061 format for UTC time: YYYY-MM-DDTHH:MM:SSZ. That's easy to convert reliably I'd think.
tell theDateModifiedObject to set {its year, its month, its day, its hours, its minutes, its seconds} to {text 1 thru 4, text 6 thru 7, text 9 thru 10, text 12 thru 13, text 15 thru 16, text 18 thru 19} of dateModified of theRef
-- Let's check the last modified date against the last run. Since there could be race conditions, let's give it a one-day grace period, that is, process all references that have been modified later than 1 day before the last run.
if theLastRun is equal to "" or theDateModifiedObject > (date (theLastRun as string)) - 1 * days then
try
set {theKey, theTitle, theURI, theZoteroID, theDateModified} to {citationKey, title, uri, itemID, dateModified} of theRef
on error
exit repeat
end try
set theSelectURI to ""
try
set theSelectURI to |select| of theRef
end try
try
set theTitle to |shortTitle| of theRef
end try
set theZoteroIDString to theZoteroID as string
set theDOI to ""
try
set theDOI to DOI of theRef
end try
set theAbstract to ""
try
set theAbstract to abstractNote of theRef
end try
set theDate to ""
try
set theDate to |date| of theRef
set theReferenceYear to theDate
end try
set theURL to ""
try
set theURL to |url| of theRef
end try
set theCreators to ""
set multipleCreators to ""
try
repeat with theCreator in creators of theRef
try
set theCreators to theCreators & multipleCreators & |firstName| of theCreator & " " & |lastName| of theCreator & " (" & |creatorType| of theCreator & ")"
set multipleCreators to " and "
end try
end repeat
end try
set theTags to {}
try
repeat with theTagItem in tags of theRef
set theTags to theTags & tag of theTagItem
end repeat
end try
-- construct group name and look for a previous group in the dictionary
set theGroupFile to theKey & " " & theTitle
set theGroupFile to (my remove:"/" fromString:theGroupFile)
set theUUID to missing value
set theUUID to (mainDict's objectForKey:theZoteroIDString)
-- create or update the group
if theUUID is missing value then
tell application id "DNtp"
set theGroup to create location theDTLocation & "/" & theGroupFile
set theUUID to uuid of theGroup
end tell
else
set theUUID to theUUID as string
tell application id "DNtp"
set theGroup to get record with uuid theUUID
if theGroup is missing value then set theGroup to create location theDTLocation & "/" & theGroupFile
set the name of theGroup to ("" & theGroupFile)
set theUUID to uuid of theGroup
end tell
end if
set theMainDictJSONData to (NSJSONSerialization's dataWithJSONObject:mainDict options:NSJSONWritingPrettyPrinted |error|:(missing value))
set theMainDictJSONString to (ca's NSString's alloc()'s initWithData:theMainDictJSONData encoding:(ca's NSUTF8StringEncoding))
set theMainDictJSONStringAS to (theMainDictJSONString as text)
tell application id "DNtp"
tell theGroup
set plain text of theDictRecord to theMainDictJSONStringAS
-- (mainDict's writeToFile:thePListFile atomically:true)
end tell
end tell
-- create or update a summary file
set theSummaryName to ("___" & theGroupFile & ".md") as text
set theSummaryDate to theDate
set theSummaryPlaceholders to {|%reference%|:theTitle, |%authors%|:theCreators, |%date%|:theSummaryDate, |%citation%|:theKey, |%doi%|:theDOI, |%abstract%|:theAbstract, |%zoteroselect%|:theSelectURI}
tell application id "DNtp"
set theTempRecord to import theTemplateFile placeholders theSummaryPlaceholders to theGroup
set thePrevSummaryName to get custom meta data for "referencesummaryfile" from theGroup
if thePrevSummaryName is missing value or thePrevSummaryName = "" then
set theSummaryRecord to get record at (the location of theTempRecord) & theSummaryName
else
set theSummaryRecord to get record at (the location of theTempRecord) & thePrevSummaryName
end if
if theSummaryRecord is missing value then
set the name of theTempRecord to theSummaryName
set theSummaryRecord to theTempRecord
else
set theTempContent to the plain text of theTempRecord
set the plain text of theSummaryRecord to theTempContent
delete record theTempRecord
set the name of theSummaryRecord to theSummaryName
end if
end tell
(mainDict's setObject:theUUID forKey:theZoteroIDString)
tell application id "DNtp"
tell theGroup
set aliases to theKey
set tags to theTags
set URL to theSelectURI
set custom meta data to {referencesummaryfile:theSummaryName, DOI:theDOI, abstract:theAbstract, citekey:theKey, zoteroid:theZoteroID}
end tell
end tell
-- add attachments and bookmarks to the group
if theURL ≠ "" then
tell application id "DNtp"
set theBookmarkRecord to lookup records with URL theURL
if theBookmarkRecord is missing value or (count of theBookmarkRecord) is less than 1 then
create record with {name:theURL, type:bookmark, URL:theURL} in theGroup
end if
end tell
end if
set theAttachmentPath to ""
set attachmentList to {}
try
set attachmentList to attachments of theRef
end try
repeat with theAttachment in attachmentList
set theAttachmentPath to ""
set theAttachmentURI to ""
set theAttachmentURL to ""
set theAttachmentLinkMode to ""
try
set theAttachmentPath to |path| of theAttachment
end try
try
set theAttachmentURI to uri of theAttachment
end try
try
set theAttachmentURL to |url| of theAttachment
end try
try
set theAttachmentLinkMode to linkMode of theAttachment
end try
if theAttachmentPath ≠ "" then
tell application id "DNtp"
tell theGroup
set theAttachmentRecord to lookup records with path theAttachmentPath
if theAttachmentRecord is missing value or (count of theAttachmentRecord) is less than 1 then
set theAttachmentRecord to indicate theAttachmentPath to theGroup
else
set theAttachmentRecord to item 1 of theAttachmentRecord
end if
if theAttachmentRecord is not missing value then
set custom meta data of theAttachmentRecord to {DOI:theDOI, abstract:theAbstract, citekey:theKey, zoteroid:theZoteroID}
end if
end tell
end tell
else if theAttachmentLinkMode = "linked_url" then
tell application id "DNtp"
set theBookmarkRecord to lookup records with URL theAttachmentURL
if theBookmarkRecord is missing value or (count of theBookmarkRecord) is less than 1 then
create record with {name:theAttachmentURL, type:bookmark, URL:theAttachmentURL} in theGroup
end if
end tell
end if
end repeat
end if -- end the modification date check
end repeat -- end dummy loop
end repeat
tell application id "DNtp"
tell theGroup
set plain text of theModRecord to theCurrentTime as text
end tell
end tell
1 Like
I’m not sure this isn’t somewhat random BUT I did notice that when I don’t run this script from Script Editor (for example from the DT menu or from Script Debugger), the scope seems to be off. The first thing I noticed was the thing you mentioned, that there were suddenly duplicates of the UUID db file and the last run file. But then I started realizing that some records were replicated in my inbox rather than iterated over in the location specified at the beginning.
What I did to fix that was to remove all instances where you use a construct like “tell theLocation” and instead just adding “in database “Work”” to every get/create record statement (that is, both for the metadata files mentioned above, but also for the group and summaryfile and so on). Suddenly those duplicates in the inbox disappeared, and now it seems to also no longer produce the other problems. I’m happy to post my version of your script if it’s useful for others, although by now I have really departed from some of your initial concepts and expanded quite a few things (but also made it slower again :)), so let me know if you’d rather I don’t clutter up your thread.
I think it’d be great to see your script - these are all a bit custom to the specific setup people have and I think anything is grist for the mill.