Hello! I’m looking forward Devonthink as a companion to or even the complete replacement of Evernote. My Evernote flow includes a lot of attachments and internal links across notes.
The Devonthink’s embedded import routine handled the most of my notebooks well, yet always get stuck after import of a single huge notebook with a lot of notes. So, I’ve managed to import this one by processing of small chunks of ENEX export packages.
Because internal links are the crucial part of my workflow, I use an apple script to embed Evernote internal links in URL filed of every note. Thus, all notes imported to the Devonthink no matter which way already do have their own local Evernote link in their URL field embedded.
Now I’ve got to:
(i) for every note in main database,
(ii) and if it contains any internal links in text (e.g., links referring to path starting with “evernote://etc”),
(iii) find another note with text, that has the same URL in URL field and is of type “formatted note” and copy internal link to it,
(iv) replace the original reference of the link with the new one.
Step (iii) has additional complications, because a note with non-image attachments becomes a group consisting of a text note that has the same name as the group and attached files separately. And every child of such group has the same internal Evernote URL in its URL field. Thus step (iii) must include an additional mechanism that finds the note with text (not an attachment file). On my opinion the easiest way to do this is to test whether a record with correct URL is of type “formatted note” (all Evernote imports with text have this type).
I’m a developer myself but I’m not used to apple script dictionary syntax. I’ve tried to write the script myself, and this is what I’ve got so far:
using terms from application "DEVONthink 3"
tell application "DEVONthink 3"
try
# (i) for every record in db check if record is formatted note
set recs to a reference to contents of current database
repeat with rec1 in recs
if ((type of rec1) as text) is "formatted note" then
log message (name of rec1) as text
# (ii) if record contains links starting with "evernote://"
set chunks to attribute run of rich text of rec1
repeat with chunk in chunks
if (URL of chunk) is not null then # ERROR: Can't get URL of "_my note text without link is here_"
log message (URL of chunk) as text
if URL of chunk starts with "evernote://" then
if exists record with URL (URL of chunk) in current database then
set recs2 to lookup records with URL in current database
# (iii) for every record2 in db if type of record2 is correct
repeat with rec2 in recs2
if type of rec is "formatted note" then
# (iv) check local link of record2 and replace with it the old one
set url2 to URL of record2
if url2 is equal to URL of chunk then
log message ((URL of chunk) as text) & " got to be replaced with " & ((reference URL of rec2) as text)
# set URL of chunk to reference URL of rec2
end if
end if
end repeat
end if
end if
end if
end repeat
end if
end repeat
on error error_message number error_number
if the error_number is not -128 then
display alert "DEVONthink 3" message error_message as warning
end if
end try
end tell
end using terms from
An error occurs at the line “if (URL of chunk) is not null then”. I’ve tried a lot of options instead of null but can’t figure out how to test if a chunk of rich text has an active URL property.
Any help to finish this script it will be highly appreciated!
No time at the moment so just a quick info: You don’t need to loop thru all records.
Instead take a look at the search command in the AppleScript dictionary (in Script Editor.app or Script Debugger.app (which got a superior dictionary explorer)).
search doesn’t use the scope: syntax that’s used in toolbar searches. To search an entire database use in root of [database] (as in the linked script).
Thanks for the posting; I need to follow through and clean up the Evernote links in my data.
The Lookup records with URL works great, but I’m just simply text searching note contents
As you noted, before Evernote export we need to save the note’s internal link
I appended the note to the note contents (actually just the note-id)
(example ID_73a4b241-2198-4072-8401-212759efe87b)
take a look at the search command in the AppleScript dictionary
Thank you! It’s a very nice approach how to get notes that have url property (target notes to link to) instead of my set recs2 to lookup records with URL in current database.
However didn’t found any search prefix or other approach that can help with searching notes, that have urls in text (and futhermore with reference text that begins with “evernote://”). Something similar to what the Links pane in the app displays but as search prefix would be great,
like search "links:evernote".
And the worst is that I am still not able to get (URL of chunk) to be handled correctly, because the most of rich text chunks obviously don’t have embedded urls, so I get “Can’t get URL of ‘My random part of text here’” error.
I’ve attached an example. There are two notes in enex file that can be imported via File > Import > Files and Folders. The first note Sed ut perspiciatis has a link to the second note Lorem ipsum with reference evernote:///view/2199673/s25/438c47ee-8c1e-418f-a2c7-54e84467ace9/438c47ee-8c1e-418f-a2c7-54e84467ace9/. I’m trying to get all such references to be replaced with Devonthink cross-note links. Both imported notes will have text starting with evernote:///view/... embedded in their url metadata field, however only the destination note has evernote:///view/2199673/s25/438c47ee-8c1e-418f-a2c7-54e84467ace9/438c47ee-8c1e-418f-a2c7-54e84467ace9/.
I wonder how developers of the app did this. Because notes imported the proposed way (File > Import > Notes from Evernote) have their links already replaced. But I can’t import one huge notebook with standard means, this is why I need all this AppleScript stuff.
BTW this is my current version of script ATM:
using terms from application "DEVONthink 3"
tell application "DEVONthink 3"
try
# (i) for every formatted note in main db
set db to database "main"
set recs to search "kind:formattednote" in root of db
repeat with rec1 in recs
log message (name of rec1) as text
# (ii) for every link in note pointing towards destination that starts with "evernote://"
set chunks to attribute run of rich text of rec1
repeat with chunk in chunks
set ref1 to (URL of chunk)
if ref1 is not null then # ERROR
if ref1 starts with "evernote://" then
set recs2 to search "kind:formattednote url:" & (ref1 as text) in root of db
# (iii) for every record2 formatted note in db that has the given url in its url field
repeat with rec2 in recs2 # there should be only one record, this line is for debug reasons
# (iv) replace evernote link with the new one
set url2 to URL of record2
if url2 is equal to URL of chunk then
log message (ref1 as text) & " got to be replaced with " & ((reference URL of rec2) as text)
# set (URL of chunk) to (reference URL of rec2)
end if
end repeat
end if
end if
end repeat
end repeat
on error error_message number error_number
if the error_number is not -128 then
display alert "DEVONthink 3" message error_message as warning
end if
end try
end tell
end using terms from
This script replaces evernote:/// link URLs with the corresponding record’s Reference URL.
Note: Do not run this script on all your records at once.
-- Replace evernote:/// Link URLs in Formatted Notes with DEVONthink Reference URLs
-- Note: This script finds a link's corresponding record by searching the link's URL in DEVONthink's URL property.
use AppleScript version "2.4"
use framework "Foundation"
use scripting additions
tell application id "DNtp"
try
set theRecords to selected records whose type = formatted note and URL starts with "evernote:///"
repeat with thisRecord in theRecords
set thisRecord_Source to source of thisRecord
if thisRecord_Source contains "<a href=\"evernote:///view/" then
set theEvernoteURLs to my regexFind(thisRecord_Source, "(?<=<a href=\\\")evernote:///view(.*?)(?=\")")
repeat with thisEvernoteURL in theEvernoteURLs
set thisEvernoteURL to thisEvernoteURL as string
set theResults to search "kind:formattednote url==" & thisEvernoteURL
if theResults ≠ {} then
set thisResult to item 1 of theResults
set thisResult_ReferenceURL to reference URL of thisResult
set thisRecord_Source to my regexReplace(thisRecord_Source, thisEvernoteURL, thisResult_ReferenceURL)
end if
end repeat
set source of thisRecord to thisRecord_Source
end if
end repeat
on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
return
end try
end tell
on regexFind(theText, thePattern)
try
set theString to current application's NSString's stringWithString:theText
set {theExpr, theError} to current application's NSRegularExpression's regularExpressionWithPattern:(thePattern) options:0 |error|:(reference)
set theMatches to theExpr's matchesInString:theString options:0 range:{0, theString's |length|()}
set theResults to {}
repeat with thisMatch in theMatches
set thisMatchRange to (thisMatch's rangeAtIndex:0)
set thisMatchString to (theString's substringWithRange:thisMatchRange) as string
set end of theResults to thisMatchString
end repeat
return theResults
on error error_message number error_number
activate
display alert "Error: Handler \"regexFind\"" message error_message as warning
error number -128
end try
end regexFind
on regexReplace(theText, thePattern, theRepacement)
try
set theString to current application's NSString's stringWithString:theText
set newString to theString's stringByReplacingOccurrencesOfString:(thePattern) withString:(theRepacement) options:(current application's NSRegularExpressionSearch) range:{location:0, |length|:length of theText}
set newText to newString as string
on error error_message number error_number
activate
if the error_number is not -128 then display alert "Error: Handler \"regexReplace\"" message error_message as warning
error number -128
end try
end regexReplace
set theRecords to selected records whose type = formatted note and URL starts with "evernote:///" and source contains "<a href=\"evernote:///view/" leads to DEVONthink 3 got an error: AppleEvent timed out. error.
I’ve got to fall back to code like this
repeat with thisRecord in theRecords
if source of thisRecord contains "<a href=\"evernote:///view/" then
to process one by one.
Also by some unknown and unreported reason (no errors being thrown) this script always stops at processing of a random note far from the end of list (250-600 of 4600).
But this is not so important, because the replacement works, thanks to you!
I hope, I’ll be able go get it working like a charm finally.
Getting selected records filtered by a whose clause is normally blazing fast. You could add a custom timeout:
with timeout of 3600 seconds
end timeout
Strange. No idea what’s going on. However, you could activate NSScriptingDebugLogLevel and see whether you can find something in Console.app. I never really used the console or NSScriptingDebugLogLevel but as you’re a developer you probably know how it works.
-- Activate NSScriptingDebugLogLevel
property theAppIdentifier : "com.devon-technologies.think3"
set theChooseFromListItems to {"Activate Debugging", "Deactivate Debugging"}
set theChoice to choose from list theChooseFromListItems with prompt "" default items (item 1 of theChooseFromListItems) with title "NSScriptingDebugLogLevel"
if theChoice is false then return
if item 1 of theChoice = item 1 of theChooseFromListItems then
set theMode to "1"
else if item 1 of theChoice = item 2 of theChooseFromListItems then
set theMode to "0"
end if
do shell script "defaults write" & space & theAppIdentifier & space & "NSScriptingDebugLogLevel " & theMode
Maybe I’m doing something wrong? I select my group (folder) first and then write kind:formattednote scope:selection in the search bar. I have formatted note condition removed from the code of the first selection query to decrease its execution time. After all I press Cmd + A to select all filtered notes.
Btw a large timeout lets query finish yet results in another very strange thing. Query returns only 200-300 notes of ~4300. By my fairy accurate approximation at least 3300 of them do have URLs in Evernote’s internal format.
I have not tried NSScriptingDebugLogLevel yet. Maybe it can provide more information on what is happening…
No idea. Perhaps not all Evernote link URLs start with evernote:///view ?
It’s hard to tell what might go wrong as I’ve never used Evernote and only tested with two files.
@BLUEFROG, @Idify uses a modified version of this script. It should find all records that got a link with a Evernote URL but seems to only find a part of the records. Any idea why?
No. Here’s the numbers atm:
Total cound of items in goup, including subgroups: 4285 kind:formattednote scope:selection leads to 4260 items - the exactly count of Evernote notes in this group. kind:formattednote scope:selection url:evernote doesn’t do anything (same 4260 items selected) since all imported notes have url starting with evernote:///view/ in URL field — it’s result of my another script that copies local link for every selected note in Evernote and pastes it into the URL field (so I’m guaranteed links will be kept after export-import to other app like Devonthink).
Today I did something new. Split this huge notebook (group) into parts and finally made to import all notebooks with File > Import > Notes from Evernote. Many urls got already replaced by this feature, but many of them are not.
I applied script to every group consisting of 150-400 records each. And the same thing happens script stops accidentally at some random item. The result of script execution is always true.
Additionally when I run this script for a single item some urls are getting replaced, some are not.
Repeating gives more replacements, so the script can be used many-many times by hand, and than there’s a large probability (almost) all urls are replaced. But to be honest it’s not a kind of result I expect from a script
@BLUEFROG, are you a Devonthink developer? Is there another API to command Devonthink, e.g. Objective-C / Swift / C / Python etc.-compatible? Apple Script has such a messy debug flow, I can’t understand, why the proposed script can’t find and replace all links with “evernote:///” in reference text. Or may be is there a way to reactivate internal relinking routine? To access DEVONthink-Text.dtMetaStore directly (as a db? text file with encoding?) to find and replace text using full power of a compiled language?
I think the reason of all this mess with AppleScript may lay in Apple’s unannounced limitations to what and how long on their wise opinion can be done using scripting. And that doesn’t symply accepts workaround of with timeout of 999999 seconds…
The current version of script:
-- Replace evernote:/// Link URLs in Formatted Notes with DEVONthink Reference URLs
-- Note: This script finds a link's corresponding record by searching the link's URL in DEVONthink's URL property.
-- Search for `kind:formattednote scope:selection` + select folder first
-- Activate NSScriptingDebugLogLevel
property theAppIdentifier : "com.devon-technologies.think3"
set theChooseFromListItems to {"Activate Debugging", "Deactivate Debugging"}
set theChoice to choose from list theChooseFromListItems with prompt "" default items (item 1 of theChooseFromListItems) with title "NSScriptingDebugLogLevel"
if theChoice is false then return
if item 1 of theChoice = item 1 of theChooseFromListItems then
set theMode to "1"
else if item 1 of theChoice = item 2 of theChooseFromListItems then
set theMode to "0"
end if
do shell script "defaults write" & space & theAppIdentifier & space & "NSScriptingDebugLogLevel " & theMode
-- debug block ends
use AppleScript version "2.4"
use framework "Foundation"
use scripting additions
set notesContentToReplace to "<a href=\"evernote:///view/"
set sourceLinkPattern to "(?<=<a href=\\\")evernote:///view(.*?)(?=\")"
with timeout of 60000 seconds
tell application id "DNtp"
try
set theRecords to selected records whose type = formatted note
set note_nr to 1 # DEBUG
set notes_cnt to (count of theRecords) as text # DEBUG
repeat with thisRecord in theRecords
if source of thisRecord contains notesContentToReplace then
log message "========= Processing note " & (note_nr as text) & " of " & notes_cnt & ":: " & (name of thisRecord) as text # DEBUG
set note_nr to note_nr + 1 # DEBUG
set thisRecord_Source to source of thisRecord
set theEvernoteURLs to my regexFind(thisRecord_Source, sourceLinkPattern)
repeat with thisEvernoteURL in theEvernoteURLs
set thisEvernoteURL to thisEvernoteURL as string
set theResults to search "kind:formattednote url==" & thisEvernoteURL
if theResults ≠ {} then
set thisResult to item 1 of theResults
set thisResult_ReferenceURL to reference URL of thisResult
set thisRecord_Source_new to my regexReplace(thisRecord_Source, thisEvernoteURL, thisResult_ReferenceURL)
log message (thisEvernoteURL & " -> " & (name of thisResult) as text) & " => " & thisResult_ReferenceURL # DEBUG
delay 1 # DEBUG async issue?
set source of thisRecord to thisRecord_Source_new
end if
end repeat
else # DEBUG
log message "========= Note " & (note_nr as text) & " of " & notes_cnt & " doesn't seem to have any Evernote urls:: " & (name of thisRecord) as text # DEBUG
end if
end repeat
on error error_message number error_number
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
return
end try
end tell
end timeout
on regexFind(theText, thePattern)
try
set theString to current application's NSString's stringWithString:theText
set {theExpr, theError} to current application's NSRegularExpression's regularExpressionWithPattern:(thePattern) options:0 |error|:(reference)
set theMatches to theExpr's matchesInString:theString options:0 range:{0, theString's |length|()}
set theResults to {}
repeat with thisMatch in theMatches
set thisMatchRange to (thisMatch's rangeAtIndex:0)
set thisMatchString to (theString's substringWithRange:thisMatchRange) as string
set end of theResults to thisMatchString
end repeat
return theResults
on error error_message number error_number
activate
display alert "Error: Handler \"regexFind\"" message error_message as warning
error number -128
end try
end regexFind
on regexReplace(theText, thePattern, theRepacement)
try
set theString to current application's NSString's stringWithString:theText
set newString to theString's stringByReplacingOccurrencesOfString:(thePattern) withString:(theRepacement) options:(current application's NSRegularExpressionSearch) range:{location:0, |length|:length of theText}
set newText to newString as string
on error error_message number error_number
activate
if the error_number is not -128 then display alert "Error: Handler \"regexReplace\"" message error_message as warning
error number -128
end try
end regexReplace
No there is no other API. You could use JXA if you’re inclined but there’s no specific support for it.
Or may be is there a way to reactivate internal relinking routine? To access DEVONthink-Text.dtMetaStore directly (as a db? text file with encoding?) to find and replace text using full power of a compiled language?
No this is not publicly available.
Remove the try block and see where it errors - something I say you should always do when initially developing a script.
JavaScript, of course. I doubt that you can access DT with ObjC withouth using either AppleScript or JavaScript, though. Python – well, you have to find a library that permits calling the scripting methods.
Personally, I’d try JavaScript because it has RegEx support on board and does not require the trip to ObjC.
Some remarks though: Having “kind:formattenote” in a loop that runs over whose type = formatted note is redundant and confusing. Also, using kind is discouraged because it depends on the region settings. You’re using only the first item of thisResult in your replacement step. Is this intended?
You’re thinking of the kind vs. type properties in DEVONthink’s scripting dictionary. Yes, in this context type should be used instead of kind for the reason you mentioned.
However, the script uses kind as literal string in a query (just like it’s used in a toolbar search).
Yes, as for each Evernote URL there can only be one corresponding DEVONthink record.
No worries, It didn’t attracted my attention as well. And thank you for the fix!
Additionally, I found a couple of childish issues with my debug flow too (forgot to count records, that don’t have urls with evernote:/// references, some of notes were having wrong urls in URL field, some links pointed to notes that were already deleted and therefore missing).
By fixing all this, and a big folder split into 8 smaller ones yesterday, today I’ve managed not only to import all Evernote records via the standard Import from Evernote tool but also to have a smooth replacement process as well.
However today also wasn’t my lucky day. I was feeling lazy and selected all 4200+ records to have their links replaced in one run and left my mac. It finished after about 2500 records with a huge (80 Gb!) memory leak that took all of my 32 Gigs of RAM and all possible space left on the system SSD and led to a halt, reboot and Devonthink database corruption and an additional 8 Gb Devonthink log (its my fault too because it seems that I added to much log message debug stuff to the script).
But! The good news are the updated version of script should work fine.
I will reimport everything again and run script for every group of 200-500 records separately tomorrow. This should finally work.