Trying to relink all URLs after import from Evernote's enex files with Apple Script

Hello! I’m looking forward Devonthink as a companion to or even the complete replacement of Evernote. My Evernote flow includes a lot of attachments and internal links across notes.
The Devonthink’s embedded import routine handled the most of my notebooks well, yet always get stuck after import of a single huge notebook with a lot of notes. So, I’ve managed to import this one by processing of small chunks of ENEX export packages.
Because internal links are the crucial part of my workflow, I use an apple script to embed Evernote internal links in URL filed of every note. Thus, all notes imported to the Devonthink no matter which way already do have their own local Evernote link in their URL field embedded.
Now I’ve got to:
(i) for every note in main database,
(ii) and if it contains any internal links in text (e.g., links referring to path starting with “evernote://etc”),
(iii) find another note with text, that has the same URL in URL field and is of type “formatted note” and copy internal link to it,
(iv) replace the original reference of the link with the new one.

Step (iii) has additional complications, because a note with non-image attachments becomes a group consisting of a text note that has the same name as the group and attached files separately. And every child of such group has the same internal Evernote URL in its URL field. Thus step (iii) must include an additional mechanism that finds the note with text (not an attachment file). On my opinion the easiest way to do this is to test whether a record with correct URL is of type “formatted note” (all Evernote imports with text have this type).

I’m a developer myself but I’m not used to apple script dictionary syntax. I’ve tried to write the script myself, and this is what I’ve got so far:

using terms from application "DEVONthink 3"
	tell application "DEVONthink 3"
		try
			# (i) for every record in db check if record is formatted note
			set recs to a reference to contents of current database
			repeat with rec1 in recs
				if ((type of rec1) as text) is "formatted note" then
					log message (name of rec1) as text
					# (ii) if record contains links starting with "evernote://"
					set chunks to attribute run of rich text of rec1
					repeat with chunk in chunks
						if (URL of chunk) is not null then # ERROR: Can't get URL of "_my note text without link is here_"
							log message (URL of chunk) as text
							if URL of chunk starts with "evernote://" then
								if exists record with URL (URL of chunk) in current database then
									set recs2 to lookup records with URL in current database
									# (iii) for every record2 in db if type of record2 is correct
									repeat with rec2 in recs2
										if type of rec is "formatted note" then
											# (iv) check local link of record2 and replace with it the old one
											set url2 to URL of record2
											if url2 is equal to URL of chunk then
												log message ((URL of chunk) as text) & " got to be replaced with " & ((reference URL of rec2) as text)
												# set URL of chunk to reference URL of rec2
											end if
										end if
									end repeat
								end if
							end if
						end if
					end repeat
				end if
			end repeat
		on error error_message number error_number
			if the error_number is not -128 then
				display alert "DEVONthink 3" message error_message as warning
			end if
		end try
	end tell
end using terms from

An error occurs at the line “if (URL of chunk) is not null then”. I’ve tried a lot of options instead of null but can’t figure out how to test if a chunk of rich text has an active URL property.

Any help to finish this script it will be highly appreciated!

No time at the moment so just a quick info: You don’t need to loop thru all records.

Instead take a look at the search command in the AppleScript dictionary (in Script Editor.app or Script Debugger.app (which got a superior dictionary explorer)).

Here‘s a script that uses this command Searching for items containing the same Alias?.

search doesn’t use the scope: syntax that’s used in toolbar searches. To search an entire database use in root of [database] (as in the linked script).

Thanks for the posting; I need to follow through and clean up the Evernote links in my data.
The Lookup records with URL works great, but I’m just simply text searching note contents

As you noted, before Evernote export we need to save the note’s internal link
I appended the note to the note contents (actually just the note-id)
(example ID_73a4b241-2198-4072-8401-212759efe87b)

take a look at the search command in the AppleScript dictionary

Thank you! It’s a very nice approach how to get notes that have url property (target notes to link to) instead of my set recs2 to lookup records with URL in current database.
However didn’t found any search prefix or other approach that can help with searching notes, that have urls in text (and futhermore with reference text that begins with “evernote://”). Something similar to what the Links pane in the app displays but as search prefix would be great,
like search "links:evernote".
And the worst is that I am still not able to get (URL of chunk) to be handled correctly, because the most of rich text chunks obviously don’t have embedded urls, so I get “Can’t get URL of ‘My random part of text here’” error.

I‘ve never used Evernote so I‘ve nothing to test. If you could post two files I‘ll take a look.

Processing: test.enex…
test.enex.zip (2.2 KB)

I’ve attached an example. There are two notes in enex file that can be imported via File > Import > Files and Folders. The first note Sed ut perspiciatis has a link to the second note Lorem ipsum with reference evernote:///view/2199673/s25/438c47ee-8c1e-418f-a2c7-54e84467ace9/438c47ee-8c1e-418f-a2c7-54e84467ace9/. I’m trying to get all such references to be replaced with Devonthink cross-note links. Both imported notes will have text starting with evernote:///view/... embedded in their url metadata field, however only the destination note has evernote:///view/2199673/s25/438c47ee-8c1e-418f-a2c7-54e84467ace9/438c47ee-8c1e-418f-a2c7-54e84467ace9/.
I wonder how developers of the app did this. Because notes imported the proposed way (File > Import > Notes from Evernote) have their links already replaced. But I can’t import one huge notebook with standard means, this is why I need all this AppleScript stuff.

BTW this is my current version of script ATM:

using terms from application "DEVONthink 3"
	tell application "DEVONthink 3"
		try
			# (i) for every formatted note in main db
			set db to database "main"
			set recs to search "kind:formattednote" in root of db
			repeat with rec1 in recs
				log message (name of rec1) as text
				# (ii) for every link in note pointing towards destination that starts with "evernote://"
				set chunks to attribute run of rich text of rec1
				repeat with chunk in chunks
					set ref1 to (URL of chunk)
					if ref1 is not null then # ERROR
						if ref1 starts with "evernote://" then
							set recs2 to search "kind:formattednote url:" & (ref1 as text) in root of db
							# (iii) for every record2 formatted note in db that has the given url in its url field
							repeat with rec2 in recs2 # there should be only one record, this line is for debug reasons
								# (iv) replace evernote link with the new one
								set url2 to URL of record2
								if url2 is equal to URL of chunk then
									log message (ref1 as text) & " got to be replaced with " & ((reference URL of rec2) as text)
									# set (URL of chunk) to (reference URL of rec2)
								end if
							end repeat
						end if
					end if
				end repeat
			end repeat
		on error error_message number error_number
			if the error_number is not -128 then
				display alert "DEVONthink 3" message error_message as warning
			end if
		end try
	end tell
end using terms from

This script replaces evernote:/// link URLs with the corresponding record’s Reference URL.

Note: Do not run this script on all your records at once.

-- Replace evernote:/// Link URLs in Formatted Notes with DEVONthink Reference URLs

-- Note: This script finds a link's corresponding record by searching the link's URL in DEVONthink's URL property.

use AppleScript version "2.4"
use framework "Foundation"
use scripting additions

tell application id "DNtp"
	try
		set theRecords to selected records whose type = formatted note and URL starts with "evernote:///"
		
		repeat with thisRecord in theRecords
			set thisRecord_Source to source of thisRecord
			if thisRecord_Source contains "<a href=\"evernote:///view/" then
				set theEvernoteURLs to my regexFind(thisRecord_Source, "(?<=<a href=\\\")evernote:///view(.*?)(?=\")")
				repeat with thisEvernoteURL in theEvernoteURLs
					set thisEvernoteURL to thisEvernoteURL as string
					set theResults to search "kind:formattednote url==" & thisEvernoteURL
					if theResults ≠ {} then
						set thisResult to item 1 of theResults
						set thisResult_ReferenceURL to reference URL of thisResult
						set thisRecord_Source to my regexReplace(thisRecord_Source, thisEvernoteURL, thisResult_ReferenceURL)
					end if
				end repeat
				set source of thisRecord to thisRecord_Source
			end if
		end repeat
		
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		return
	end try
end tell

on regexFind(theText, thePattern)
	try
		set theString to current application's NSString's stringWithString:theText
		set {theExpr, theError} to current application's NSRegularExpression's regularExpressionWithPattern:(thePattern) options:0 |error|:(reference)
		set theMatches to theExpr's matchesInString:theString options:0 range:{0, theString's |length|()}
		set theResults to {}
		repeat with thisMatch in theMatches
			set thisMatchRange to (thisMatch's rangeAtIndex:0)
			set thisMatchString to (theString's substringWithRange:thisMatchRange) as string
			set end of theResults to thisMatchString
		end repeat
		return theResults
	on error error_message number error_number
		activate
		display alert "Error: Handler \"regexFind\"" message error_message as warning
		error number -128
	end try
end regexFind

on regexReplace(theText, thePattern, theRepacement)
	try
		set theString to current application's NSString's stringWithString:theText
		set newString to theString's stringByReplacingOccurrencesOfString:(thePattern) withString:(theRepacement) options:(current application's NSRegularExpressionSearch) range:{location:0, |length|:length of theText}
		set newText to newString as string
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"regexReplace\"" message error_message as warning
		error number -128
	end try
end regexReplace

2 Likes

Thank you, you are awesome!

set theRecords to selected records whose type = formatted note and URL starts with "evernote:///" and source contains "<a href=\"evernote:///view/" leads to DEVONthink 3 got an error: AppleEvent timed out. error.
I’ve got to fall back to code like this

repeat with thisRecord in theRecords
	if source of thisRecord contains "<a href=\"evernote:///view/" then

to process one by one.

Also by some unknown and unreported reason (no errors being thrown) this script always stops at processing of a random note far from the end of list (250-600 of 4600).
But this is not so important, because the replacement works, thanks to you!
I hope, I’ll be able go get it working like a charm finally.

1 Like

Hmm that’s odd.

Getting selected records filtered by a whose clause is normally blazing fast. You could add a custom timeout:

with timeout of 3600 seconds
	
end timeout

Strange. No idea what’s going on. However, you could activate NSScriptingDebugLogLevel and see whether you can find something in Console.app. I never really used the console or NSScriptingDebugLogLevel but as you’re a developer you probably know how it works.

-- Activate NSScriptingDebugLogLevel

property theAppIdentifier : "com.devon-technologies.think3"

set theChooseFromListItems to {"Activate Debugging", "Deactivate Debugging"}
set theChoice to choose from list theChooseFromListItems with prompt "" default items (item 1 of theChooseFromListItems) with title "NSScriptingDebugLogLevel"
if theChoice is false then return

if item 1 of theChoice = item 1 of theChooseFromListItems then
	set theMode to "1"
	
else if item 1 of theChoice = item 2 of theChooseFromListItems then
	set theMode to "0"
end if

do shell script "defaults write" & space & theAppIdentifier & space & "NSScriptingDebugLogLevel " & theMode

Maybe I’m doing something wrong? I select my group (folder) first and then write kind:formattednote scope:selection in the search bar. I have formatted note condition removed from the code of the first selection query to decrease its execution time. After all I press Cmd + A to select all filtered notes.

Btw a large timeout lets query finish yet results in another very strange thing. Query returns only 200-300 notes of ~4300. By my fairy accurate approximation at least 3300 of them do have URLs in Evernote’s internal format.

I have not tried NSScriptingDebugLogLevel yet. Maybe it can provide more information on what is happening…

No idea. Perhaps not all Evernote link URLs start with evernote:///view ?

It’s hard to tell what might go wrong as I’ve never used Evernote and only tested with two files.

@BLUEFROG, @Idify uses a modified version of this script. It should find all records that got a link with a Evernote URL but seems to only find a part of the records. Any idea why?

Not sure as I’m getting an immediate error when running the script from Trying to relink all URLs after import from Evernote's enex files with Apple Script - #6 by Idify

Ah sorry. I‘ve edited my post:

No worries! Your script worked as expected, though I don’t have 4000+ Evernote notes.

  • Do you see ~4300 files in the toolbar search results?
    • If so, what if you add url:evernote ? What do you see then?
1 Like

No. Here’s the numbers atm:
Total cound of items in goup, including subgroups: 4285
kind:formattednote scope:selection leads to 4260 items - the exactly count of Evernote notes in this group.
kind:formattednote scope:selection url:evernote doesn’t do anything (same 4260 items selected) since all imported notes have url starting with evernote:///view/ in URL field — it’s result of my another script that copies local link for every selected note in Evernote and pastes it into the URL field (so I’m guaranteed links will be kept after export-import to other app like Devonthink).

Today I did something new. Split this huge notebook (group) into parts and finally made to import all notebooks with File > Import > Notes from Evernote. Many urls got already replaced by this feature, but many of them are not.
I applied script to every group consisting of 150-400 records each. And the same thing happens script stops accidentally at some random item. The result of script execution is always true.
Additionally when I run this script for a single item some urls are getting replaced, some are not.
Repeating gives more replacements, so the script can be used many-many times by hand, and than there’s a large probability (almost) all urls are replaced. But to be honest it’s not a kind of result I expect from a script :slight_smile:

@BLUEFROG, are you a Devonthink developer? Is there another API to command Devonthink, e.g. Objective-C / Swift / C / Python etc.-compatible? Apple Script has such a messy debug flow, I can’t understand, why the proposed script can’t find and replace all links with “evernote:///” in reference text. Or may be is there a way to reactivate internal relinking routine? To access DEVONthink-Text.dtMetaStore directly (as a db? text file with encoding?) to find and replace text using full power of a compiled language?
I think the reason of all this mess with AppleScript may lay in Apple’s unannounced limitations to what and how long on their wise opinion can be done using scripting. And that doesn’t symply accepts workaround of with timeout of 999999 seconds

The current version of script:

-- Replace evernote:/// Link URLs in Formatted Notes with DEVONthink Reference URLs

-- Note: This script finds a link's corresponding record by searching the link's URL in DEVONthink's URL property.
-- Search for `kind:formattednote scope:selection` + select folder first

-- Activate NSScriptingDebugLogLevel
property theAppIdentifier : "com.devon-technologies.think3"

set theChooseFromListItems to {"Activate Debugging", "Deactivate Debugging"}
set theChoice to choose from list theChooseFromListItems with prompt "" default items (item 1 of theChooseFromListItems) with title "NSScriptingDebugLogLevel"
if theChoice is false then return

if item 1 of theChoice = item 1 of theChooseFromListItems then
	set theMode to "1"
	
else if item 1 of theChoice = item 2 of theChooseFromListItems then
	set theMode to "0"
end if

do shell script "defaults write" & space & theAppIdentifier & space & "NSScriptingDebugLogLevel " & theMode
-- debug block ends

use AppleScript version "2.4"
use framework "Foundation"
use scripting additions

set notesContentToReplace to "<a href=\"evernote:///view/"
set sourceLinkPattern to "(?<=<a href=\\\")evernote:///view(.*?)(?=\")"

with timeout of 60000 seconds
	tell application id "DNtp"
		try
			set theRecords to selected records whose type = formatted note
			set note_nr to 1 # DEBUG
			set notes_cnt to (count of theRecords) as text # DEBUG
			repeat with thisRecord in theRecords
				if source of thisRecord contains notesContentToReplace then
					log message "========= Processing note " & (note_nr as text) & " of " & notes_cnt & ":: " & (name of thisRecord) as text # DEBUG
					set note_nr to note_nr + 1 # DEBUG
					set thisRecord_Source to source of thisRecord
					set theEvernoteURLs to my regexFind(thisRecord_Source, sourceLinkPattern)
					repeat with thisEvernoteURL in theEvernoteURLs
						set thisEvernoteURL to thisEvernoteURL as string
						set theResults to search "kind:formattednote url==" & thisEvernoteURL
						if theResults ≠ {} then
							set thisResult to item 1 of theResults
							set thisResult_ReferenceURL to reference URL of thisResult
							set thisRecord_Source_new to my regexReplace(thisRecord_Source, thisEvernoteURL, thisResult_ReferenceURL)
							log message (thisEvernoteURL & " -> " & (name of thisResult) as text) & " => " & thisResult_ReferenceURL # DEBUG
							delay 1 # DEBUG async issue? 
							set source of thisRecord to thisRecord_Source_new
						end if
					end repeat
				else # DEBUG
					log message "========= Note " & (note_nr as text) & " of " & notes_cnt & " doesn't seem to have any Evernote urls:: " & (name of thisRecord) as text # DEBUG
				end if
			end repeat
			
		on error error_message number error_number
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
			return
		end try
	end tell
end timeout

on regexFind(theText, thePattern)
	try
		set theString to current application's NSString's stringWithString:theText
		set {theExpr, theError} to current application's NSRegularExpression's regularExpressionWithPattern:(thePattern) options:0 |error|:(reference)
		set theMatches to theExpr's matchesInString:theString options:0 range:{0, theString's |length|()}
		set theResults to {}
		repeat with thisMatch in theMatches
			set thisMatchRange to (thisMatch's rangeAtIndex:0)
			set thisMatchString to (theString's substringWithRange:thisMatchRange) as string
			set end of theResults to thisMatchString
		end repeat
		return theResults
	on error error_message number error_number
		activate
		display alert "Error: Handler \"regexFind\"" message error_message as warning
		error number -128
	end try
end regexFind

on regexReplace(theText, thePattern, theRepacement)
	try
		set theString to current application's NSString's stringWithString:theText
		set newString to theString's stringByReplacingOccurrencesOfString:(thePattern) withString:(theRepacement) options:(current application's NSRegularExpressionSearch) range:{location:0, |length|:length of theText}
		set newText to newString as string
	on error error_message number error_number
		activate
		if the error_number is not -128 then display alert "Error: Handler \"regexReplace\"" message error_message as warning
		error number -128
	end try
end regexReplace

Sorry. I forgot to use the the new source for replacements. Replacing in the old source of course doesn’t work as expected.

That’s the kind of thing that happens with bad test material.

I’ve updated the script.

No. I am support, documentation, and automation.

No there is no other API. You could use JXA if you’re inclined but there’s no specific support for it.

Or may be is there a way to reactivate internal relinking routine? To access DEVONthink-Text.dtMetaStore directly (as a db? text file with encoding?) to find and replace text using full power of a compiled language?

No this is not publicly available.

Remove the try block and see where it errors - something I say you should always do when initially developing a script.

JavaScript, of course. I doubt that you can access DT with ObjC withouth using either AppleScript or JavaScript, though. Python – well, you have to find a library that permits calling the scripting methods.
Personally, I’d try JavaScript because it has RegEx support on board and does not require the trip to ObjC.

Some remarks though: Having “kind:formattenote” in a loop that runs over whose type = formatted note is redundant and confusing. Also, using kind is discouraged because it depends on the region settings. You’re using only the first item of thisResult in your replacement step. Is this intended?

You’re thinking of the kind vs. type properties in DEVONthink’s scripting dictionary. Yes, in this context type should be used instead of kind for the reason you mentioned.

However, the script uses kind as literal string in a query (just like it’s used in a toolbar search).

Yes, as for each Evernote URL there can only be one corresponding DEVONthink record.

No worries, It didn’t attracted my attention as well. And thank you for the fix!

Additionally, I found a couple of childish issues with my debug flow too (forgot to count records, that don’t have urls with evernote:/// references, some of notes were having wrong urls in URL field, some links pointed to notes that were already deleted and therefore missing).
By fixing all this, and a big folder split into 8 smaller ones yesterday, today I’ve managed not only to import all Evernote records via the standard Import from Evernote tool but also to have a smooth replacement process as well.
However today also wasn’t my lucky day. I was feeling lazy and selected all 4200+ records to have their links replaced in one run and left my mac. It finished after about 2500 records with a huge (80 Gb!) memory leak that took all of my 32 Gigs of RAM and all possible space left on the system SSD and led to a halt, reboot and Devonthink database corruption and an additional 8 Gb Devonthink log (its my fault too because it seems that I added to much log message debug stuff to the script).

But! The good news are the updated version of script should work fine.
I will reimport everything again and run script for every group of 200-500 records separately tomorrow. This should finally work.