The Ultimate Email Import Script

Note: The attached script has been seriously transformative at improving my workflow for importing email and file attachments into Devonthink. It includes concepts from mutltiple email scripts posted here over the years by others (including @cgrunenberg , @BLUEFROG , and @pete31 among others). Please feel free to use the script and post your tweaks or improvements that work with your particular use case.

I have long sought ways to import email into DT3 and have discussed and contributed a number of ideas in this forum.

My particular workflow for email involves regularly receiving both email message and file attachments which I need to file into specific Groups which I set up for each client case.

The DT3 scripts for Apple Mail handle this well, but that requires that I sync my email to each Mac I use and it limits my ability to forward email when using iOS Mail or when borrowing some other computer when out of the home/office.

For now I have been using a combination of forwarding emails via Zapier as discussed here Cool Tip: Auto Forward Office 365 Email into DT3 - Zapier + Indexed Dropbox Folder or saving Bookmarks to webmail as discussed here Solved: Saving Office 365 Web Emails in DT3 . These work but have some limitations in terms of handling file attachments; plus I would prefer to have the email imported into DT3 and not dependent on maintaining the original email as in the Bookmark solution. Plus both of these solutions require that I manually file the email to a specific group after it is imported into DT3.

I took a look at a number of solutions and scripts posted here over the years by multiple contributors as above. When I came across a post on Reddit by developer @8isnothing discussing his experience with iOS and macOS automation, I asked if he could help implement some ideas I had. [I know enough Applescript to be dangerous and edit existing scripts but do not have the time to do a full-scale project like this.]

Thus the attached Import to Devonthink script was born.

Before using the script do as follows:

(1) Edit the script include UUIDs for an “Invalid Link” group and a “No Link” group

(2) Edit the script to include your email address for notifications of script success, or if not desired edit the boolean to turn this feature off

(3) Set up a Mail rule to execute the script:

(a) One option is to set up an email account used solely for importing email. This is preferred because you can bcc: this email on outgoing mail and thus the email will be filed, but in doing so you are not publicly revealing an email address that others can use to spam your Devonthink database:

image

(b) An alternate option is to use your existing email and set up mail rules to identify subject lines with import instructions at the end to import to a specific Group based on an x-devonthink-item link or based on a Group name:

image

Once that is done you use the script as follows:

  • If you end a Subject line with an x-devonthink-item link corresponding to a Group, then the email is filed in that group; if there are attachments; then the email and attachments are stored in a subgroup using the email name

This is my subject x-devonthink-item://XXXXX-XXX-XXX-XXX-XXXX

  • If you end a Subject line with text such as //Group: Group Name Here, then the script searches for any Group containing that text and adds the email as above. If more than one Group matches, then replicants are created; however, if there are group matches in more than one database, then a duplicate is created since replicants must be in the same database

This is my subject //Group: Group Name Here

  • If you include an invalid x-devonthink-item link or your //Group: does not exist, then the item is created in your “Invalid Link” group.

  • If you do not include any //Group: or x-devonthink-item link at all, then the item goes to your No Link group.

  • You may wish to use the No Link feature intentionally in order to create a new Group on the fly as you forward email by ending your subject with text such as:

This is my subject @Group Name Here

You can do that with a smart rule similar to this:

image

Note that if you use an @ filing SmartRule as above, there is currently a bug which creates duplicate groups on future use - that bug is to be fixed soon as discussed here "File Using @ Destination" Script --> Duplicate Group Names .

  • If you have activated the boolean in the script for email notification, you will receive an email confirming that the script has run and the destination(s) into which the email was filed.

Again this particular workflow works extremely well for my use case where I often get client-specific messages and attachments via email. It can obviously be modified for other uses. I remain curious to see what features others may contribute.

-- This script is released to the public via Devonthink Forums with the request that any further revisions will be shared with the community
-- It was conceived of by @rkaplan and written as a work for hire by @8isnothing
-- Substantial help and prior influence on this concept is noted and appreciated from @cgrugenberg, @BLUEFROG, @pete31, and many others

--LOG global vars
global shouldLog
global logSubject
global logToAddresses
global logAttachmentsList
global logGroupId
global logGroupsList
global logDatabasesList
global logFinalStringList
--

using terms from application "Mail"
	on perform mail action with messages these_messages for rule this_rule
		tell application "Mail"
			-- SETTINGS
			-- Invalid link UUID
			set invalidLinkUUID to "INSERT GROUP UUID"
			-- No link UUID
			set noLinkUUID to "INSERT GROUP UUID"
			
			--LOG settings
			set shouldLog to true
			set logSubject to "Script Log"
			set logToAddresses to {"your@email.com"} -- add as many emails as you want, between quotes separated by comma
			set logAttachmentsList to {}
			set logGroupId to ""
			set logGroupsList to {}
			set logDatabasesList to {}
			set logFinalStringList to {}
			--
			
			set the message_count to the count of these_messages
			repeat with i from 1 to the message_count
				set this_message to item i of these_messages
				
				-- Create temp folder if it doesn't exist
				do shell script "mkdir -p ~/.scriptTemp"
				
				-- GET EMAIL DATA
				
				-- Get whole subject field
				try
					set this_subject to (subject of this_message) as Unicode text
					if this_subject is "" then error
				on error
					set this_subject to "Missing Subject"
				end try
				
				-- Check if there's any DEVONthink link
				if this_subject contains "x-devonthink-item://" then
					
					-- Get Group UUID
					set groupUUID to my getGroupId("x-devonthink-item://", this_subject)
					-- Get clean subject field
					set cleanSubject to my getCleanSubject("x-devonthink-item://", this_subject)
					-- Set link type
					set linkType to "uuid"
					
					-- Check if UUID exists
					tell application id "DNtp"
						try
							set uuidCheck to get record with uuid groupUUID
							set uuidCheck to uuid of uuidCheck
							if uuidCheck is "" then error
						on error
							set groupUUID to invalidLinkUUID
						end try
					end tell
				end if
				
				
				-- Check if there's any "//Group:"
				if this_subject contains "//Group:" then
					
					-- Get Group name
					set groupName to my getGroupId("//Group:", this_subject)
					-- Get subject
					set cleanSubject to my getCleanSubject("//Group:", this_subject)
					-- Set link type
					set linkType to "name"
					
					-- Check if Group exists
					tell application id "DNtp"
						try
							set searchResults to get search groupName
							if searchResults is equal to {} then error
							
							-- Create empty Group list
							set groupList to {}
							
							--Handle wildcard
							if last character of groupName is "*" then
								set groupName to texts 1 thru -2 of groupName
							end if
							
							-- Check if result is a Group
							repeat with searchResult in searchResults
								
								set resultType to the type of searchResult
								set resultName to the name of the searchResult
								
								if resultType is equal to group and resultName contains groupName then
									set resultUUID to the uuid of searchResult
									
									-- Save result to list
									set end of groupList to {uuid:resultUUID}
								end if
							end repeat
							
							if groupList is equal to {} then error
							
						on error
							set groupUUID to invalidLinkUUID
							set linkType to "uuid"
						end try
					end tell
				end if
				
				-- If there's no "//Group:" and "x-devonthink-item://"
				if this_subject does not contain "//Group:" and this_subject does not contain "x-devonthink-item://" then
					
					set groupUUID to noLinkUUID
					set cleanSubject to this_subject
					
					-- Set link type
					set linkType to "uuid"
				end if
				
				-- Get content
				set this_source to the source of this_message
				
				-- Create .eml file name 
				set newFile to (cleanSubject & ".eml") as rich text
				
				-- Set the folder path
				set theFolder to (system attribute "HOME") & "/.scriptTemp/"
				
				-- Set the file path
				set newFilePath to theFolder & newFile as rich text
				
				-- Save .eml  file
				try
					set referenceNumber to open for access newFilePath with write permission
					try
						write this_source to referenceNumber
					on error
						display alert "Error saving temporary file"
						close access referenceNumber
					end try
					close access referenceNumber
					
				on error
					set newFile to "Invalid Subject.eml"
					set newFilePath to theFolder & newFile
					try
						set referenceNumber to open for access newFilePath with write permission
						try
							write this_source to referenceNumber
						on error
							display alert "Error saving temporary file"
							close access referenceNumber
						end try
						close access referenceNumber
					end try
				end try
				
				-- GET ATTACHMENTS 
				-- Count attachments
				set attachments_count to the count of mail attachments of this_message
				
				--Loop attachments
				repeat with anAttachment in mail attachments of this_message
					-- Get attachment name
					set attachmentName to name of anAttachment
					-- Save attachment
					save anAttachment in POSIX file (theFolder & attachmentName)
					
					--LOG
					my addToLogAttachmentsList(attachmentName)
					--
				end repeat
				
				-- Convert theFolder to POSIX	
				set theFolderPOSIX to theFolder as rich text
				set theFolderPOSIX to POSIX path of theFolderPOSIX
				
				-- Import to DEVONthink 3
				tell application id "DNtp"
					
					-- If "x-devonthink" detected
					if linkType is "uuid" then
						-- Get link group by UUID
						set mainGroup to get record with uuid groupUUID
						my importToDEVON(attachments_count, theFolderPOSIX, cleanSubject, mainGroup, newFilePath)
					end if
					
					-- If "//Group:" detected
					if linkType is "name" then
						if (count of groupList) > 1 then
							
							set mainGroup to get record with uuid uuid of item 1 of groupList
							set importedItem to get my importToDEVON(attachments_count, theFolderPOSIX, cleanSubject, mainGroup, newFilePath)
							
							repeat with groupKey in groupList
								-- Loop through the list but ignore item 1
								if uuid of groupKey is not equal to uuid of mainGroup then
									set secondaryGroup to get record with uuid uuid of groupKey
									try
										set successCheck to get replicate record importedItem to secondaryGroup
										set successCheck to uuid of successCheck
										if successCheck is {} then error
										
										--LOG
										my addToLogGroupsList(name of secondaryGroup, uuid of secondaryGroup)
										--
									on error
										set importedItem to get duplicate record importedItem to secondaryGroup
										
										--LOG
										my addToLogGroupsList(name of secondaryGroup, uuid of secondaryGroup)
										my addToLogDatabasesList(database of secondaryGroup)
										--
									end try
								end if
							end repeat
						else
							set mainGroup to get record with uuid uuid of item 1 of groupList
							my importToDEVON(attachments_count, theFolderPOSIX, cleanSubject, mainGroup, newFilePath)
						end if
					end if
				end tell
				
				-- Set mail flag
				tell application "Mail"
					set flag index of this_message to 0
				end tell
				-- Delete temp folder permanently
				do shell script "rm -rf ~/.scriptTemp"
				
				--LOG
				my logSaveLog(cleanSubject, attachments_count, invalidLinkUUID, noLinkUUID)
				--
			end repeat
			
			-- Delete temp folder permanently 2 pass
			do shell script "rm -rf ~/.scriptTemp"
			
			
			--LOG
			my logCheckIfAllImported(message_count)
			my logSendLog()
			--
		end tell
	end perform mail action with messages
end using terms from



-- FUNCTIONS

-- Get clean subject
on getCleanSubject(myString, wholeSubject)
	
	set cleanSubject to wholeSubject
	set AppleScript's text item delimiters to myString
	set cleanSubject to item 1 of every text item of cleanSubject
	set AppleScript's text item delimiters to ""
	
	-- Clean any "/" character	
	if cleanSubject contains "/" then
		set cleanSubject to fixEmailSubject(cleanSubject)
	end if
	
	if cleanSubject is equal to "" then
		set cleanSubject to "Missing Subject"
	end if
	set cleanSubject to cleanEmptySpace(cleanSubject)
	return cleanSubject
end getCleanSubject


-- Fix email subject
on fixEmailSubject(theText)
	set AppleScript's text item delimiters to "/"
	set theTextItems to every text item of theText
	set AppleScript's text item delimiters to "-"
	set theText to theTextItems as string
	set AppleScript's text item delimiters to ""
	return theText
end fixEmailSubject


-- Get Group UUID/Name
on getGroupId(myString, wholeSubject)
	set groupId to wholeSubject
	set AppleScript's text item delimiters to myString
	set groupId to item 2 of every text item of groupId
	set AppleScript's text item delimiters to ""
	
	set groupId to cleanEmptySpace(groupId)
	
	--LOG
	my addToLogGroupId(myString, wholeSubject)
	--
	
	return groupId
end getGroupId


-- Clean empty space
on cleanEmptySpace(myString)
	-- Check if first character is empty space
	repeat while first character of myString is " "
		set myString to text 2 thru -1 of myString
	end repeat
	
	-- Check if last character is empty space
	repeat while last character of myString is " "
		set myString to text 1 thru -2 of myString
	end repeat
	
	return myString
end cleanEmptySpace


-- Import to DEVONtink
on importToDEVON(attachments_count, theFolderPOSIX, cleanSubject, mainGroup, newFilePath)
	-- Check if there's any attachment
	tell application id "DNtp"
		if attachments_count > 0 then
			tell application "System Events"
				set titleList to POSIX path of disk items of folder theFolderPOSIX
			end tell
			tell application id "DNtp"
				-- Create subgroup
				set subGroup to create record with {name:cleanSubject, type:group} in mainGroup
				repeat with posixPath in titleList
					-- Import files
					import posixPath to subGroup
				end repeat
				
				--LOG
				my addToLogGroupsList(name of mainGroup, uuid of mainGroup)
				my addToLogDatabasesList(database of mainGroup)
				--
			end tell
			
			return subGroup
		else
			-- In case threre's no attachment
			set posixPath to POSIX path of newFilePath
			set importedItem to import posixPath to mainGroup
			
			--LOG
			my addToLogGroupsList(name of mainGroup, uuid of mainGroup)
			my addToLogDatabasesList(database of mainGroup)
			--
			
			return importedItem
		end if
	end tell
end importToDEVON




--LOG
on addToLogAttachmentsList(attachmentName)
	if shouldLog is true then
		set end of logAttachmentsList to attachmentName
	end if
end addToLogAttachmentsList


on addToLogGroupsList(groupName, groupUUID)
	if shouldLog is true then
		set end of logGroupsList to {name:groupName, uuid:groupUUID}
	end if
end addToLogGroupsList

on addToLogDatabasesList(myDatabase)
	if shouldLog is true then
		tell application id "DNtp"
			set myDatabase to name of myDatabase
		end tell
		set end of logDatabasesList to myDatabase
	end if
end addToLogDatabasesList

on addToLogGroupId(myString, wholeSubject)
	if shouldLog is true then
		set groupId to wholeSubject
		set AppleScript's text item delimiters to myString
		set groupId to item 2 of every text item of groupId
		set AppleScript's text item delimiters to ""
		set groupId to myString & groupId
		
		set groupId to cleanEmptySpace(groupId)
		
		set logGroupId to groupId
	end if
end addToLogGroupId

on logCheckIfAllImported(message_count)
	if shouldLog is true then
		-- Check if all emails were imported successfully
		if message_count > (count of logFinalStringList) then
			display alert "ERROR" & "\n" & (message_count - (count of logFinalStringList)) & " emails were not imported"
		end if
	end if
end logCheckIfAllImported


on logSaveLog(cleanSubject, attachments_count, invalidLinkUUID, noLinkUUID)
	if shouldLog is true then
		
		set attachmentsNamesString to ""
		set groupNamesString to ""
		set databaseNamesString to ""
		set myErrorAlert to ""
		
		set myTime to time string of (current date)
		set myDate to do shell script "date +'%m/%d/%Y'"
		
		repeat with attachmentName in logAttachmentsList
			if attachmentsNamesString is "" then
				set attachmentsNamesString to "\"" & attachmentName & "\""
			else
				set attachmentsNamesString to attachmentsNamesString & ", " & "\"" & attachmentName & "\""
			end if
		end repeat
		
		repeat with databaseName in logDatabasesList
			if databaseNamesString is "" then
				set databaseNamesString to databaseName
			else
				set databaseNamesString to databaseNamesString & ", " & databaseName
			end if
		end repeat
		
		repeat with groupInfo in logGroupsList
			if uuid of groupInfo is invalidLinkUUID or uuid of groupInfo is noLinkUUID then
				set myErrorAlert to "****"
			end if
			if groupNamesString is "" then
				set groupNamesString to "\"" & name of groupInfo & "\""
			else
				if uuid of groupInfo is invalidLinkUUID or uuid of groupInfo is noLinkUUID then
					set groupNamesString to groupNamesString & ", " & "\"*" & name of groupInfo & "*\""
				else
					set groupNamesString to groupNamesString & ", " & "\"" & name of groupInfo & "\""
				end if
			end if
		end repeat
		
		
		
		
		set logText to "Error Alert: " & myErrorAlert & "\nDate: " & myDate & "\nTime: " & myTime & "\nSubject: " & cleanSubject & "\nGroup/Link: " & logGroupId & "\nDestination Databases: " & databaseNamesString & "\nDestination Group: " & groupNamesString & "\nAttachments: " & attachmentsNamesString
		
		set end of logFinalStringList to logText
		
		-- Clean global vars
		set logAttachmentsList to {}
		set logGroupId to ""
		set logGroupsList to {}
		set logDatabasesList to {}
		
	end if
end logSaveLog


on logSendLog()
	if shouldLog is true then
		tell application "Mail"
			set theContent to ""
			
			repeat with finalString in logFinalStringList
				if theContent is "" then
					set theContent to finalString
				else
					set theContent to theContent & "\n\n" & finalString
				end if
			end repeat
			
			set logMessage to make new outgoing message with properties {subject:logSubject, content:theContent, visible:false}
			tell logMessage
				repeat with toAddress in logToAddresses
					make new to recipient at end of every to recipient with properties {address:toAddress}
				end repeat
			end tell
			send logMessage
		end tell
	end if
end logSendLog


Import to DEVONthink v3.2.4.scpt.zip (32.8 KB)

5 Likes

Many thanks to you and @8isnothing for this excellent script.

However, on my computer ~/.scriptTemp is not deleted after the rules is executed, which means that the content of all the old emails is added to the last imported email.

Am I doing something wrong?

Thanks.

It works for me

I will let @8isnothing give his input

Do DT3 and Script Editor have Full Disk Access?

Thank you for your quick reply.

Yes, they now have, but the behaviour is the same.

As a workaroud, I have added the command “do shell script “rm -rf ~/.scriptTemp”” at the beginning of the script and now it work, so this is not a permission issue.

This is weird. The script has this exact terminal command in the end of each loop. But nice to hear it’s working fine for you now!

Race condition ?

Don’t think so.
Most likely it was related to the Full Disk Access setting

Sorry for being late to the party. Since I’m currently fiddling around with something similar, I’d like to ask if any of you have tried to save only attachments of a particular MIME type. And if so, how did you go about it.

Background: When I try to access the mime type property of an attachment, I get the Apple Script error -10000 (interestingly for both JavaScript and AppleScript). Accessing the name property works ok, also downloaded, file size gives the same error -10000. The error, BTW, is not related to the download status of the attachment, I see it happening also with attachments that are downloaded.

Thanks in andvance for any ideas (and yes, I did find the thread concerning this problem from 2015).

I have a solution, give me some minutes

Well, I got sidetracked … Script: Import mail attachments filtered by MIME type