Adding email with IMAP flags (keywords / tags)

I realise that topics similar to this come up from time to time, but I’ve not quite seen this particular issue covered.

My setup:

  • MailMate on macOS
  • Preside on iOS/iPadOS
  • Dovecot IMAP server (personally run)

What I want to do:

  • Archive old mail from macOS into DevonThink
  • Retain IMAP flags/tags/keywords

I have a significant email archive which has made use of IMAP flags (which seem to be interchangeably be called “tags”, “flags”, or “keywords” depending upon who is writing about it) which I’d like to remove from my IMAP server and store in DevonThink.

I know about the MailMate included “Command” that adds something to DT, and whilst that works, it does not preserve IMAP tags. Nothing seems to, and, with hundreds of tags, and many messages with multiple tags applied, it would be a huge loss to not be able to retain these.

As both Preside and MailMate work natively with my IMAP server to retain these flags/keywords, it’s been a very consistent and stable way to categorise messages and use various searching/smart-folder type technologies for a very long time.

Has anyone found a way to do what I am after with DevonThink? A way to export/add/drag/whatever messages and have any existing tags created or respected within DT? Even if I have to first manually create the tags in DT, I’m open to that.

Thank you!

If the „tags“ are stored in the mail headers, this might be scriptable. But I doubt that they are (check the headers in your mail client to find out). Otherwise, you’d have to query your IMAP server for the flags of each mail stored in DT and set the tags in DT accordingly.
Doing so does probably require a script that accepts a message ID (aka uuid in DT) and uses that to retrieve the information from the mail server.
That can be written in Perl and probably Python using the appropriate IMAP module, I think.
The result of this script can then be used in a JavaScript or AppleScript script to add it to the DT records.

Thank you very much for the thoughts - it turns out that there MAY be a slightly easier way, but, I’m not proficient enough in DT and AppleScript to get there.

I was looking at the included Bundles with MailMate and noticed that Eagle Filer has a bundle which does exactly what I want. I took advantage of their 30 day free trial to test, and without fail, the MM Bundle which extracts Tags from MailMate communicates via AppleScript to Eagle Filer and preserves those tags for archived mail.

From what I understand of the respective scripts, this seems like something that DevonThink can also do, but, the only missing piece for me is: How does AppleScript talk to DevonThink so that Tags are added to imported items?

The Eagle Filer AppleScript has two components. First is the Add.mmCommand:

{
	name          = 'Add';
	input         = 'raw';
	environment   = 'MM_SUBJECT=${subject.prefix:+${subject.prefix} }${subject.blob:+[${subject.blob}] }${subject.body:no subject}\nMM_MESSAGE_ID=${message-id.split}\nMM_TAGS=${##tags}\n';
	command       = '#!/bin/bash\n"${MM_BUNDLE_SUPPORT}/bin/add"';
	keyEquivalent = "^A";
	uuid          = 'F2C8D509-4239-4EA0-9D85-AF44025A0982';
}

Which, as you can see, grabs the tags from MailMate directly.

That then calls the “add” command, which reads:

#!/bin/bash

name=${MM_SUBJECT//:/_}
name=${name//\//_}
name=${name:0:250}

mkdir -p "$TMPDIR/com.freron.MailMate.EagleFiler"
tmpfile="$TMPDIR/com.freron.MailMate.EagleFiler/${name}.eml"
cat > "${tmpfile}"

url="message://%3c${MM_MESSAGE_ID}%3e"
tags="${MM_TAGS}"

osascript "${MM_BUNDLE_SUPPORT}/bin/EagleFilerImport.scpt" "${tmpfile}" "${url}" "${tags}"

# unlink "${tmpfile}" # Handled by EagleFiler

Again, pretty straightforward, as it’s calling the AppleScript with the Tags as an argument.

That AppleScript reads:

on run _argv
	-- Use separate handler to avoid creating implicit properties that would modify the script file when run.
	set _tagsString to item 3 of _argv
	set AppleScript's text item delimiters to " "
	set _tags to _tagsString's text items
	my importToEagleFiler(item 1 of _argv, item 2 of _argv, _tags)
end run

on importToEagleFiler(_emlPath, _sourceURL, _tags)
	try
		tell application "EagleFiler"
			-- Wait for document to be re-opened if we are launching EagleFiler.
			repeat 10 times
				if not (exists document 1) then
					delay 1
				end if
			end repeat
			
			-- If lots of imports are queued, this may take a while. EagleFiler would finish the import,
			-- but the script would report a timeout error, anyway. So make the script wait longer.
			with timeout of 10 * 60 seconds
				if _tags is {} then -- use default tags
					import files {_emlPath} source URL _sourceURL with deleting afterwards
				else
					import files {_emlPath} source URL _sourceURL tag names _tags with deleting afterwards
				end if
			end timeout
		end tell
	on error _errorMessage number _errorNumber
		tell application "System Events" -- Why not MailMate?
			activate
			display alert "EagleFiler: " & _errorNumber message _errorMessage
		end tell
	end try
end importToEagleFiler

And it works fine. So, it seems possible to me that if I understood AppleScript and DevonThink a bit more, I could modify the existing bundle to perform a similar task.

The current MailMate Bundle for DevonThink is pretty simple - it has the Add.mmCommand which sets up some variables (and is easy enough to modify to include the Tags), and then references the bin/add command, which embeds AppleScript within it:

#!/bin/bash

tmpfile="$TMPDIR/8D444CFE-C1F9-4F82-AD8B-4EA7EC7A77C2.eml"
cat > "${tmpfile}"

# Tricky substitution to make it work for " and \ in the AppleScript below.
name=${MM_SUBJECT//\\/\\\\}
name=${name//\"/\\\"}
url="message://%3c${MM_MESSAGE_ID}%3e"

osascript <<END

try
	tell application id "DNtp"
		set theRecord to import "${tmpfile}" from "MailMate" name "${name}"
		set modification date of theRecord to creation date of theRecord
		set the URL of theRecord to "${url}" # Allows it to be opened in MailMate using ⌃⌘O
	        display notification "Added message \"${name}\" to DEVONthink" with title "MailMate"
	end tell
on error errMsg number eNum
	tell application "System Events"
		activate
		display alert "DEVONThink: " & eNum message errMsg
	end tell
end try

END

unlink "${tmpfile}"

So, if DevonThink can be told via AppleScript to set tags, it seems that I should be able to modify the osascript above to inherit the tags from the message, and set them for the imported item.

The closest I’ve come to writing AppleScript was HyperCard, decades ago, and that’s a completely different beast.

According to Script Editor and the DevonThink Dictionary, there should be a way to set tags on a record, but that’s about as far as I’ve got so far.

Any pointers from anyone would be appreciated!

I made it a bit further by keeping things simple, and now have the DevonThink Pro Bundle able to import tags, however, the problem I now have is that if a single message has multiple tags, they are not being split correctly.

My solution was this:

  1. Modify Add.mmCommand to contain the following environment:
environment   = 'MM_SUBJECT=${subject.prefix:+${subject.prefix} }${subject.blob:+[${subject.blob}] }${subject.body:no subject}\nMM_TAGS=${##tags}\nMM_MESSAGE_ID=${message-id.split}\n';
  1. Modify the support bin/add script to read:
#!/bin/bash

tmpfile="$TMPDIR/8D444CFE-C1F9-4F82-AD8B-4EA7EC7A77C2.eml"
cat > "${tmpfile}"

# Tricky substitution to make it work for " and \ in the AppleScript below.
name=${MM_SUBJECT//\\/\\\\}
name=${name//\"/\\\"}
url="message://%3c${MM_MESSAGE_ID}%3e"
tags="${MM_TAGS}"

osascript <<END

try
	tell application id "DNtp"
		set theRecord to import "${tmpfile}" from "MailMate" name "${name}"
		set modification date of theRecord to creation date of theRecord
		set tags of theRecord to "${tags}"
		set the URL of theRecord to "${url}" # Allows it to be opened in MailMate using ⌃⌘O
	        display notification "Added message \"${name}\" to DEVONthink" with title "MailMate"
	end tell
on error errMsg number eNum
	tell application "System Events"
		activate
		display alert "DEVONThink: " & eNum message errMsg
	end tell
end try

END

unlink "${tmpfile}"

This works for single tags just fine, but, if a message has more than one tag (which is common for me), I’ll need to figure out how to split the tags correctly. I noticed when manually adding tags to DevonThink that hitting “tab” would complete one tag, and allow for entry of another, so, perhaps the solution is as simple as replacing spaces with tabs in MM_TAGS - EDIT - no, it’s not that easy, but the solution can’t be far off…

Solved.

Made the changes to the environment variable, and then modified the “add” script to read as follows and it’s working.

#!/bin/bash

tmpfile="$TMPDIR/8D444CFE-C1F9-4F82-AD8B-4EA7EC7A77C2.eml"
cat > "${tmpfile}"

# Tricky substitution to make it work for " and \ in the AppleScript below.
name=${MM_SUBJECT//\\/\\\\}
name=${name//\"/\\\"}
url="message://%3c${MM_MESSAGE_ID}%3e"
mm_tags="${MM_TAGS}"

osascript <<END

try
	tell application id "DNtp"
		set mm_tagsString to "${mm_tags}"
		set {TID, AppleScript's text item delimiters} to {AppleScript's text item delimiters, " "}
		set tagList to text items of mm_tagsString
		set theRecord to import "${tmpfile}" from "MailMate" name "${name}"
		set modification date of theRecord to creation date of theRecord
		set tags of theRecord to tagList
		set AppleScript's text item delimiters to TID	
		set the URL of theRecord to "${url}" # Allows it to be opened in MailMate using ⌃⌘O
	        display notification "Added message \"${name}\" to DEVONthink" with title "MailMate"
	end tell
on error errMsg number eNum
	tell application "System Events"
		activate
		display alert "DEVONThink: " & eNum message errMsg
	end tell
end try

END

unlink "${tmpfile}"
2 Likes

Good that it works for you now. I’d really not have tried to wade through this mix of bash and Applescript code.