Same mail with different mail ids in mail.app when running Apple Scripts from DEVONthink 3

Hi there,

I hope you can shed some light on a weird situation with mails in mail.app.

I recently started looking into automating the mail import from Apples Mail.app. I’m interested to import both the mail in EML-format and existing PDF-attachments.

Problem

I use a modified version of the standard mail scripts provided by DEVONthink. When I run them, the selection contains always two items, although only one mail is selected. Important: But for now I’m only aware, that there’s this single mail. I have not tried to find another one so far.

The code in in Script Debugger.app:

The selected mail:

I searched within the mail.app and the result is just this, so no duplicated mail in my mailbox.

I searched the settings of the mail.app to see if there’s a problematic option, but found none. Disabling “Include related messages” doesn’t fix the problem.

To my setup:
I use a remote mail provider and run a local mailserver which fetches mails from the remote one, if this might cause some of the problems.

[ mailprovider ] <--IMAP fetch & remote delete -- [internal mail server]

Could you please post the code as such, included in backticks:
```
code goes here
```
That makes it infinitely easier to read (as in: I don’t have to click on an image to see the code) and easy to copy/paste.

Sure.

-- Import attachments of selected messages to DEVONthink.
-- Created by Christian Grunenberg on Fri May 18 2012.
-- Copyright (c) 2012-2020. All rights reserved.

tell application "Mail"
	try
		tell application id "DNtp"
			if not (exists current database) then error "No database is in use."
		end tell
		
		set theSelection to the selection
		set theFolder to (POSIX path of (path to temporary items))
		
		if the length of theSelection is less than 1 then error "One or more messages must be selected."
		
		repeat with theMessage in theSelection
			set theSender to the sender of theMessage
			repeat with theAttachment in mail attachments of theMessage
				-- try
				if downloaded of theAttachment and (name of theAttachment ends with ".pdf") then
					set theFile to theFolder & (name of theAttachment)
					tell theAttachment to save in theFile
					tell application id "DNtp"
						set theAttachmentRecord to import theFile
						set URL of theAttachmentRecord to theSender
						perform smart rule trigger import event record theAttachmentRecord
					end tell
				end if
				-- end try
			end repeat
		end repeat
	on error error_message number error_number
		if error_number is not -128 then display alert "Mail" message error_message as warning
	end try
end tell

Modified in what way?

As to your problem: Perhaps you’re dealing with a mail thread here, consisting of two messages?

If I select this mail entry, the selection list has two elements. In any case, the problem is not DT here, as it just works with the selection in Apple’s Mail. If that app reports two elements, then that’s that.
(SInce you’re using list view, the small blue right-pointing arrow might be to the left of the subject. Or it’s not visible in dark mode?).

Sounds a bit contrived – why not have Apple’s Mail retrieve the mail from the remote server?

@chrillek

As to your problem: Perhaps you’re dealing with a mail thread here, consisting of two messages?

My first guess was, that the setting/threat thing might be the problem. But even without enabling the setting and choosing the single message, the problem occurs. But what’s really weird: The same message is send twice to the AppleScript script.

Modified in what way ?

Checking for the extension of the attachment. I’m only interested in PDF documents. The original code adds all attachments to DEVONthink.

If I select this mail entry, the selection list has two elements. In any case, the problem is not DT here, as it just works with the selection in Apple’s Mail.

Thanks for pointing that out. I was aware of that. I reached out to the community, in case someone had a similar problem with the mail.app before.

For now I check for duplicate mails reported by the mail.app. A quite ugly and somewhat imperformant implementation, but it works. Any suggestions to improve the code are highly welcome.

-- Import attachments of selected messages to DEVONthink.
-- Created by Christian Grunenberg on Fri May 18 2012.
-- Copyright (c) 2012-2020. All rights reserved.

tell application "Mail"
	try
		tell application id "DNtp"
			if not (exists current database) then error "No database is in use."
		end tell
		
		set theSelection to the selection
		set theFolder to (POSIX path of (path to temporary items))
		
		if the length of theSelection is less than 1 then error "One or more messages must be selected."
		set messageList to {}
		repeat with theMessage in theSelection
			set theSubject to subject of theMessage
			set theSender to sender of theMessage
			set theCurrentMessageItem to (theSender & theSubject)
			
			set theFoundItem to 0
			repeat with theMessageItem in messageList
				if (contents of theMessageItem) is equal to theCurrentMessageItem then
					set theFoundItem to 1
					exit repeat
				end if
			end repeat
			
			if theFoundItem = 1 then
				exit repeat
			end if
			
			set the |messageList| to the |messageList| & theCurrentMessageItem
			
			repeat with theAttachment in mail attachments of theMessage
				-- try
				if downloaded of theAttachment and (name of theAttachment ends with ".pdf") then
					set theFile to theFolder & (name of theAttachment)
					tell theAttachment to save in theFile
					tell application id "DNtp"
						set theAttachmentRecord to import theFile
						set URL of theAttachmentRecord to theSender
						perform smart rule trigger import event record theAttachmentRecord
					end tell
				end if
				-- end try
			end repeat
		end repeat
	on error error_message number error_number
		if error_number is not -128 then display alert "Mail" message error_message as warning
	end try
end tell

Sounds a bit contrived – why not have Apple’s Mail retrieve the mail from the remote server?

I prefer to be not dependent on a single OS/Application. Some time ago I used Linux + Thunderbird. Switching to MacOS + Mail.app was easy for mails. I just needed to configure my internal mail server (IMAP). And for the “running cost”, I don’t like to be dependent on the external mail provider either - to paranoid. :slight_smile:

Did you check the source of the selected message? Also, instead of comparing the file extension to pdf, I’d go for the MIME-type.

And then you could, of course, find the two messages with a script like this (JavaScript, I don’t do AS):

(() => {
  const app = Application("Mail");
  const mailbox = app.accounts['Bru6'].mailboxes['INBOX'];
  [247497,246889].forEach(messageID => {
    const message = mailbox.messages.whose({id: messageID})[0];
    console.log(`${messageID}: ${message.subject()} from ${message.sender()} at ${message.dateReceived()}`);
  })
})()

And here’s the AS equivalent

tell application "Mail"
	set mbox to mailbox "INBOX" of account "DEVONthink"
	set msgIDs to {29870, 29891}
	repeat with mid in msgIDs
		set msg to item 1 of (messages of mbox whose id is mid)
		log (mid & ": " & subject of msg & " from " & sender of msg & " at " & date received of msg) as string
	end repeat
end tell

@chrillek Maybe the reason for this is, it’s a mail containing a plaintext and html view of the message.

--0e33c8288f42879da587dd802accf2513969e42d39224b4e34c36b2b906f
Content-Type: multipart/alternative; boundary=c0a358084674dc6eee37dafde0e1fb81ceaac6c6b0b61c6798afa775ffb6

--c0a358084674dc6eee37dafde0e1fb81ceaac6c6b0b61c6798afa775ffb6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0

Vielen Dank f=FCr Ihren Einkauf.
Sie finden die Rechnung im Anhang dieser Mail im PDF Format.
Wir hoffen, Sie bald wieder als Kunden begr=FC=DFen zu d=FCrfen.

--c0a358084674dc6eee37dafde0e1fb81ceaac6c6b0b61c6798afa775ffb6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=iso-8859-1
Mime-Version: 1.0

<html>Vielen Dank f=FCr Ihren Einkauf.<br/> Sie finden die Rechnung im Anha=
ng dieser Mail im PDF Format.<br/> Wir hoffen, Sie bald wieder als Kunden b=egr=FC=DFen zu d=FCrfen.</html>

--c0a358084674dc6eee37dafde0e1fb81ceaac6c6b0b61c6798afa775ffb6--

For now this is my script I’m now quite happy with. I switched to MIME types following the suggestions in this thread: Script: Import mail attachments filtered by MIME type - #4 by pete31. It’s a mixture from multiple “base” scripts from DEVONthink. To handle the missing “continue”-statement for “repeat” in AppleScript I moved to separate functions using guard clauses. For now I link the original mail and the attachment visually via Author-Mail_ID-Subject within the record’s name.

-- Import attachments of selected messages to DEVONthink.
-- Created by Christian Grunenberg on Fri May 18 2012.
-- Copyright (c) 2012-2020. All rights reserved.
property pNoSubjectString : "(no subject)"

tell application "Mail"
	try
		tell application id "DNtp"
			if not (exists current database) then error "No database is in use."
		end tell
		
		set theSelection to the selection
		
		if the length of theSelection is less than 1 then error "One or more messages must be selected."
		
		set messageList to {}
		repeat with theMessage in theSelection
			my parseMail(theMessage, messageList)
		end repeat
	on error error_message number error_number
		if error_number is not -128 then display alert "Mail" message error_message as warning
	end try
end tell

on parseMail(theMessage, messageList)
	tell application "Mail"
		if my isUnique(theMessage, messageList) is false then return
		
		set theMailRecord to my importMail(theMessage)
		
		repeat with theAttachment in mail attachments of theMessage
			my parseAttachment(theAttachment, theMailRecord, theMessage)
		end repeat
	end tell
end parseMail

on importMail(theMessage)
	tell application "Mail"
		tell theMessage
			set {theMessageId, theDateReceived, theDateSent, theSender, theSubject, theSource, theReadFlag} to {the id, the date received, the date sent, the sender, subject, the source, the read status}
		end tell
	end tell
	
	if theSubject is equal to "" then set theSubject to pNoSubjectString
	
	tell application id "DNtp"
		set theRecord to create record with {name:theSubject & ".eml", type:unknown, creation date:theDateSent, modification date:theDateReceived, URL:theSender, source:(theSource as string), unread:(not theReadFlag)}
		perform smart rule trigger import event record theRecord
		
		set theMetaData to meta data of theRecord
		set theAuthor to |kMDItemAuthors| of theMetaData
		
		if theAuthor is not "" then
			set theSender to theAuthor
		end if
		
		set theNewName to theSender & "-" & theMessageId & "-" & theSubject
		set the name of theRecord to theNewName
	end tell
	
	return theRecord
end importMail

on parseAttachment(theAttachment, theMailRecord, theMessage)
	tell application "Mail"
		if not downloaded of theAttachment then return
		
		set theFolder to (POSIX path of (path to temporary items))
		set theFile to theFolder & (name of theAttachment)
		tell theAttachment to save in theFile
		
		if my isPDF(theFile) is false then return
		
		my importAttachment(theFile, theMailRecord, theMessage)
	end tell
end parseAttachment

on importAttachment(theFile, theMailRecord, theMessage)
	tell application "Mail"
		tell theMessage
			set {theMessageId, theDateReceived, theDateSent, theSender, theSubject, theSource, theReadFlag} to {the id, the date received, the date sent, the sender, subject, the source, the read status}
		end tell
	end tell
	
	tell application id "DNtp"
		set theMetaData to meta data of theMailRecord
		set theAuthor to |kMDItemAuthors| of theMetaData
		
		if theAuthor is not "" then
			set theSender to theAuthor
		end if
		
		set theAttachmentRecord to import theFile
		set unread of theAttachmentRecord to (not theReadFlag)
		set URL of theAttachmentRecord to theSender
		
		set theOldName to the (name of theAttachmentRecord)
		set name of theAttachmentRecord to theSender & "-" & theMessageId & "-" & theSubject & "-" & theOldName
		perform smart rule trigger import event record theAttachmentRecord
	end tell
end importAttachment

on isUnique(theMessage, messageList)
	tell application "Mail"
		tell theMessage
			set {theDateSent, theSender, theSubject} to {the date sent, the sender, subject}
		end tell
	end tell
	
	set theCurrentMessageItem to (theSender & theSubject & theDateSent)
	
	repeat with theMessageItem in messageList
		if (contents of theMessageItem) is equal to theCurrentMessageItem then
			return false
		end if
	end repeat
	
	set end of messageList to theCurrentMessageItem
	
	return true
end isUnique

on isPDF(thePath)
	set theMimeType to do shell script "file -b --mime-type " & quoted form of thePath
	
	if theMimeType is "application/pdf" then
		return true
	end if
	
	return false
end isPDF

No. I tried that, too. Different mail parts still have the same message ID. You have two messages. I’d still like to get to the base of this instead of working around it.

As to your MIME-type code: There are simpler ways to achieve “von hinten durch die Brust ins Auge”. mime type is a property of every mail attachment. Saving the attachment as a file and then using the file command might work (depending on the file type), but why not use what Mail already provides you with instead?
Apparently, accessing Apple’s mime type property doesn’t work (not even in AppleScript). So, your approach to determining the MIME type is best. What a mess.

Perhaps using JavaScript and its methods and datatypes alleviates some of the pain. For example, instead of looping over all messages in isUnique, you could use a Set which by definition cannot contain duplicate elements.

Aside: Did you try my sample code to figure out what Mail delivers as selection with two elements when you select one mail?

Aside: Did you try my sample code to figure out what Mail delivers as selection with two elements when you select one mail?

Yeah. Thanks for providing that code. I tried it. I delivers the very “same” mail. At least when I have a look at the contents.

Perhaps using JavaScript and its methods and datatypes alleviates some of the pain. For example, instead of looping over all messages in isUnique , you could use a Set which by definition cannot contain duplicate elements.

I had a look at that too and decided, to stay with AppleScript for now. Getting into JXA is on the list for my future self.

That is really weird. And shouldn’t happen, unless you really get sent two messages (which still wouldn’t explain why selecting one makes Mail’s selection property contain two). I’d probably write the complete content of those two incantations to files and see if they differ in any way.

But the message IDs are different from your first screenshot… Does that mean that the issue happens with more than one message? Always from the same sender? Did you check if your local IMAP server has these two messages? (I guess you can poke around in it’s folders :wink: )

But the message IDs are different from your first screenshot… Does that mean that the issue happens with more than one message? Always from the same sender?

Nope. it’s the same mail by content. I added some more code to the script to archive the selected mail by moving them to a different mailbox. After moving the mail back, it had two different IDs.

I’d probably write the complete content of those two incantations to files and see if they differ in any way.

Somehow Mail.app hides duplicate mails. I still have the Thunderbird.app installed and checked the mail. Within Thunderbird there are two mails which look the same.

Thanks for pushing me further to get to the reason for that issue.

I did some more research: By changing this setting Mail.app shows duplicate mails:

Michael Tsai - Blog - Showing Duplicate Messages in Apple Mail - at the very end of the article:

$ defaults write com.apple.mail _AlwaysShowDuplicates -bool true

Thanks @chrillek for your help.

This is for now the final version of the script which:

  • Adds the mail to DEVONthink
  • Adds all PDF documents to DEVONthink - based on MIME-type
  • Archives added mails based on the year the mail was received
  • Handles duplicates in the selection (by not parsing/moving them)
  • Takes ~1:20 min for 194 mails including 4 duplicates - removing the duplicate check does not reduce the runtime significantly
  • Logs found duplicates and sanitises the log output

If you’re interested using this script:

  1. Create a file at “~/Library/Scripts/Applications/Mail/Add mail and PDF attachments to DEVONthink.scpt” with the content given below.

  2. Open Mail.app

  3. Select mails

  4. Run script from global script menu

  5. Mails from selection which are left over might be duplicates by date sent, subject and sender

-- Import attachments of selected messages to DEVONthink.
-- Created by Christian Grunenberg on Fri May 18 2012.
-- Copyright (c) 2012-2020. All rights reserved.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

property pNoSubjectString : "(no subject)"
property pMailboxAccount : "CHANGE ME"
-- property pArchiveMailbox : "Archive"
property pArchiveMailbox : "Archive.Test"

tell application "Mail"
	try
		tell application id "DNtp"
			if not (exists current database) then error "No database is in use."
		end tell
		
		set theSelection to the selection
		if the length of theSelection is less than 1 then error "One or more messages must be selected."
		
		my createMailboxForThisYear(pMailboxAccount, pArchiveMailbox)
		
		set messageList to {}
		repeat with theMessage in theSelection
			my parseMail(theMessage, messageList, pMailboxAccount, pArchiveMailbox)
		end repeat
	on error error_message number error_number
		if error_number is not -128 then display alert "Mail" message error_message as warning
	end try
end tell

on createMailboxForThisYear(theMailboxAccount, theArchiveMailbox)
	tell application "Mail"
		set theYear to (year of (get current date))
		make new mailbox in account theMailboxAccount with properties {name:theArchiveMailbox & "/" & theYear}
	end tell
end createMailboxForThisYear

on archiveMail(theMessage, theMailboxAccount, theArchiveMailbox)
	tell application "Mail"
		tell theMessage
			set {theDateReceived, theDateSent} to {the date received, the date sent}
			set theYear to (year of theDateReceived)
		end tell
	end tell
	
	tell application "Mail"
		set the read status of theMessage to true
		set mailbox of theMessage to mailbox (theArchiveMailbox & "/" & theYear) of account theMailboxAccount
	end tell
end archiveMail

on parseMail(theMessage, messageList, theMailboxAccount, theArchiveMailbox)
	if my isUnique(theMessage, messageList) is false then return
	
	tell application "Mail"
		set theMailRecord to my importMail(theMessage)
		
		repeat with theAttachment in mail attachments of theMessage
			my parseAttachment(theAttachment, theMailRecord, theMessage)
		end repeat
	end tell
	
	my archiveMail(theMessage, theMailboxAccount, theArchiveMailbox)
end parseMail

on importMail(theMessage)
	tell application "Mail"
		tell theMessage
			set {theMessageId, theDateReceived, theDateSent, theSender, theSubject, theSource, theReadFlag} to {the id, the date received, the date sent, the sender, subject, the source, the read status}
		end tell
	end tell
	
	if theSubject is equal to "" then set theSubject to pNoSubjectString
	
	tell application id "DNtp"
		set theRecord to create record with {name:theSubject & ".eml", type:unknown, creation date:theDateSent, modification date:theDateReceived, URL:theSender, source:(theSource as string), unread:(not theReadFlag)}
		perform smart rule trigger import event record theRecord
		
		set theMetaData to meta data of theRecord
		try
			set theAuthor to |kMDItemAuthors| of theMetaData
		on error
			set theAuthor to |kMDItemAuthorEmailAddresses| of theMetaData
		end try
		
		set theNewName to theAuthor & "-" & theMessageId & "-" & theSubject
		set the name of theRecord to theNewName
	end tell
	
	return theRecord
end importMail

on parseAttachment(theAttachment, theMailRecord, theMessage)
	tell application "Mail"
		if not downloaded of theAttachment then return
		
		set theFolder to (POSIX path of (path to temporary items))
		set theFile to theFolder & (name of theAttachment)
		tell theAttachment to save in theFile
		
		if my isPDF(theFile) is false then return
		
		my importAttachment(theFile, theMailRecord, theMessage)
	end tell
end parseAttachment

on importAttachment(theFile, theMailRecord, theMessage)
	tell application "Mail"
		tell theMessage
			set {theMessageId, theDateReceived, theDateSent, theSender, theSubject, theSource, theReadFlag} to {the id, the date received, the date sent, the sender, subject, the source, the read status}
		end tell
	end tell
	
	tell application id "DNtp"
		set theMetaData to meta data of theMailRecord
		set theUrl to URL of theMailRecord
		
		try
			set theAuthor to |kMDItemAuthors| of theMetaData
		on error
			set theAuthor to |kMDItemAuthorEmailAddresses| of theMetaData
		end try
		
		set theAttachmentRecord to import theFile
		set unread of theAttachmentRecord to (not theReadFlag)
		set URL of theAttachmentRecord to theUrl
		
		set theOldName to the (name of theAttachmentRecord)
		set name of theAttachmentRecord to theAuthor & "-" & theMessageId & "-" & theSubject & "-" & theOldName
		perform smart rule trigger import event record theAttachmentRecord
	end tell
end importAttachment

on isUnique(theMessage, messageList)
	tell application "Mail"
		tell theMessage
			set {theDateSent, theSender, theSubject} to {the date sent, the sender, subject}
		end tell
	end tell
	
	set theCurrentMessageItem to (theSender & theSubject & theDateSent)
	
	repeat with theMessageItem in messageList
		if (contents of theMessageItem) is equal to theCurrentMessageItem then
			log "[WARN] msg=\"Duplicate in selection detected\" sender=\"" & my sanitizeString(theSender) & "\"" & " date-sent=\"" & (the short date string of theDateSent) & " " & (the time string of theDateSent) & "\" subject=\"" & my sanitizeString(theSubject) & "\""
			return false
		end if
	end repeat
	
	set end of messageList to theCurrentMessageItem
	
	return true
end isUnique

on isPDF(thePath)
	set theMimeType to do shell script "file -b --mime-type " & quoted form of thePath
	
	if theMimeType is "application/pdf" then
		return true
	end if
	
	return false
end isPDF

on sanitizeString(theString)
	set theCleanedString to do shell script "echo " & quoted form of theString & "|sed \"s/[^[:alnum:][:space:]._-]//g\""
	return theCleanedString
end sanitizeString