Script for automatic email import from mail server

Hello,

in the “Requests & Suggestions” I asked for the possibility to import email directly from an IMAP-mailserver to DEVONthink.

This would improve my workflow for all kinds of personal notes, because I could send or save notes to my mail INBOX or Drafts.
Using the email-program or its web-interface is quite common to me. One could also use it when you’re on the go. Just use the smartphone, without need of special software.

Because there seems to be no solution at the moment, I started coding a tiny script to do this job. It’s made, to import emails from an IMAP-server and save them to the DEVONthink Inbox.
Not all emails should be imported, but the ones marked with "N: " at the beginning of the email-subject.

Here is my first shot. Maybe here is some one interested in helping to improve it.
Vinz

It does:

  • Connect to my IMAP server and download all emails with “Note …” in subject.
  • Store them as textfiles to the DEVONthink Inbox for further handling.
  • Subject is the filename. Content is the content.

But be aware:

  • It re-downloads emails, that already have been downloaded.
  • It overwrites files with same name in Inbox.
  • There is no error handling.
  • There are names, passwords and absolute path in the script.
  • This is for testing only.

To improve:

  • Save text or html
  • Save attachments, especially PDFs
# Python
import imaplib, email
import getpass # maybee use of: mail.login(getpass.getuser(), getpass.getpass())
import os, re, datetime 

# = = = = = = = = = = = = = = = = = = = = = = = = = = = =
# This program connects to your IMAP-server and downloads
# emails with subject beginning with "Note " to DEVONthink.
#
# Disclaimer: This is for amusement only. Never execute
# this on any system!
# 2011-11-13 Vinz
# 2011-11-18 Vinz minor improvement


# = = = = = = = = = = = = = = = = = = = = = = = = = = = =
# function to get email plaintext
def get_first_text_block(email_message_instance):
    maintype = email_message_instance.get_content_maintype()
    if maintype == 'multipart':
        for payload in email_message_instance.get_payload():
            if payload.get_content_maintype() == 'text':
                return payload.get_payload()
    elif maintype == 'text':
        return email_message_instance.get_payload()

# = = = = = = = = = = = = = = = = = = = = = = = = = = = =
# function to get emails: Subject beginning with "N: " and maximum 30 days old
def get_my_notes(mail_instance):
	date = (datetime.date.today() - datetime.timedelta(30)).strftime("%d-%b-%Y")
	result, data_mails = mail_instance.uid('search', None, '(SENTSINCE {date} HEADER Subject "N:")'.format(date=date))
	email_uids = data_mails[0].split()
	ret_val = []
	for email_uid in email_uids:
		print email_uid
		result, data_onemail = mail_instance.uid('fetch', email_uid, '(RFC822)')
		raw_email = data_onemail[0][1]
		email_message = email.message_from_string(raw_email)
		email_subject = email_message['Subject']
		email_text = get_first_text_block(email_message)
		ret_val += [[email_subject, email_text]]
	return ret_val

# = = = = = = = = = = = = = = = = = = = = = = = = = = = =
# function to save my notes to DEVONthink
def save_notes(my_notes):
	i = 0
	for note in my_notes:
		filename = note[0]
		filename = re.sub("^N[: ]\s*(?i)", "", filename) 
		filename = re.sub(r"[\/\\\:\*\?\"\<\>\|.]", '_', filename)
		content = note[1]
		
		if re.match("\<html\>(?i)", content):
			filename = filename + ".html"
		else:
			filename = filename + ".txt"
		
		inbox_path = os.path.join(os.environ['HOME'], "Library/Application Support/DEVONthink Pro 2/Inbox/")
		if not os.path.exists(inbox_path):
			print "Can't find DEVONthink INBOX. Check path in script."
			return
		filepath = os.path.join(inbox_path, filename)
		filepath = unique_filename(filepath)
		
		f = open(filepath, 'w+')
		f.write(content)
		f.close()
		print "Saved new note: " + filename
		i+=1
	print "{0} new messages saved.".format(i)

# = = = = = = = = = = = = = = = = = = = = = = = = = = = =
# function to create a unique filename
# may have security issues: see
# http://stackoverflow.com/questions/183480/is-this-the-best-way-to-get-unique-version-of-filename-w-python
def unique_filename(file_name):
	counter = 1
	file_name_parts = os.path.splitext(file_name) # returns ('/path/file', '.ext')
	while os.path.lexists(file_name): 
		file_name = file_name_parts[0] + '_' + str(counter) + file_name_parts[1]
		counter += 1
	return file_name
	
	

# = = = = = = = = = = = = = = = = = = = = = = = = = = = =
# main program
# connect to imap server inbox
mail = imaplib.IMAP4_SSL('imap.domain.com')
mail.login('your.name@domain.com', 'your_password')
mail.select("INBOX")
# mail.select("Drafts")

# get my notes an save to DEVONthink
notes = get_my_notes(mail)
save_notes(notes)

mail.close()
mail.logout()

There’s another way.

Using Christian’s script, below (there is a compiled version of this in the DEVONthink installation folder Extras > Scripts > Mail), create an Inbox Rule in Mail that looks at whatever combination of factors you want (e.g., look only at a particular mailbox, or subjects containing a particular prefix, etc.) - then perform the following AppleScript. I realized you wanted to scrape your IMAP mail file directly, but I think this solution gets you to the same point by having Mail monitor itself.

SAMPLE RULE

SCRIPT

-- Mail Rule Action Import to DEVONthink Pro.
-- Created by Christian Grunenberg on Mon Apr 19 2004.
-- Copyright (c) 2004-2009. All rights reserved.

-- this string is used when the message subject is empty
property pNoSubjectString : "(no subject)"

using terms from application "Mail"
   on perform mail action with messages theMessages for rule theRule
       tell application "Mail"
           repeat with theMessage in theMessages
               my importMessage(theMessage)
           end repeat
       end tell
   end perform mail action with messages
end using terms from

on importMessage(theMessage)
   tell application "Mail"
       try
           set theDateReceived to the date received of theMessage
           set theDateSent to the date sent of theMessage
           set theSender to the sender of theMessage
           set theSubject to subject of theMessage
           if theSubject is equal to "" then set theSubject to pNoSubjectString
           set theSource to the source of theMessage
           set theReadFlag to the read status of theMessage
           tell application id "com.devon-technologies.thinkpro2"
               set theRecord to create record with {name:theSubject & ".eml", type:unknown, creation date:theDateSent, modification date:theDateReceived, URL:theSender, source:(theSource as string)} in incoming group
               set unread of theRecord to (not theReadFlag)
           end tell
       end try
   end tell
end importMessage

This particular script doesn’t do attachments, but that should be easy to add. Or, the compiled version that currently ships with DEVONthink will display in-line images. Use whichever version you wish.

this looks like a good idea! But I made a rule just like you have in the screenshot but nothing happened. I am using “mail act-on” so I can make a rule and then activate with a shortcut key to test it out. But nothing happened in DT Pro 2.3.1.1 on a mac osx 10.6.8 using mail.app 4.5

Hello korm,
thanks very much for this hint.

On my system there is ```

~/Library/Scripts/Applications/Mail/Add message(s) to DEVONthink.scpt


So I copied your attachment to ```

~/Library/Scripts/Applications/Mail/Import Mail to DEVONthink Pro.scpt

``` and added the rule in Mail.

And yes, this works very well. :-)
I like this, because it seems to be the better solution. Later on maybe I’ll extend this script.

Thank you and kind regards
Vinz

P.S: Is there a website you got Christian's script from? Maybe Christian already has a new version with attachments? :)

I would like to use this script, but I would want to route the messages to the inbox of a dedicated database (instead of importing them to DtP’s general inbox).
Please, what would be the syntax then? I’m a script doofus, I beg your pardon!

I tried the script and the whole email (including the attachments) got imported. Did this with pdfs attached, jpgs attached. Please, what do you mean with “doesn’t do attachments”?

Kind regards,
Bernd

The script tells DEVONthink to put the email message in “the incoming group”. This is the destination set in DEVONthink > Preferences > Import:

You can do one of these:

  • Change the preference to “Inbox of current database”. Then, before you use the script select make sure your email database is selected in DEVONthink.
  • Change the preference to “Select group”. Then, when you use the script a display similar to “Tools > Show Groups & Tags” will pop up and ask for a destination.
  • Leave your current preference alone (you’re using "Global Inbox) and change the code in the script from [size=85]```

set theRecord to create record with {name:theSubject & “.eml”, type:unknown, creation date:theDateSent, modification date:theDateReceived, URL:theSender, source:(theSource as string)} in incoming group

[/size] to [size=85]

set theRecord to create record with {name:theSubject & “.eml”, type:unknown, creation date:theDateSent, modification date:theDateReceived, URL:theSender, source:(theSource as string)} in display group selector

* Hard code your destination group into the script.


The first two options apply to all imports until you change them.  The third option is very flexible, but you might get tired of seeing the group selector pop up.  The last option requires you to keep your code up to date.  (I don't have time, now, to suggest that code.  Maybe later.)

If attachments are working, I'm happy for you.  :wink:

korm, thank you!

The last listed option is what I am looking for, so I would need to define the path to the inbox of the database in the script and I don’t know how to.

I file my mail into two different databases, one is “private”, the other is “work”. I would like to fully automize this - new incoming email included-, so that I would ideally never have to spend time in Apple Mail itself. And of course I do not want to define a location everytime new mails come in.

So, if you find the time someday to give me the needed code, I would love to try a new flow for email here.

Kind regards,
Bernd

PS: And yes, attachments are working for me, just checked with other formats additionally (rtf/doc/tif). Hope I do not miss something here.

Hello Bernd

I’m not the Applescript expert, so I still have not figured out how to select the correct database.

But meanwhile I found out, how to store the imported mails to a specific DEVONthink-group (the folders).

This script creates a new Group “MailImport” inside your Database and saves the mail there. You could use two scripts. One with “Private”-Group and one with “Work”-Group.

Maybe you backup you data before testing my scripts. :wink:

Gruess
Vinz

-- Mail Rule Action Import to DEVONthink Pro.
-- Created by Christian Grunenberg on Mon Apr 19 2004.
-- Copyright (c) 2004-2009. All rights reserved.

-- this string is used when the message subject is empty
property pNoSubjectString : "(no subject)"

using terms from application "Mail"
   on perform mail action with messages theMessages for rule theRule
	   tell application "Mail"
		   repeat with theMessage in theMessages
			   my importMessage(theMessage)
		   end repeat
	   end tell
   end perform mail action with messages
end using terms from

on importMessage(theMessage)
   tell application "Mail"
	   try
		   set theDateReceived to the date received of theMessage
		   set theDateSent to the date sent of theMessage
		   set theSender to the sender of theMessage
		   set theSubject to subject of theMessage
		   if theSubject is equal to "" then set theSubject to pNoSubjectString
		   set theSource to the source of theMessage
		   set theReadFlag to the read status of theMessage
		   tell application id "com.devon-technologies.thinkpro2"
		   
			   set destination_group_location to "/MailImport"
			   if not (exists record at destination_group_location) then
			      set destination_group to create location destination_group_location
			   else
			      set destination_group to get record at destination_group_location in current database
			   end if
			   set theRecord to create record with {name:theSubject & ".eml", type:unknown, creation date:theDateSent, modification date:theDateReceived, URL:theSender, source:(theSource as string)} in destination_group

			   set unread of theRecord to (not theReadFlag)
		   end tell
	   end try
   end tell
end importMessage
1 Like

Hey Vinz,

thank you for sharing your script (works fine) and for your input!

Your idea would work for me, if I would use only one database, but I usually have at least four databases open.
Your script chooses the database last selected in DtPO while new mails rush in all the time, so I would have no control where my mails go to.

My problem remains: I need to define the database in the script.

Kind regards,
Bernd

Just replace…


            set destination_group_location to "/MailImport"
            if not (exists record at destination_group_location) then
               set destination_group to create location destination_group_location
            else
               set destination_group to get record at destination_group_location in current database
            end if

…with….


	-- Ensure that the database is open
	set destination_database to open database "/path/to/my/database.dtBase2"
	set destination_group to create location "/My Messages" in destination_database

Note: The “create location” creates a new group only if necessary.

Super, thank you, Christian!

Works like charm and it even opens the database when closed before. This is what I needed. Don’t know how I will like the method in the long run, but I love the idea to have new mail directly filed into DtPO and process from there.
Now I have all mail in one “place”, no more searching in different apps, I can label to-dos there, smart lists show me old and new mails from my most important contacts and accounts no matter where they are. Hallelujah! :smiley:

Now I only need to remember to file outgoing mails.

Kind regards,
Bernd