Import emails and attachments separately, creating automatic groups and enabling space savings

[smiling]

Hi there,

Since there has been some work done on creating automations for this and similar tasks, I thought I’d write an official feature request here: it would be great if DT could import emails without integrating the attachments inside the emails, but linking to the attachments that would be separate items in DT. At the time of the import, the email and the attachment would be grouped together. The benefits (already mentioned here: Separate/import e-mail attachments for better search V2 - #28 by smiling) compared to today’s situation would be:

1. Separate attachments are treated with OCR and are therefore searchable
2. Bidirectional links between email and file, in particular a backlink between the file and the email
3. In addition, the email and the attachment would be grouped together
4. When the attachment gets actually removed from the eml, this allows for space savings through deduplication (e.g. all the forwarded mails with attachments will otherwise take up much more space)
5. Search works better in separate attachments, even indexed ones (shows where the search text was found)
6. The file names of separate attachments can be found when searched for (currently, the filenames are not indexed when inside a message)

For those interested in the feature today, @mdbraber and @AWD have done some great work to make it possible already today (I haven’t managed to get it to work yet, but I’m confident), relying heavily on scripts:

Separate/import e-mail attachments for better search V2

Last year I wrote a script to separate attachments from e-mails (.eml files). Recently I updated this script to make it more effective and efficient. The script does the following: Check (via Python script) each individual email for non-inline attachments that should be imported into DT. The script ignores e.g. .ics attachments or winmail.dat attachments - see the Python script for what it ignores. Convert the e-mail to RTF and check which attachments from the RTF match the found attachments …

Have a great day!

[cgrunenberg]

@smiling

it would be great if DT could import emails without integrating the attachments inside the emails, but linking to the attachments that would be separate items in DT.

Thanks for the suggestion! DEVONthink usually doesn’t change files on import, a future release might support this but no promises.

[smiling]

Thank you for your reply! I understand why you generally wouldn’t like to change files on import, I was just adding a data point in your user voice counter.

[smiling]

@stratadata made me think of another advantage:

@stratadata

But one thing I’ve realised today is that another benefit of extracting the attachments form the e-mails is not just decreased storage if I make duplicate attachments replicants (I’ll probably only do that selectively), but as base64 encoded data takes up 33% more space than the unencoded data, I will save some space, too!

Looking at the relative sizes of things, the imported attachments for those 777 e-mails total 732.2MB – base64 encoding would increase that to 976.3MB, so a saving of almost a quarter of GB of space on one mailbox alone.

I’ll add it in the original post if I can still change it. Edit: no, I can’t.

All the best!