How does Archiving email survives a crash?

Hi!
(DT3.6.3, Big Sur 11.1, Mail.app 14.0)

I hope you are safe and well! :slight_smile:

Another question about email archiving: I am trying to archive an entire office 365 email account, which Mail.app (which, in Mail.app, right-click on account > Get Account Info) reports containing just North of 63.4K emails, 5.01GB. I have tried a few times, but have not been able to go through archiving it completely without something crashing, either Mail.app, DT3, or something else. Troubleshooting has been difficult because neither the log in DT3 (which seems to start from zero after a crash!), nor the system’s console have been particularly helpful. The system’s console does tend to show memory access problems, but it’s pretty dense and inconclusive to my naked eyes.
(5GB of emails may be a lot, but I am not sure why that would cause a crash. Also the whole process takes between 15-20h!)

My question is simple, accepting that I won’t necessarily understand what causes the crashes and accepting that tech is not bulletproof just yet, how does the archiving function resumes after a crash?
Can I be 100% certain that clicking “Archive” again on that account, will simply resumes where it stopped? (in my experimentation, it happened once that archiving simply flattened the folders and lost the structure, which was odd)

I want to archive to keep the structure, and would like to avoid having to import every single folder manually.

Related to that question, after I have done the above a few times, supposedly having archived everything, the number of items reported in Mail.app vs DT3 archive tab vs DT3 database folders, are not the same. How can I be certain I have archived everything that needs to be archived?
I am guessing DT3 counts folders as items, and other subtleties bias the count?

Thanks!

Kind regards,

Etienne

Not an immediate answer to your question. But what you describe is one of the reasons why I started the poll below and posted it earlier today. Apparantly most people up to now think an IMAP engine in DT wouldn’t be a solution or think it’s not necessary otherwise.

That said: I’ve worked around crashes like yours in the past by breaking up the import in several parts. For example by sorting the e-mails by year and placing them in a corresponding folder on the IMAP server. I then imported every year in sequence by hand.

I can concur. Either with or without plugin and full disk access, importing is very slow in comparison to the time it takes Mail to import the same e-mails from the server over the internet.

Welcome @eroesch

  1. After a crash and relaunch, DEVONthink will verify the database to make sure it’s intact.
  2. Archiving in smaller batches is a wiser idea than trying to force-ingest everything in one go… as you can see here but an unexpected crash stumbling the import.
  3. Emails are imported with a specific UUID. This keeps them from being imported more than once in the same database.
  4. This UUID is likely responsible for your discrepancy in email counts as if you have duplicates files in other submailboxes, they will not be imported into the same database again (and yes, that is by design).

Thanks a ton, @BLUEFROG.

  1. Archiving in smaller batches is a wiser idea than trying to force-ingest everything in one go… as you can see here but an unexpected crash stumbling the import.

You mean “importing in smaller batches”, right? I don’t believe archiving allows you to select the folders to import, unless I didn’t understand?

After another unsuccessful afternoon of wanting to force-feed 5GB of emails to DT3 through “archiving”, I have decided to give up, and try “importing” in batches instead. Hopefully, that’ll help.

Before I do that, suppose I manage to import and reproduce the original structure of the folders on the account, and that I import into the Apple Mail folder at the root of my database (which has been created by the archive process earlier), … would that interfere or support my using the archive function again, in the future, on the whole account?

(that said, “importing” would flatten the structure before importing, right?)

Have a great productive happy week with DT3, everyone!

PS: and yes, I think a quick imap service would be useful, and filled in the survey.

Etienne

1 Like

You can only archive a mailbox. Folders, no. But submailboxes can be archived, and we suggest it when you have a large number of emails.

As per the Help > Documenation > In & Out > Archiving Email

I hope that’s clear.

You can also try disabling the two Conversations options in Preferences > Email to better performance.

Grand, @BLUEFROG! That clarified a lot! Thanks a ton! Somehow I missed that part of the help!
I also think that, I am/was not clear between what outlook understands to be a folder vs a (sub)mailbox.

I have started archiving what I now understand to be a submailbox, i.e. what I called a folder earlier. It does complete fast enough, and it does keep the folder structure in the database. Additionally, it does seem to allow for incremental archiving, one submailbox at a time. Great! :slight_smile:

(but it did crash once already, … possibly because I was using Mail.app on the same account that was being syphoned into DT3)

I will carry on archiving piecemeal, and checking as the process is going.

Thanks a lot for the prompt feedback!

Keep safe.

Etienne

Glad to be of service and you’re welcome!

Just remember, subsequent uses of the Archive Mailbox function will only import new emails and would of course be much faster :slight_smile:

Hi! Closing this thread. In summary: I have successfully archived that whole office365 account, every email 2016-2021 (through mail.app). Archiving the whole account in one go did not work, with either mail or/and usually both dt3 crashing. I split the mailboxes (which I came to understand are the folders shown in outlook) per year, and archived though instead. Archiving kept the structure; it took a bit of time, but I managed to archive everything. :slight_smile:

Glad to hear it. Thanks for the follow-up.