email importing is slow and keeps locking up

DEVONthink Tips
1

I am trying to import my emails into devo office and I am having a hard time. I do have the latest version of the beta and have installed the tools. I am importing from both apple mail and entourage. I am trying to bring in about 2k from apple and 5k from entourage. The app starts and gets about half of the emails in and then stops for way too long. I end up having to force quit devo and try again. Is there a limit to the amount of emails you should bring in at once?

2

There is no limit, other than a practical one based on the amount of RAM you have. But the numbers you are quoting are quite reasonable.
Sometimes it may take a while to get something done, but patience is a virtue here. There is a lot of error detection going on and eventually you should get a time-out if something goes badly wrong.
Here are some things that may help:

  1. Run a “Backup and Optimize” before you import a large amount of email.
  2. The default setting is not to “Import whole conversations”. Even though this is a cool feature IMHO, people complained that it slows down the import too much.

I have also had a couple of reports about certain “haxies” that may cause problems, such as ShapeShifter and some Mail bundles. Since the massive import is a one time thing, you may want to consider disabling these for that one big import.

You can also contact support@devon-technologies.com with the contents related to our app in the Console (Applications > Utilities).

3

The mail import process locks up on me too. I’m trying to import a few thousand messages from an Exchange 2003 based e-mail account, and the process locks up about 10 minutes in while loading messages.

MacBookPro, Intel Core 2 Duo 2.33 GHz, 2 GB RAM

4

I had the same problem, over & over. However, it seems that everytime it locks up, if i manually Force Quit the “DevonDataHelper” process in Activity Monitor (but leave DTProOffice running), DTProO reopens the thread and finishes the task without losing your progress. Seems to be the only way I can even import more than 100 messages at a time. My DT database was only a gig or so before this, shouldn’t be too bloated.

5

thank you for the tip. Cant’ believe you have to go through this to get the application to work as advertised.

6

This is a rare problem and I have an educated guess it has to do with the HTML to RTF conversion. Do you have “Download remote images in HTML messages” enabled in the Mail preferences of DTPO?
Secondly, since I can’t reproduce this (see my related post where I imported ~100k messages), if you have this problem there is something you can do to help us.
Please run the Activity Monitor in Applications > Utilities. One after another select DTPO and DevonDataHelper, and for each application click the “Trace” button in the toolbar. Save the results of each window that comes up. Then send these files to support@devon-technologies.com, this may help us to detect what kind of problem this is.
Thank you very much!

7

hiya,

first post and all i’m doing is reviving an old post, sorry… :stuck_out_tongue:

i’m trying to import several mail archives into devonthink pro office that so far have been all over the place. unfortunately, i have the same problem. i don’t mind it taking ages to import a few thousand messages but it keeps hanging altogether as soon as i try to import more than a few hundred messages at the time.
in the system and console logs i can see that the devondatahelper keeps crashing:

Aug 4 09:31:40 huit crashdump[1095]: DevonDataHelper crashed
Aug 4 09:31:41 huit crashdump[1095]: crash report written to: /Users/MyUsername/Library/Logs/CrashReporter/DevonDataHelper.crash.log

sometimes it recovers by itself and eventually finishes the import, but sometimes it keeps just consuming cpu cycles for hours without doing anything until i kill it. in such a case, should i just keep it running and wait for it to continue??? when that happened the last time, i kept an eye on the activity monitors ‘Disk Activity’ and found that there was none over a long period. so it looked really dead…

any ideas?

cheers,
olaf

8

If it happens again, use Activity Monitor (in Applications > Utilities) to create a “Trace” (it’s one of the buttons on the toolbar) of DEVONthink Pro Office (you have to select the application first). In the new window that appears, save its contents and email it to support@devon-technologies.com. Then we can take a look at what’s going on.

9

hi annard,

sorry for being a bit slow :blush: , but i can’t find any “Trace” button in the Activity Monitor. there’s “Inspect” and i can add a “Sample Process” button.

in any case, should i do trace the “DEVONThink Pro” or “DevonDataHelper” process?

cheers,
olaf

10

My fault: it’s “Sample Process”. You can do it on both programs, that would help a lot and Mail as well. The more info the better. :wink:
Also, a copy of the console.log entries you see at that time would help as well.

11

I am having the same problem with the import hanging when importing Thunderbird messages.

These a biggish mail files of 50 to 200Mb and the smaller ones imported fine.

They all imported into a trial copy of Eaglefiler OK, but I obviously really want them in DevonThink.

So I was hoping a solution had been found.

Graham

12

Maybe it will be better with the next maintenance release that support a new version of Pantomime. I have used all versions to import such large mailboxes and it took a couple of hours but it worked.
I would advise to run an optimise database operation on the database before importing such a huge batch of messages.

13

Thanks, maybe the import isn’t really frozen, but just working very slowly.

I will give it another try.

Graham

14

Couldn’t get it to work as a direct import, but imported the Thunderbird mailboxes into mail.app, and Devonthink imported from Mail.app with no real problems.

Some individual messages wouldn’t import, but in the main it worked well and much quicker than I had expected.

Graham

15

Sorry about these posts, but my solution seems not to be a solution at all because Mail.app has only imported a small proportion of original messages. 2000 out of 6000, for one folder :frowning:

Graham.

16

hi annard,

didn’t get around more email imports until tonight. just downloaded 1.3.2 and was hoping for an improvment but mailbox import still stalls on large imports. i’m just about to send an email with all the requested info to support.

thanks,
olaf

17

I’ve had this problem too and here’s what I’ve found:

  1. DP seems to load everything into memory and then dump it in the database. I watched as DT used 16GB of memory in my Mac Pro while importing one folder (it took about two hours to load a folder). If you have a serious amount of mail, I recommend an Xserver with XserveRAID and 32GB of memory. A Mac Pro with 16GB and and XserveRAID isn’t quite enough to really run this application.

  2. DP only uses one core–and it pegs that one.

  3. “Ya gotta have faith and believe”. Watching the Activity Monitor indicates that even though DP shows not responding, it’s still working. I believe it’s paging during the periods of unresponsiveness. I guess it pages or writes a tmp file to the system disk–is there a way to have it page to the faster RAID array?

  4. You want to keep the console log up, because sometimes it crashes.
    OS Version: 10.4.10 (Build 8R2218b)
    Report Version: 4

Command: DEVONthink Pro
Path: /Applications/DEVONthink Pro.app/Contents/MacOS/DEVONthink Pro
Parent: WindowServer [14622]

Version: 1.3.1 (1.3.1)

PID: 21549
Thread: 0

Exception: EXC_BAD_ACCESS (0x0001)
Codes: KERN_PROTECTION_FAILURE (0x0002) at 0x00000000

Thread 0 Crashed:

  1. No crashes after upgrading to 1.3.2 so far.

  2. I never had DevonDataHelper restart automatically after killing it in Activity Monitor. After a force quit it just stayed down and took a couple of DT restarts to come back up. I never noticed any difference whether it was up or down. My guess is that because it takes 29MB when idle, a force quit frees up memory and allows DT to have a bit more.

  3. Sometimes rebuilding the mailbox helps. mailbfr osx.topicdesk.com/content/view/41/57/ works better than a rebuild with the OS X tools.

  4. DT really hates it when you delete a mailbox (in mail.app). The import mail window sits and spins for a very long time causing you to quit (not force quit) the application from the Dock icon.

  5. Start up after importing seems to be kind of slow. This wouldn’t be a problem if you could open multiple databases at once, but because you have to go back and forth…

Sorry I don’t have a better solution than to throw more hardware at it and use the latest version.