Forced to 'force quit' DTPO whilst importing Entourage

DEVONthink Tips
1

Hi everyone,

Recently purchased a license for beta2 and decided to create a new database to house all of my email from Entourage. For 2006, I have just over 10000 ‘inbox’ items. I started to to the import and left it for an hour. Upon returning to my desk, didn’t seem to be responding, and Finder told me that DTPO had indeed stopped responding.

Thinking that asking a program to import 10K messages was a bit cruel, I created a separate folder in Entourage for each month and moved the corresponding email items into each. The result was approx 1000 messages in each folder, give or take.

I couldn’t even get DTPO to import a full month. I had to force quit after about 5 min as it stopped responding and the fans on my MB 2.0GHz (2GB RAM) started to spin in high gear, suggesting a big load on the CPUs. I then split one month into two separate folders, thus around 500 emails each. Again, DTPO couldn’t import: fans would spin up and it stopped responding. Had to force quit (usually after about 3 min)

I can import around 100-150 messages at a time, but anything over that doesn’t work. Nothing in CrashReporter for DTPO, by the way.

Just wondering if anyone has any advice.

Top of the New Year to everyone!

Cheers!

David

2

Hi, David. I’ve imported over 19,000 Entourage messages and thousands more from Mail into a new mail archive database.

I’m running on a MacBook Pro 2.0 GHz with 2 GB RAM under OS X 10.4.8.

If you check DTPO’s Preferences > Email - Conversations options, note that message threading and grouping takes a great deal of processing time. Un-checking those options will speed up operations greatly (and you will probably have no difficulty in searching and sorting to pull conversations together pretty well).

Note that DTPO can be busily thinking about something when the Finder reports it as being unresponsive.

3

Thanks for the reply, Bill.

Those two items were already unchecked, thankfullly.

I did another test. 1) Checked the integrity of my entourage database, just on the off chance that DTPO is working on something that is, well, unworkable. All seems fine according the test. 2) Created a entirely new folder in Entourage and put all messages received in Feb 2006 in there. 781 items in total, with the largest email clocking in at 755K. All but three are less than 500K. 3) Re-started both DVTO and Entourage and tried to import the Feb 2006 folder into DTPO. No luck. I left it for 1 hour and 15 minutes: the fans were screaming the entire time. Activity Monitor reported near 100% use of CPUs by DTPO.

Tried another test:

  1. Re-started the MB and repaired permissions. Nothing to repair, according to Disk Utility. 2) Created an entirely new DTPO database (on the off-chance that the initial mail database was somehow corrupt). 3) Attempted to import the same Feb email folder from above. Once again, fans came on full force and DTPO (the process) ate away at 100% of CPU cycles. No luck importing, either. I stopped it after 15min.

So, at this point, something is seriously wrong, but not sure what to do next.

David

4

Just a quick update:

Tried another test. I created a new folder in Entourage and slowly added messages to it. Each time after adding messages, I check to see whether DTPO would import them or not. The results:

  1. Importing 160 messages worked.

  2. Importing 211 messages worked.

  3. Importing 305 messages worked.

  4. Importing 388 messages worked.

  5. Importing 473 messages worked.

  6. Importing 526 messages worked (took only 2 minutes to ‘load’ messages, which is where the process has been hanging in my experiences above in earlier posts).

I’ll keep going, but will do a bigger increase in the number of messages to import this time:

  1. Importing 733 messages worked. Hmm…

Keep going…

  1. Importing 781 messages (the full month of Feb ‘received email’ in my database) DIDN’T work.

A-ha. Interestingly, those last few emails that I added to the folder (before trying step 8) were from 28 Feb 2006. Not sure if one of them had a problem or not, so I tried another experiment. I took 750 messages from March, thinking that maybe there’s a magical number between 733 and 781 where I’m having problems.

Tried to import the all 750 in fell swoop. That worked fine. Only took two minutes to load the messages.

So, my first thought was that there is something about some email that I received on 28 Feb that DTPO just doesn’t like. I took all email received on 28 Feb (47 items) and put it in a separate folder and tried to import them. Didn’t work. Fans screaming. DTPO locks up.

Right, so I’ve isolated the problem. Now to look at those 47 emails:

  1. the biggest one is 430K.

  2. 11 have attachments (file types are either .doc or .pdf, with some .gif files from HTML email)

Overall, nothing terribly remarkable about any of them.

I removed the 11 with attachments to see if the remaining 36 emails without attachments would be read/loaded by DTPO. They loaded no problem.

So, it must be one of the attachments. Just to be sure, I took the 11 messages with attachments and tried to import them as a group. Didn’t work. Definitely one of the attachments/emails. You can see where this is going.

One-by-one, I tried to import each of the 11. All but one were loaded fine. The very last one contained a .pdf attachment announcing a conference in Turkey. I can open the attachment fine now, and the message itself is unremarkable and written in plain text. It is no value to me now (I didn’t go to the conference in case anyone is wondering), so I deleted.

As a final test, I now have 47 email messages from 28 Feb 2006, so tried to ‘load’ these. No problem.

Right, so I was able to isolate the problem, but the bigger question is how often am I going to have to do this for the 10K messages I have in my entourage database for 2006 (and that’s just my INBOX, not counting the 8K sent items).

Very tired, but curiously satisfied…

5

OK, another update:

I’ve managed to import everything. One other email (from sometime in June 2006) gave a similar problem to the one above, but I took the opportunity to prune some older emails that I no longer needed. The problem email was in the group that I deleted (and probably another conference announcement). The other good news is that my entire sent items folder from Entourage (totalling almost 8K items) imported without a problem and rather quickly. Was quite impressed.

I guess the lesson here, and for those who may encounter this problem in the future, is that some errant email messages, for some reason, may not play well with DTPO. I doubt highly it’s a problem with DTPO itself and more to do with perhaps the structure of the email, headers, etc.

Cheers!

David

6

It was probably caused by a crash during HTML conversion in the WebKit code from Apple, in the next release we made sure that this can’t crash our application again.

7

Well, I had a similar problem, except that in my case I was working with PDF files.

I lost count of how many times DTPO crashed or unexpectedly quit. I also had to Force Quit the application once or twice, and often the fans on my system would start working hard while using DTPO.

At first I also thought that the amount of files being imported and/or the size of the files were the culprits. I was wrong; a single 5.9 MB PDF was causing some of the crashes.

I can work with the 5.9 MB file in Acrobat, PDFPen Pro, Preview, etc., but could not import it or index it with DTPO. And I can’t explain all of the crashes, though.

It took me many hours to finally get my information into a DTPO database. This is a process that typically has taken me minutes.

I sent a crash report, so hopefully there will be a fix in the near future. For now, I had to remove 3 files from my database.

OS X 10.4.8
PowerPC G5 Dual 2 GHz, 3GB DDR SDRAM

8

Beside the problem, I’m quite impressed by the step-by-step procedure you’ve used to isolate the incident.
If you’re not working in a IT helpdesk or related position, I give you a “+100” for the way you handle your problem.

Something to evangelize ! Bill ? :smiley: