Archiving Gmail in Apple Mail Slow and Buggy

jarifi12 · July 8, 2020, 12:29pm

I have archived my iCloud mail without issue. When archiving my gmail account inside apple mail the same process seems buggy and very slow ( yes it’s a large mailbox - 50000 messages).
Is there a different process I should use other than the import page and clicking on archive?
I did it successfully once and then the next day I wanted to archive again just the new emails(maybe 50) and I got the beach all spinning again and had to force quit which seems to corrupt my other databases ie. some more orphaned files and missing files). Any suggestions?

Another related issue is when I go to the import page and the Apple Mail is visible it sometimes shows the proper 4 mailboxes and other times displays other mailboxes that are not active. Why?

Thank you

BLUEFROG · July 8, 2020, 12:36pm

I had already responded in the other thread regarding your second point.

To ensure proper the Apple Mail plugin is installed and enabled correctly, do this…

Quit Mail.
Quit DEVONthink 3.
Enable Full disk access and all Automation requests for DEVONthink 3 in System Preferences > Security & Privacy > Privacy.
In the Finder, hold the Option key and choose Go > Library.
Go into the Mail/Bundles folder and delete the DEVONthink.mailbundle.
Relaunch DEVONthink 3 and install the Mail plugin from DEVONthink 3 > Install Add-Ons.
Relaunch Mail and enable the plugin via Apple Mail’s Preferences > General > Manage Plug-ins.

jarifi12 · July 8, 2020, 1:38pm

I am sorry for what appears to be duplication. I am a reasonably experienced Mac user but despite reading instructions in the help system and a Take Control book about DTPO , I have experienced a great amount of frustration and wasted time.

Again other than making sure that I follow your instructions (which I have already done, having read another post) and rebuilding/repairing my Apple mailboxes in Apple mail is there anything else to account for the peculiar behavior of the inactive mailboxes in Apple Mail sometimes showing up in DTPO?

Assuming I get the email archive set up in a new database in DTPO can you please give me instructions on how to update that database on a weekly basis with new emails to be archived from that Apple mail mailbox.

Thank you

BLUEFROG · July 8, 2020, 2:07pm

Are you referring to DEVONthink Pro Office, as in the 2.x line?

And you would just use the Archive Mailbox function periodically.

jarifi12 · July 8, 2020, 2:40pm

DevonThink Pro 3.5.1

jarifi12 · July 8, 2020, 2:43pm

When I tried repeating the Archive routine for only 30 new emails using the importer screen and clicking on “archive” I got the spinning beach ball and had to force quit DevonThink.

BLUEFROG · July 8, 2020, 2:53pm

Did you follow my instructions?

jarifi12 · July 8, 2020, 3:35pm

I had followed your instructions last evening (I had read a prior post with the same instructions). And when I did that last evening I got the spinning beach ball when I attempted re-archiving the mailbox with just the new messages. I will try again tonight starting from scratch following your instructions.

BLUEFROG · July 8, 2020, 3:54pm

Given the size of the mailbox, you can also try entering this in /Applications/Utilities/Terminal.app (each as a separate command)…

defaults write com.devon-technologies.thinkpro2 MailImport.FetchTimeout -float 300
defaults write com.devon-technologies.thinkpro2 MailImport.Timeout -float 300

…while DEVONthink isn’t running to increase the timeout to 5 minutes.

Another possibility is to split the mailbox into multiple smaller ones.

Also, you don’t have to import the entire mailbox at once. Importing smaller batches often proves to be less taxing as a large amount of work is involved in indexing this data as it comes in.

jarifi12 · July 8, 2020, 5:01pm

Thank you very much. I very much appreciate your patience.

Rich

BLUEFROG · July 8, 2020, 5:08pm

No problem

jarifi12 · July 10, 2020, 11:57am

So far so good. Have a smaller mailbox that’s behaving.
Two questions…

Can you give me step by step instructions to update this mail database. Do I go to the import page, choose the specific mailbox and just click archive? Do I have to put in specific dates or will it automatically know it’s an incremental archive? How long should adding just 10 more emails to the archive take?
Could you give me steps to take to archive my 50000 email account in smaller increments? Again do I put in dates on the archive page to break up the archive into smaller sizes or is there a better way?

Once again I appreciate your time and efforts on my behalf!

Thanks, Rich

BLUEFROG · July 10, 2020, 1:27pm

You’re welcome.

You can’t import the same emails into a database more than once. When you go to the Import sidebar and choose the same mailbox, it will only show you emails that haven’t been imported. So you’d just just press the Archive Mailbox button again.
Yes you could put in dates in the Import sidebar. You could also create mailboxes in Mail.app and sort emails into them.

eMaX · July 11, 2020, 11:15am

How about this

jarifi12 · July 11, 2020, 1:07pm

So for an iCloud mailbox everything works. For my imap mailbox through my internet provider everything works. When I try to archive my two gmail accounts inside Apple mail things get wonky. Again these are large accounts. I go to the import page. The accounts appear under Apple mail. For one of the accounts there are ~23000 items. I clicked on the archive button and all seems to be working and about ½ hour into importing it seems to stall. I have my MBP not to sleep and I leave it. This morning (letting it run overnight) I found an error message that DEVONthink has unexpectedly quit. I will post the error report below. I reopen DEVONthink and lo and behold all the messages are there. Of course I have to reopen all the databases and go through the Corning’s that the databases are still running. I click continue and some of the databases show either a small amount of orphaned files which are repairable or a small amount of missing files.

I have now tried again and put in specific dates to have less files. When I clicked archive it it took about 20 minutes to populate and there are emails outside the date range I specified and in fact seem to match the default date range that the archive window shows.

So my questions are…

Is there something peculiar about DEVONthink importing gmail mailboxes in Apple mail and is there a workaround.
Should I follow your earlier instruction and clear the mail bundles and reinstall the Apple mail plugin after each crash.
Why with each crash do I seem to lose some files (ie missing files and orphaned files) even in the databases I’m not archiving or importing data into currently?

Here’s the crash report …

Process: DEVONthink 3 [59454]
Path: /Applications/DEVONthink 3.app/Contents/MacOS/DEVONthink 3
Identifier: com.devon-technologies.think3
Version: 3.5.1 (3.5.1)
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: DEVONthink 3 [59454]
User ID: 501

Date/Time: 2020-07-11 00:46:57.064 -0400
OS Version: Mac OS X 10.15.5 (19F101)
Report Version: 12
Bridge OS Version: 3.0 (14Y908)
Anonymous UUID: AA9D3C23-7737-0D0A-B44C-F91599DE2085

Sleep/Wake UUID: EBE0CA04-F7F5-4CDB-A18D-2DD100D45746

Time Awake Since Boot: 350000 seconds
Time Since Wake: 25000 seconds

System Integrity Protection: enabled

Crashed Thread: 6 Dispatch queue: com.devon-technologies.email.import

Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00007000085b0ff8
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [59454]

VM Regions Near 0x7000085b0ff8:
Stack 000070000852e000-00007000085b0000 [ 520K] rw-/rwx SM=COW thread 9
–> STACK GUARD 00007000085b0000-00007000085b1000 [ 4K] —/rwx SM=NUL stack guard for thread 6
Stack 00007000085b1000-0000700008633000 [ 520K] rw-/rwx SM=COW thread 6

Thread 0:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff679e2dfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff679e3170 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff2d8a6f85 __CFRunLoopServiceMachPort + 247
3 com.apple.CoreFoundation 0x00007fff2d8a5a52 __CFRunLoopRun + 1319
4 com.apple.CoreFoundation 0x00007fff2d8a4ece CFRunLoopRunSpecific + 462
5 com.apple.HIToolbox 0x00007fff2c4d3abd RunCurrentEventLoopInMode + 292
6 com.apple.HIToolbox 0x00007fff2c4d37d5 ReceiveNextEventCommon + 584
7 com.apple.HIToolbox 0x00007fff2c4d3579 _BlockUntilNextEventMatchingListInModeWithFilter + 64
8 com.apple.AppKit 0x00007fff2ab1b829 _DPSNextEvent + 883
9 com.apple.AppKit 0x00007fff2ab1a070 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
10 com.apple.AppKit 0x00007fff2ab0bd7e -[NSApplication run] + 658
11 com.apple.AppKit 0x00007fff2aaddb86 NSApplicationMain + 777
12 com.devon-technologies.think3 0x0000000102164130 0x1020d6000 + 581936
13 libdyld.dylib 0x00007fff678a1cc9 start + 1

Thread 1:: JavaScriptCore bmalloc scavenger
0 libsystem_kernel.dylib 0x00007fff679e5882 __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff67aa6425 _pthread_cond_wait + 698
2 libc++.1.dylib 0x00007fff64b75592 std::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 18
3 com.apple.JavaScriptCore 0x00007fff320a1224 void std::__1::condition_variable_any::wait<std::__1::unique_lockbmalloc::Mutex >(std::__1::unique_lockbmalloc::Mutex&) + 84
4 com.apple.JavaScriptCore 0x00007fff320a5a2b bmalloc::Scavenger::threadRunLoop() + 299
5 com.apple.JavaScriptCore 0x00007fff320a55f9 bmalloc::Scavenger::threadEntryPoint(bmalloc::Scavenger*) + 9
6 com.apple.JavaScriptCore 0x00007fff320a7cd7 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void ()(bmalloc::Scavenger), bmalloc::Scavenger*> >(void*) + 39
7 libsystem_pthread.dylib 0x00007fff67aa6109 _pthread_start + 148
8 libsystem_pthread.dylib 0x00007fff67aa1b8b thread_start + 15

Thread 2:: com.apple.NSEventThread
0 libsystem_kernel.dylib 0x00007fff679e2dfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff679e3170 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff2d8a6f85 __CFRunLoopServiceMachPort + 247
3 com.apple.CoreFoundation 0x00007fff2d8a5a52 __CFRunLoopRun + 1319
4 com.apple.CoreFoundation 0x00007fff2d8a4ece CFRunLoopRunSpecific + 462
5 com.apple.AppKit 0x00007fff2acbd144 _NSEventThread + 132
6 libsystem_pthread.dylib 0x00007fff67aa6109 _pthread_start + 148
7 libsystem_pthread.dylib 0x00007fff67aa1b8b thread_start + 15

Thread 3:: com.apple.CFSocket.private
0 libsystem_kernel.dylib 0x00007fff679eb0fe __select + 10
1 com.apple.CoreFoundation 0x00007fff2d8cfce3 __CFSocketManager + 641
2 libsystem_pthread.dylib 0x00007fff67aa6109 _pthread_start + 148
3 libsystem_pthread.dylib 0x00007fff67aa1b8b thread_start + 15

Thread 4:: LocalStorage
0 libsystem_kernel.dylib 0x00007fff679e5882 __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff67aa6425 _pthread_cond_wait + 698
2 com.apple.JavaScriptCore 0x00007fff32033602 WTF::ParkingLot::parkConditionallyImpl(void const*, WTF::ScopedLambda<bool ()> const&, WTF::ScopedLambda<void ()> const&, WTF::TimeWithDynamicClockType const&) + 3858
3 com.apple.WebKitLegacy 0x00007fff3e5b2405 bool WTF::Condition::waitUntilWTF::Lock(WTF::Lock&, WTF::TimeWithDynamicClockType const&) + 165
4 com.apple.WebKitLegacy 0x00007fff3e5b55f1 std::__1::unique_ptr<WTF::Function<void ()>, std::__1::default_delete<WTF::Function<void ()> > > WTF::MessageQueue<WTF::Function<void ()> >::waitForMessageFilteredWithTimeout<WTF::MessageQueue<WTF::Function<void ()> >::waitForMessage()::‘lambda’(WTF::Function<void ()> const&)>(WTF::MessageQueueWaitResult&, WTF::MessageQueue<WTF::Function<void ()> >::waitForMessage()::‘lambda’(WTF::Function<void ()> const&)&&, WTF::Seconds) + 161
5 com.apple.WebKitLegacy 0x00007fff3e5b4d36 WebCore::StorageThread::threadEntryPoint() + 86
6 com.apple.JavaScriptCore 0x00007fff3204de4f WTF::entryPoint(WTF::NewThreadContext*) + 223
7 com.apple.JavaScriptCore 0x00007fff3111adf9 WTF::wtfThreadEntryPoint(void*) + 9
8 libsystem_pthread.dylib 0x00007fff67aa6109 _pthread_start + 148
9 libsystem_pthread.dylib 0x00007fff67aa1b8b thread_start + 15

Thread 5:: CVDisplayLink
0 libsystem_kernel.dylib 0x00007fff679e5882 __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff67aa6425 _pthread_cond_wait + 698
2 com.apple.CoreVideo 0x00007fff2f83731d CVDisplayLink::runIOThread() + 711
3 libsystem_pthread.dylib 0x00007fff67aa6109 _pthread_start + 148
4 libsystem_pthread.dylib 0x00007fff67aa1b8b thread_start + 15

Thread 6 Crashed:: Dispatch queue: com.devon-technologies.email.import
ons/A/libQuadrature.dylib

I truncated the report to file the limit of your reply space

VM Region Summary:
ReadOnly portion of Libraries: Total=798.7M resident=0K(0%) swapped_out_or_unallocated=798.7M(100%)
Writable regions: Total=10.3G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=10.3G(100%)

                            VIRTUAL   REGION

REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Accelerate framework 1408K 11
Activity Tracing 256K 1
CG backing stores 880K 2
CG image 9568K 247
CG raster data 16K 2
CoreAnimation 216.1M 471
CoreData Object IDs 4100K 2
CoreGraphics 8K 1
CoreImage 420K 35
CoreUI image data 5152K 46
Foundation 106.0M 5
IOKit 7940K 1
Image IO 5772K 273
JS VM Isolated Heap 33.0M 382
Kernel Alloc Once 8K 1
MALLOC 9.8G 2480
MALLOC guard page 208K 43
MALLOC_LARGE (reserved) 1992K 3 reserved VM address space (unallocated)
MALLOC_REALLOC (reserved) 20K 5 reserved VM address space (unallocated)
Memory Tag 242 12K 1
SQLite page cache 1344K 21
STACK GUARD 56.1M 15
Stack 15.1M 15
VM_ALLOCATE 320K 40
WebKit Malloc 95.1M 102
__DATA 59.2M 530
__DATA_CONST 362K 12
__FONT_DATA 4K 1
__LINKEDIT 400.6M 42
__OBJC_RO 32.2M 1
__OBJC_RW 1888K 1
__TEXT 398.1M 510
__UNICODE 564K 1
libnetwork 1664K 24
mapped file 465.8M 412
shared memory 660K 19
=========== ======= =======
TOTAL 11.7G 5758
TOTAL, minus reserved VM space 11.7G 5758

Model: MacBookPro14,2, BootROM 207.0.0.0.0, 2 processors, Dual-Core Intel Core i7, 3.5 GHz, 16 GB, SMC 2.44f5
Graphics: kHW_IntelIrisGraphics650Item, Intel Iris Plus Graphics 650, spdisplays_builtin
Memory Module: BANK 0/DIMM0, 8 GB, LPDDR3, 2133 MHz, 0x802C, 0x4D5435324C31473332443450472D30393320
Memory Module: BANK 1/DIMM0, 8 GB, LPDDR3, 2133 MHz, 0x802C, 0x4D5435324C31473332443450472D30393320
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x171), Broadcom BCM43xx 1.0 (7.77.111.1 AirPortDriverBrcmNIC-1610.1)
Bluetooth: Version 7.0.5f6, 3 services, 27 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
USB Device: USB 3.0 Bus
USB Device: Apple T1 Controller
Thunderbolt Bus: MacBook Pro, Apple Inc., 41.4
Thunderbolt Bus: MacBook Pro, Apple Inc., 41.4

I realize this is a lot of work on your part. I really like the program and can see its potential benefit for me. I don’t want to give up.

Thanks,
Rich

BLUEFROG · July 11, 2020, 1:57pm

Hold the Option key and choose Help > Report bug to start a support ticket.

I don’t want to give up.

I’m not sure why you would since this is a one-off event. You’re not going to be trying to email 20,000+ emails daily or weekly.

jarifi12 · July 11, 2020, 2:36pm

Just looking at the report and didn’t realize I am close to full on my 500GB SSD. Will do some cleaning up and free up some more space.

jarifi12 · July 12, 2020, 12:05pm

Followed all your suggestions/instructions. Attempted to update my main email archive. Did not get spinning beach ball, continued to add emails for 10 hours, went to bed and awoke with an error message that DEVONthink unexpectedly quit.

All the emails seemed to have been added. When I reopened the databases again received an error message that some files from my Documents ALL PDFs database were missing.

I will send you a bug report.

Should I reinstall the whole app preserving the databases? Any other ideas? Also trying to limit the archived emails by date on the import page doesn’t seem to help.

Finally even though 99% of my emails have been archived when I attempt to re-archive daily to add the newest emails the initial screen takes a long time to populate. Something must be wrong. I can’t believe that other users who want to keep their email archive updated would be willing to wait through the delay.

Thanks,
Rich