Random Freezing while browsing search results in large PDF database

olivierroymd · November 28, 2023, 7:09pm

My Devonthink has been freezing randomly (mostly when I browse through the search results) for as long as I can remember. I’ve had this issue on 3 different Macs now. It never goes away. Freezes can last anywhere from 1 to 5 seconds. They can occur each time I select a new PDF for a few times but sometimes, it is smooth. The randomness makes it very difficult to track the issue. I have tried disabling cloud syncing, disabling search options such as Live while typing and Partial matches while typing or Fuzzy search. It is not related to the size of PDF to display. I have only 17 items larger than 100 MB (and 129 larger than 10 MB) and the biggest is 418 MB. The likelihood of freezing happening is quite similar for large and small PDF.

I have a M1 Max Macbook Pro 16 with 32 GB of RAM.

I have a single database of around 7k pdfs. They are all stored (locally, always on device) in Dropbox and indexed in Devonthink. I also use Devonthink ToGo so I use the sync function through Dropbox (pushed every hour).

I have been searching through the forum for a while now and have not found a satisfactory result.

I like Devonthink but it is so frustrating that I am considering alternatives.

rmschne · November 28, 2023, 7:18pm

Using the still-buggy Sonoma operating system?

olivierroymd · November 28, 2023, 7:26pm

Yes. It is possible that it is slightly worse since Sonoma but I have noticed the problem since I don’t know… Mojave? But it does not explain the whole story here.

rmschne · November 28, 2023, 7:34pm

Dunno. Maybe. Probably needs a debugger to really figure it out. might be some other posts here.

kewms · November 28, 2023, 7:45pm

Is Dropbox configured to store them offline, or only in the cloud? This sounds like the kind of delay you might see if you’re waiting for a download from the Dropbox server.

olivierroymd · November 28, 2023, 7:57pm

Always locally, on device. I edited the initial post.

olivierroymd · November 28, 2023, 8:02pm

Here is the Instrument.app log for Devonthink recorded during 2 severe hangs (~10 s each).

Weight	Self Weight		Symbol Name
3.85 s  100.0%	0 s	 	DEVONthink 3 (58103)
2.81 s   73.0%	0 s	 	 start
2.81 s   73.0%	0 s	 	  0x100511514
2.81 s   73.0%	0 s	 	   NSApplicationMain
2.81 s   73.0%	0 s	 	    -[NSApplication run]
2.80 s   72.8%	0 s	 	     -[DevonApplication _nextEventMatchingEventMask:untilDate:inMode:dequeue:]
2.80 s   72.8%	0 s	 	      -[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]
2.80 s   72.7%	0 s	 	       _DPSNextEvent
2.80 s   72.7%	0 s	 	        _BlockUntilNextEventMatchingListInModeWithFilter
2.80 s   72.7%	0 s	 	         ReceiveNextEventCommon
2.80 s   72.7%	0 s	 	          RunCurrentEventLoopInMode
2.80 s   72.7%	0 s	 	           CFRunLoopRunSpecific
2.80 s   72.7%	0 s	 	            __CFRunLoopRun
2.75 s   71.5%	0 s	 	             __CFRunLoopDoTimers
2.75 s   71.5%	0 s	 	              __CFRunLoopDoTimer
2.75 s   71.5%	0 s	 	               __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__
2.75 s   71.3%	0 s	 	                __NSFireDelayedPerform
2.74 s   71.1%	0 s	 	                 0x100651de0
2.73 s   71.0%	0 s	 	                  0x100651b98
2.73 s   71.0%	0 s	 	                   0x1006623b0
2.73 s   70.9%	1.00 ms	 	                    0x100660748
2.29 s   59.3%	1.00 ms	 	                     0x10060ab04
1.95 s   50.7%	5.00 ms	 	                      0x1005bddb0
1.81 s   47.1%	0 s	 	                       -[NSFileCoordinator coordinateReadingItemAtURL:options:error:byAccessor:]
1.81 s   47.0%	1.00 ms	 	                        -[NSFileCoordinator(NSPrivate) _coordinateReadingItemAtURL:options:error:byAccessor:]
1.63 s   42.4%	3.00 ms	 	                         -[NSFileCoordinator(NSPrivate) _withAccessArbiter:invokeAccessor:orDont:andRelinquishAccessClaim:]
1.47 s   38.1%	0 s	 	                          __85-[NSFileCoordinator(NSPrivate) _coordinateReadingItemAtURL:options:error:byAccessor:]_block_invoke.207
1.47 s   38.1%	0 s	 	                           __73-[NSFileCoordinator coordinateReadingItemAtURL:options:error:byAccessor:]_block_invoke
1.47 s   38.1%	0 s	 	                            -[NSFileCoordinator _invokeAccessor:thenCompletionHandler:]
1.17 s   30.2%	1.00 ms	 	                             0x10066114c
472.00 ms   12.2%	3.00 ms	 	                              +[DTMetaDataStore(Tagging) getFinderTags:error:]
459.00 ms   11.9%	1.00 ms	 	                               -[NSFileManager(DevonFileManagerExtensions) getResourceValueForKey:atPath:error:]
440.00 ms   11.4%	0 s	 	                                -[NSURL getResourceValue:forKey:error:]
438.00 ms   11.3%	2.00 ms	 	                                 CFURLCopyResourcePropertyForKey
435.00 ms   11.3%	0 s	 	                                  _FSURLCopyResourcePropertyForKeyInternal(__CFURL const*, __CFString const*, void*, void*, __CFError**, unsigned char)
428.00 ms   11.1%	0 s	 	                                   prepareValuesForBitmap(__CFURL const*, __FileCache*, _FilePropertyBitmap*, __CFError**)
332.00 ms    8.6%	0 s	 	                                    ExternalProviderPrepareValues(__CFURL const*, __FileCache*, __CFString const* const*, void const**, long, void const*, __CFError**)
325.00 ms    8.4%	2.00 ms	 	                                     _DS_CFURLGetPropertyForKey
323.00 ms    8.3%	0 s	 	                                      DS_CFURLGetTagProperty(__CFURL const*, void const**, __CFError**)
321.00 ms    8.3%	0 s	 	                                       NodeCopyTagsAndAttributes
288.00 ms    7.4%	0 s	 	                                        TFSInfo::CopyTagsAndAttributes(bool) const
284.00 ms    7.3%	0 s	 	                                         TFSInfo::CopyTagsForPath() const
282.00 ms    7.3%	1.00 ms	 	                                          _MDItemCopyExAttrWithURL
277.00 ms    7.2%	1.00 ms	 	                                           _MDCopyExtendedAttributeInternal
273.00 ms    7.0%	0 s	 	                                            copyExtendedAttribute
265.00 ms    6.8%	265.00 ms	 	                                             getxattr
7.00 ms    0.1%	0 s	 	                                             CFPropertyListCreateWithData
1.00 ms    0.0%	1.00 ms	 	                                             __CFDataInit
2.00 ms    0.0%	2.00 ms	 	                                            CFStringGetCString
1.00 ms    0.0%	1.00 ms	 	                                            _platform_memset
1.00 ms    0.0%	1.00 ms	 	                                           _MDCopyExtendedAttributeForPath
1.00 ms    0.0%	1.00 ms	 	                                           CanonicalFileURLStringToFileSystemRepresentation
1.00 ms    0.0%	0 s	 	                                           CFURLGetFileSystemRepresentation
1.00 ms    0.0%	1.00 ms	 	                                           DYLD-STUB$$bzero
1.00 ms    0.0%	0 s	 	                                          TRef<__CFURL const*, TRetainReleasePolicy<__CFURL const*>>::~TRef()
1.00 ms    0.0%	1.00 ms	 	                                          TAutoRef<void const*, TRetainReleasePolicy<void const*>>::~TAutoRef()
3.00 ms    0.0%	0 s	 	                                         TCFURLInfo::DecodeEncodedTagString(void const*)
1.00 ms    0.0%	1.00 ms	 	                                         -[__NSPlaceholderArray initWithCapacity:]
13.00 ms    0.3%	0 s	 	                                        TFSInfo::initialize(__CFURL const*, LSProperties, bool, bool, bool)
10.00 ms    0.2%	0 s	 	                                        TFSInfo::GetLabel() const
5.00 ms    0.1%	1.00 ms	 	                                        std::__1::__shared_weak_count::__release_shared[abi:v160006]()
3.00 ms    0.0%	0 s	 	                                        std::__1::shared_ptr<TFSInfo> std::__1::allocate_shared[abi:v160006]<TFSInfo, std::__1::allocator<TFSInfo>, void>(std::__1::allocator<TFSInfo> const&)
1.00 ms    0.0%	1.00 ms	 	                                        operator delete(void*)
1.00 ms    0.0%	1.00 ms	 	                                        std::__1::__shared_weak_count::__release_weak()
1.00 ms    0.0%	1.00 ms	 	                                       -[__NSArrayM objectAtIndex:]
1.00 ms    0.0%	1.00 ms	 	                                       CFArrayGetCount
7.00 ms    0.1%	0 s	 	                                     _FileCacheSetPropertyValueForKey
94.00 ms    2.4%	0 s	 	                                    corePropertyProviderPrepareValues(__CFURL const*, __FileCache*, __CFString const* const*, void const**, long, void const*, __CFError**)
1.00 ms    0.0%	1.00 ms	 	                                    _FileCacheDecrementInProvider(__FileCache*)
1.00 ms    0.0%	1.00 ms	 	                                    GetFromDesktopServices(__CFURL const*, void const*, void const**, __CFError**)
4.00 ms    0.1%	3.00 ms	 	                                   ExternalProviderCopyValues(__CFURL const*, __FileCache const*, __CFString const* const*, void const**, void const**, long, void const*)
2.00 ms    0.0%	0 s	 	                                   _FileCacheGetForURL(__CFURL const*, void*)
1.00 ms    0.0%	1.00 ms	 	                                   _FileCacheLockTransitionToPreparer(__FileCache const*)
1.00 ms    0.0%	1.00 ms	 	                                  prepareValuesForBitmap(__CFURL const*, __FileCache*, _FilePropertyBitmap*, __CFError**)
2.00 ms    0.0%	1.00 ms	 	                                 -[NSURL(NSURL) _cfurl]
11.00 ms    0.2%	1.00 ms	 	                                _CFURLCreateWithFileSystemPath
5.00 ms    0.1%	0 s	 	                                _CFRelease
1.00 ms    0.0%	0 s	 	                                -[NSURL(NSURL) initFileURLWithPath:isDirectory:]
1.00 ms    0.0%	1.00 ms	 	                                -[NSURL(NSURL) release]
8.00 ms    0.2%	1.00 ms	 	                               -[NSUserDefaults(NSUserDefaults) boolForKey:]
2.00 ms    0.0%	2.00 ms	 	                               +[NSFileManager defaultManager]
204.00 ms    5.3%	1.00 ms	 	                              0x100602340
148.00 ms    3.8%	0 s	 	                              +[DTMetaDataStore objectForKey:atPath:traverseLink:error:]
129.00 ms    3.3%	0 s	 	                              -[NSURL getResourceValue:forKey:error:]
129.00 ms    3.3%	0 s	 	                               CFURLCopyResourcePropertyForKey
128.00 ms    3.3%	1.00 ms	 	                                _FSURLCopyResourcePropertyForKeyInternal(__CFURL const*, __CFString const*, void*, void*, __CFError**, unsigned char)
116.00 ms    3.0%	1.00 ms	 	                                 prepareValuesForBitmap(__CFURL const*, __FileCache*, _FilePropertyBitmap*, __CFError**)
113.00 ms    2.9%	0 s	 	                                  corePropertyProviderPrepareValues(__CFURL const*, __FileCache*, __CFString const* const*, void const**, long, void const*, __CFError**)
2.00 ms    0.0%	2.00 ms	 	                                  PropertyMaskIsNotZero(_FilePropertyMask const*, long*, long*)
4.00 ms    0.1%	0 s	 	                                 corePropertyProviderCopyAndCacheValues(__CFURL const*, __FileCache*, __CFString const* const*, void const**, void const**, long, void const*)
3.00 ms    0.0%	1.00 ms	 	                                 corePropertyProviderCopyValues(__CFURL const*, __FileCache const*, __CFString const* const*, void const**, void const**, long, void const*)
2.00 ms    0.0%	0 s	 	                                 _FileCacheLock(__FileCache const*)
1.00 ms    0.0%	0 s	 	                                 getPropertyForKey(__CFString const*)
1.00 ms    0.0%	1.00 ms	 	                                 addPropertyAndDependenciesToBitmap(_FileProperty const*, _FilePropertyBitmap*)
1.00 ms    0.0%	0 s	 	                                _CFURLHasFileURLScheme
78.00 ms    2.0%	0 s	 	                              0x100660114
36.00 ms    0.9%	1.00 ms	 	                              0x1006013a0
33.00 ms    0.8%	1.00 ms	 	                              -[NSUserDefaults(NSUserDefaults) boolForKey:]
16.00 ms    0.4%	0 s	 	                              -[NSArray(NSArrayDevonExtensions) containsObjectIdenticalTo:]
14.00 ms    0.3%	0 s	 	                              +[DSStore finderCommentModificationDateAtPath:]
7.00 ms    0.1%	0 s	 	                              -[NSURL(NSURL) path]
4.00 ms    0.1%	0 s	 	                              -[NSString(NSStringDevonCloudExtensions) isGoogleDriveFolder]
4.00 ms    0.1%	1.00 ms	 	                              0x1005fd9e8
3.00 ms    0.0%	0 s	 	                              -[NSString(NSStringDevonCloudExtensions) isOneDriveFolder]
3.00 ms    0.0%	3.00 ms	 	                              objc_msgSend
2.00 ms    0.0%	0 s	 	                              validatedTags
2.00 ms    0.0%	1.00 ms	 	                              0x1005fd27c
2.00 ms    0.0%	0 s	 	                              0x100661598
2.00 ms    0.0%	0 s	 	                              0x1005fd30c
1.00 ms    0.0%	1.00 ms	 	                              0x10060c94c
1.00 ms    0.0%	0 s	 	                              0x1005fd380
1.00 ms    0.0%	1.00 ms	 	                              -[NSObject isMemberOfClass:]
1.00 ms    0.0%	1.00 ms	 	                              0x10075a561
1.00 ms    0.0%	1.00 ms	 	                              0x10078e38d
234.00 ms    6.0%	0 s	 	                             0x1006617c4
55.00 ms    1.4%	0 s	 	                             -[NSFileCoordinatorAccessorBlockCompletion decrement]
6.00 ms    0.1%	1.00 ms	 	                             +[NSFileCoordinatorAccessorBlockCompletion completionWithBlock:]
2.00 ms    0.0%	0 s	 	                             0x1005bdf30
2.00 ms    0.0%	2.00 ms	 	                             0x100777da1
1.00 ms    0.0%	0 s	 	                             NSFileCoordinatorGetTSD
1.00 ms    0.0%	1.00 ms	 	                             0x100789221
138.00 ms    3.5%	0 s	 	                          -[NSFileAccessClaim _protectIfNecessaryFileAtURL:withOptions:forReading:]
21.00 ms    0.5%	1.00 ms	 	                          _NSSetThreadSpecificMaterializationState
2.00 ms    0.0%	2.00 ms	 	                          objc_msgSend
2.00 ms    0.0%	0 s	 	                          -[NSFileAccessClaim claimID]
1.00 ms    0.0%	1.00 ms	 	                          _objc_rootAutorelease
1.00 ms    0.0%	1.00 ms	 	                          objc_msgSend$_protectIfNecessaryFileAtURL:withOptions:forReading:
93.00 ms    2.4%	1.00 ms	 	                         -[NSFileCoordinator(NSPrivate) _blockOnAccessClaim:withAccessArbiter:]
53.00 ms    1.3%	0 s	 	                         -[NSFileReadingClaim initWithPurposeID:url:options:claimer:]
9.00 ms    0.2%	0 s	 	                         _os_log_impl
8.00 ms    0.2%	0 s	 	                         -[NSFileReadingClaim dealloc]
4.00 ms    0.1%	2.00 ms	 	                         +[NSArray arrayWithObjects:]
3.00 ms    0.0%	1.00 ms	 	                         consumeSandboxTokens
2.00 ms    0.0%	1.00 ms	 	                         -[NSFileAccessClaim claimID]
1.00 ms    0.0%	1.00 ms	 	                         objc_retain
1.00 ms    0.0%	1.00 ms	 	                         objc_retain_x28
1.00 ms    0.0%	1.00 ms	 	                         DYLD-STUB$$free
1.00 ms    0.0%	1.00 ms	 	                         _nanov2_free
1.00 ms    0.0%	1.00 ms	 	                        objc_msgSend
75.00 ms    1.9%	0 s	 	                       -[NSString(NSStringDevonPathExtensions) isRemoteVolumePath]
34.00 ms    0.8%	3.00 ms	 	                       0x1005bd800
13.00 ms    0.3%	0 s	 	                       0x1005bd904
6.00 ms    0.1%	2.00 ms	 	                       -[NSString(NSStringDevonCloudExtensions) isVolatileFolder]
4.00 ms    0.1%	0 s	 	                       +[NSDate timeIntervalSinceReferenceDate]
2.00 ms    0.0%	0 s	 	                       -[NSURL(NSURL) path]
1.00 ms    0.0%	1.00 ms	 	                       objc_msgSend
1.00 ms    0.0%	1.00 ms	 	                       _CFRelease
278.00 ms    7.2%	0 s	 	                      0x100609e8c
29.00 ms    0.7%	0 s	 	                      objc_autoreleasePoolPop
8.00 ms    0.2%	0 s	 	                      _CFURLCreateWithFileSystemPath
7.00 ms    0.1%	1.00 ms	 	                      _CFRelease
4.00 ms    0.1%	3.00 ms	 	                      0x1005fd920
1.00 ms    0.0%	1.00 ms	 	                      0x10055d234
1.00 ms    0.0%	1.00 ms	 	                      0x10061b350
1.00 ms    0.0%	1.00 ms	 	                      0x100769241
1.00 ms    0.0%	1.00 ms	 	                      AutoreleasePoolPage::push()
1.00 ms    0.0%	1.00 ms	 	                      -[NSURL(NSURLDevonExtensions) initFileURLWithAbsolutePath:]
421.00 ms   10.9%	0 s	 	                     0x10060a3a0
10.00 ms    0.2%	0 s	 	                     objc_autoreleasePoolPop
6.00 ms    0.1%	2.00 ms	 	                     0x1005f256c
2.00 ms    0.0%	0 s	 	                     0x1005dce70
1.00 ms    0.0%	1.00 ms	 	                     objc_msgSend
1.00 ms    0.0%	1.00 ms	 	                     0x100763e81
1.00 ms    0.0%	1.00 ms	 	                     0x1005f25d0
1.00 ms    0.0%	1.00 ms	 	                     0x100764801
1.00 ms    0.0%	1.00 ms	 	                     0x100764421
1.00 ms    0.0%	1.00 ms	 	                     0x1005f2738
3.00 ms    0.0%	0 s	 	                    0x10060ab04
3.00 ms    0.0%	0 s	 	                     0x1005bddb0
3.00 ms    0.0%	0 s	 	                      -[NSFileCoordinator coordinateReadingItemAtURL:options:error:byAccessor:]
3.00 ms    0.0%	0 s	 	                       -[NSFileCoordinator(NSPrivate) _coordinateReadingItemAtURL:options:error:byAccessor:]
3.00 ms    0.0%	0 s	 	                        -[NSFileCoordinator(NSPrivate) _withAccessArbiter:invokeAccessor:orDont:andRelinquishAccessClaim:]
3.00 ms    0.0%	0 s	 	                         __85-[NSFileCoordinator(NSPrivate) _coordinateReadingItemAtURL:options:error:byAccessor:]_block_invoke.207
3.00 ms    0.0%	0 s	 	                          __73-[NSFileCoordinator coordinateReadingItemAtURL:options:error:byAccessor:]_block_invoke
3.00 ms    0.0%	0 s	 	                           -[NSFileCoordinator _invokeAccessor:thenCompletionHandler:]
3.00 ms    0.0%	0 s	 	                            0x1006624e4
1.00 ms    0.0%	1.00 ms	 	                    0x1007537c1
2.00 ms    0.0%	0 s	 	                  0x1006dd7a4
7.00 ms    0.1%	0 s	 	                 0x1005d5474
1.00 ms    0.0%	0 s	 	                 -[NSDictionary(NSDictionary) writeToFile:atomically:]
3.00 ms    0.0%	0 s	 	                _CFAutoreleasePoolPop
3.00 ms    0.0%	0 s	 	                ___NSRunLoopTimerCreateWithHandler_block_invoke
1.00 ms    0.0%	0 s	 	                -[NSApplication(NSQuietSafeQuit) _updateCanQuitQuietlyAndSafely]
1.00 ms    0.0%	0 s	 	                __NSFireTimer
46.00 ms    1.1%	0 s	 	             __CFRunLoopDoObservers
1.00 ms    0.0%	1.00 ms	 	             __kdebug_trace64
1.00 ms    0.0%	0 s	 	       _NSSendEventToDequeuingObservers
8.00 ms    0.2%	0 s	 	     -[NSApplication _handleEvent:]
1.02 s   26.5%	0 s	 	 start_wqthread
833.00 ms   21.6%	0 s	 	  _pthread_wqthread
187.00 ms    4.8%	187.00 ms	 	  start_wqthread
18.00 ms    0.4%	0 s	 	 thread_start
18.00 ms    0.4%	0 s	 	  _pthread_start
18.00 ms    0.4%	0 s	 	   _NSEventThread

I could provide the longer more detailed if requested (can’t put link here).

rmschne · November 28, 2023, 8:09pm

I do not know how to read the log you provided but someone might.

Wondering if the Dropbox file is set to be offline but at that moment of calling it it is not my so then downloads it.

And if you take the large files out of the Dropbox folder and put say in ~/documents local does performance change?

Edit: and import the PDFs and compare performance.

BLUEFROG · November 28, 2023, 9:08pm

Hold the Option key and choose Help > Report bug to start a support ticket. Attach the process sample.

olivierroymd · November 28, 2023, 9:30pm

Done

cgrunenberg · November 29, 2023, 6:14am

You’re absolutely right, according to the sample the delay is caused by file coordination. Maybe due to network reliability/performance or due to unresponsive Dropbox servers this seems to cause a noticeable delay every then and now.

There’s a hidden preference to disable file coordination but in that case DEVONthink might not automatically downloaded indexed files which are not locally available or are not up to date locally.

Anyway, we’ll investigate this.

olivierroymd · November 29, 2023, 7:39am

Thank you for the investigation. I’m not sure how this could be related to a network issue because the files are stored locally. For example, there is no delay if I open any of those PDF in Preview.app from the Finder.

BLUEFROG · November 29, 2023, 2:15pm

Are you opening a massive folder of subfolders and files in Preview? Nope. Not the same mechanisms.

olivierroymd · November 29, 2023, 2:30pm

I got a response from the Support team. It seems to be, indeed, a coordination issue. I turned on the DisableFileCoordination hidden preference and, so far so good. I’ll test for a while before calling it a success but it looks good. Thanks.

rmschne · November 29, 2023, 4:31pm

@bluefrog

The “DEVONthink Manual” says:

DisableFileCoordination: DEVONthink no longer uses file coordination, registering its interest in certain files and waiting for replies. This can cause very long delays if the coordinating process isn’t responding quickly.

In layman’s terms, what does this mean and how might this be a solution to this issue? Just curious.

BLUEFROG · November 29, 2023, 5:31pm

It’s often the case, more than one process can access a file in the filesystem. File coordination allows an app to say “Hey! I’m going to modify this file. Does anyone care?” Apps that have an interest in the document can reply. But those apps don’t always respond or not in a timely fashion.

kewms · November 29, 2023, 5:37pm

And Dropbox would “have an interest” in any file in the Dropbox folder, whether it’s stored locally or on the server. Makes sense.

BLUEFROG · November 29, 2023, 6:10pm

Indeed. But it isn’t always responsive to such interactions. It seems a somewhat common malady of cloud services, as iCloud and OneDrive have also been involved in such slow behaviors when file coordination is on.

It’s also possible the sheer number of files involved could be at play too. Anecdotally, it seems to happen when people are indexing large volumes of cloud-synced data. However, @cgrunenberg would know more authoritatively if that were true.