New database can't be opened after indexing, Devonthink 3 Beta 4

Yesterday I have a fresh installl of MacOS Mojave (10.14.1), then later updated to 10.14.5

This morning I had a fresh install of Devonthink 3 Beta 4, with all default add-ons. I created a new database on a separate hard disk, and chose “index” (not import) of over 6000 pdf files in the new database, which took about several hours to complete, with resulted new database file “xxx.dtBase2” close to 4GB.

When I restart the iMac, and open Devonthink again, it hang during startup and unresponsive, and I have to force it to quit, the subsequent trying failed too, and Devonthink always suspend and stop responding when open the new database “xxx.dtBase2”

So would you suggest what to do next here, to give up the newly created indexed database, and recreate database and build index again? or there could be some workaround/fix…also I am not sure if this is due to Beta version, or other potential reasons?

Thanks.

The external disk is a 2T Lacie Prism, and connected to iMac using a USB to Type-C, and the connection seems ok for everything else. The last try by opening the “xxx.dtBase2” ended with the process consuming all the memory and basically the iMac screen went black, and I had to power it down.

Both macOS and Lacie disk are formatted as APFS encrypted, and iMac(late 2015) has 8GB memory.

Thanks for the comments, and I will try to start with a small database on the boot up disk. But to me this is a concern for Devonthink handling large database, as the database will normally grow eventually.

Could you please launch Apple’s Activity Monitor application (see Applications > Utilities), choose the frozen DEVONthink 3 while trying to open this database in the list of processes, select the menu item View > Sample Process and send the result to cgrunenberg - at - devon-technologies.com? Thanks in advance!

After the last failed attempt, I deleted the database and created a new database on the default startup disk followed earlier suggestion. This time I add folder(s) for indexing piece by piece, so far the new database has around 36GB files indexed and sized 1.2GB, and it takes about 10-15 seconds to re-open, so everything seems ok this time.

I will add more folders in the database for indexing later, and if the hanging problem happens again, I will follow the steps above and send you the 'Sample Process" result. Thank you for the support.

Thanks for the info! Does File > Optimize Database… improve this?

Double the indexed file to 72GB now and the database grows to 2.2GB, and it takes about 20 seconds to open.

Then I did the “Optimize Database” from the menu, after the optimization was done, and reopen Devonthink, it took about the same amount of time like 20 seconds, only the database size now appears to be doubled to 4.3GB? Then I did the “Optimize Database” again, and restart time was the same around 20 seconds, and database size tripled to 6.3GB…is this supposed to happen? what database optimization do during the process?

The optimization creates also internal backups of the metadata & index (up to 2 copies).

Thanks. Is it possible to scale it back, or this is no performance harm other than taking more disk space?

I also try “Verify and Repair” today which quickly finishes without prompt. But then restart Devonthink takes more time and hang for a while, for which I have sent you the sample log by Activity Monitor.

The backup uses only additional disk space and slightly slows down the optimization.

Is there a limit here about how many files/indexed words that Devonthink can currently handle?

For example, the database properties shows it currently indexed 1,659,012 unique words and 250,775,000 words in total.

There’s no real limit but recommended are databases not containing more than 200-300 thousand items items and 200-300 million words.