Recently DEVONthink (3.8.3) running on an M1 MacPro laptop (BigSUR 11.6, 13 inch, 2020, 16GB memory, 1TB internal drive [editorial: way, way too small for real professional work]) has taken to causing my system to run out of application memory when I open certain large databases, in particular one that indexes my Zotero storage tree (>10,000 items). This was not a problem in the past, just recently. No troubles on my much more capacious MacPro with 64GB, but it would be nice to not have to shut everything down and restart due to the memory issue. Could there be a leak of some sort?
The same has happened to me with my Mac mini having 64GB ram. But DT3 was not the problem. My email utility Small Cubed was taking half the the memory. I did an extensive search on the web and found many users having the same problem with a variety of apps. I now suspect macOS.
Have you checked in the macOS Activity Monitor what actually used the most memory? Maybe something else had occupied almost the complete memory already and then the additionally opened DEVONthink database just was the one thin mint waffle on top that was too much.