You are correct. The actual document files that are stored within the database package file are not loaded into memory unless and until they are called by an Open command (or some other command that involves pulling the file from disk into RAM).
But that doesn’t make my comments about RAM moot.
The artificial intelligence features that make DEVONthink distinctive are built into the core of the database. When you drop a new file into the database, its text content is analyzed and merged into the database Concordance. That’s rather like indexing text content in ‘ordinary’ databases. Now DEVONthink ‘knows’ that a term used in a search will be included in that newly added file, for example.
But DEVONthink can do some things that ‘ordinary’ databases can’t do, and they involve a lot of ‘artificial thinking’ and use memory.
Now that you’ve added a new document into your database you can select it, click on the Magic Hat icon and let DEVONthink suggest where it might appropriately fit into your database group structure. That’s the ‘Classify’ AI routine.
Classify looks at the text content characteristics within each group in your database and compares the text content of your new document with those, in order to suggest one or more groups into which you might place that new document.
Classify doesn’t merely look at the occurrences and frequencies of use of words in the new document and in the content of each group, but at their contextual relationships, i.e., patterns of usage and occurrence.
Assuming that you started the organization of content into groups whose content has real (non-random) kinds of similar relationships that are relatively distinctive for each group, Classify will become more and more effective as your database grows in size.
That is, in a very literal sense, mind-boggling. The human brain isn’t ‘built’ to do that sort of thing very well, especially when your database contains hundreds or even thousands of groups. DEVONthink and your Mac can do that almost instantly — but there’s a lot of data being processed in RAM.
Notice that when you pressed the Magic Hat button the side panel also contains a See Also list, suggesting other documents in the database (regardless of their group organization) that may be similar to the new document you are viewing.
This is even more mind-boggling. That list of suggestions involved an analysis of the contextual usage of word in your new document and a comparison to the contextual usage of words in each and every one of the tens of thousands of documents in the database.
I can’t do that. I’m probably aware of some other documents in my database that I know are related in content (terms, concepts) to the one I just added. But I can’t hold in my head the content of all of the many thousands of documents (comparing each to the one I’m viewing), and it’s likely that there are some relationships that I haven’t thought about.
That last point is what often excites me about See Also suggestions when I’m exploring my database for ideas; I really appreciate a tip that there is a relationship of words/ideas (ideas can be represented as patterns of words) that I hadn’t thought of previously.
Perspectives for using Classify and See Also:
You are the human being, and DEVONthink is a tool you are using to help you mine and analyze the information content in a collection of documents. You actually have enormously more computing power than does your Mac, but the Mac and DEVONthink can handle some tasks that would be difficult or time-consuming if you tried to do them yourself.
DEVONthink doesn’t know anything about your kinds of knowledge and expertise, such as chemistry, genetics or writing novels.
You are responsible to evaluate and judge the utility to you of DEVONthink’s Classify and See Also suggestions. Some of them should be dismissed immediately as useless, on the basis of your knowledge and expertise. Others are useful, and once in a while See Also can help one get a startling new insight about something.
Often enough, those suggestions save me time and effort (thinking is really hard, sometimes). On that basis, I consider DEVONthink the best research assistant I’ve ever had.
Give DEVONthink enough free RAM to do all the kinds of data processing you ask of it, and it remains quick and responsive. Run out of free RAM, and you move into Virtual Memory use and start seeing spinning balls, which I find irritating.
If your Mac can hold more RAM, adding RAM is pretty cheap as a way to avoid that spinning ball. Another way, especially if your database has grown and your RAM is maxed out, is to spit databases topically or historically so as to keep your processing needs within the limits of free RAM. There have been a number of discussion threads on spitting databases. As DEVONthink Pro 2 and DEVONthink Pro Office 2 allow multiple open databases, you can always assemble a set of open databases like informational Lego blocks, when necessary. (Although currently, in public beta 7, See Also and Classify cannot work across databases, that will come in the future, per Christian’s recent post.)