Ok, so I have lots of stuff in my database now, 1.7Gb and over 52 million words. As a replacement for the Finder, I find DT a great way to organization information. I just stick it in the database and never worry about; I really like the built-in previewing and editing capabilities.
But as a general search tool DT is falling way, way short of my expectations. I keep a certain body of text in DT (because it’s text) that I often have to do quick searches on. I know this material really well, so I usually know what file it’s in.
As I’m chatting with friends and need to look things up, I go of course to DT and open up the search tool.
I selected my options today and the part of the database I wanted to search, and then clicked on the search text to enter my query. And got a spinning wait cursor. 30 seconds later, I was able to type in my query. I hit return, more spinning. About a full minute later I get my query results, but they aren’t quite what I wanted. I click on the first entry anyway, to do a search within the text. Another wait cursor, again about a minute.
Now I’m frustrated, my friend is waiting, and I decide to just browse the file tree myself and forget the search tool. I click the window and click on the folder. More spinning wait cursors, this time for more than a minute.
I give up and just export the files to disk so I can search them manually in Emacs.
This has happened to me a lot when I try to use DT as a general database: tons and tons of inexplicable waiting. Waiting for menus to pull down, waiting for GUI elements to become active, waiting for documents to display when I click on them. Way, way too much waiting.
As long as I use it as a better Finder, I don’t have to wait much. It’s now become my main file-store. But as a more usable tool for tracking down information, I just don’t have that much patience.
I have tons of memory, a fast PowerBook, a fast hard drive, and I Backup&Optimize regularly (usually after every large import, or manipulation of the database). Can any of you DT wizards clue me in on what I’m doing wrong? I would love to start using DT much more heavily than I do, but the wait times are killing me.
Although you say you have tons of memory my guess is that the slowdowns result from heavy use of Virtual Memory. If you can monitor available free memory and the rate of pageouts in VM you will find that there’s low free memory and lots of pageouts taking place. That means that operations are having to go to your hard disk to access code and data, rather than accessing them in physical RAM. And of course disk access is MUCH slower than RAM access.
On my old 500 MHz TiBook, Virtual Memory use increases when I switch from one mode, e.g., searches to another mode, e.g., Classification or See Also. So long as I continue in a given mode, things stay pretty fast. But switching results in slowdowns because I’m then heavily into disk access in VM swap files. This is verified by lots of pageouts occurring. My rule of thumb is to expect slowdowns to increase by the time my VM swap files equal or near my 1 GB of physical RAM.
A quick ‘fix’ is to quite and relaunch DT Pro. This can result in starting up with more free RAM and quicker operations. Restarting the computer clears the VM files and will definitely perk up speed.
By contrast, I’ve got a PowerMac G5 dual core 2.3 GHz with 5 GB RAM. I can do very intensive database work in DT Pro for days without a single pageout and with only 1 64 MB Virtual Memory swap file. Most single-term searches take 50 milliseconds or less, and an exact Phrase search is often completed in 5 or 6 milliseconds or less. Of course, the dual core G5 is much faster than the G4 TiBook, but that’s not the major relevant factor for not encountering slowdowns. The most important factor is that I’ve got enough free RAM to not need Virtual Memory and consequent slowing down of database operations. When I change processes I might see a beach ball for a second as the computer retrieves new code and data from disk to RAM, but that’s it.
That conclusion about RAM seems confirmed by my experience with my Rev. B iMac with a 2 GHz G5 and 3 GB RAM. I can do very intensive work on a large DT Pro database for many hours without slowdowns, but eventually VM swap files will grow, with growing pageouts as well. Eventually, operations slow down. Quitting and relaunching DT Pro will keep me happy for a lot more work.
So free physical RAM is the key. Either have lots of RAM in the first place, or free up RAM by an occasional quit and relaunch or a computer restart. If it’s difficult to keep enough free RAM during intensive use of memory-demanding operations, try quitting other applications as an additional aid.
Your PowerBook should be much faster than my old TiBook, which is maxed out at 1 GB RAM. I can still get very satisfactory search speeds on my TiBook, with few and short beach balls, for 2 or three hours of work with a large DT Pro database before things start slowing down. Most single-term searches run no more than a second or two, sometimes less. When performance slows, I run Backup & Optimize and then quit and relaunch DT Pro. If that doesn’t free up enough RAM, it’s time for a restart.
One more tip: I did all my DT Pro database work on the old TiBook until May, 2005 when I got my iMac. I made a practice of ‘splitting out’ portions of my database that were totally unrelated to my main interests, to keep my main database to about 20,000,000 total words (2.72 GB disk space, as PDFs are stored in the database Files folder). For example, I’ve got a collection of Newton user lists that date back several years and constitutes by itself a sizable database. Splitting out the Newton database reduced memory requirements for my main database, and was no loss to the purposes of my main database. In sum, I’ve got a total of 6 DT Pro databases for various – and quite unrelated – purposes. I dump everything into my main database and once in a while export material that would better fit into one of the other databases. The result is that my TiBook can still run my main database at acceptable speed when Im on travel. But I suspect that it would indeed choke on your 52,000,000 word database.
Just a small tip: if you open the Search tool from the upper menu, you get a separate dialogue window that lets you enter a search term, but the search does not begin until you click the magnifying-glass button. No more spinning beach ball. All the operators are available there, plus some other nice additions. --WH
I agree with howarth. I do all my searches using the Tools > Search window.
And while viewing a document, you can select a word or phrase and choose Services > DEVONthink Pro > Lookup. A search window will open with the search term already entered. Select the appropriate search options, e.g., All Words or Phrase and hit Return or click the Search button (magnifying glass).
As I frequently change search options, I prefer the Search window because the options are visible and immediately available.
My PowerBook does have 2Gb of RAM, which is as much as I can put into it. Your idea of breaking out less-accessed parts of the database sounds like a good idea, though. A lot of it is purely long-term storage, and almost never gets accessed. I’ll give that a try.
Also, I do use Tools | Search exclusively now. I don’t even display a toolbar anymore.
