Scrolling treeview uses 90% CPU, very sluggish on large data

DEVONthink Tips
1

I have a DT Pro database that is around 2.4GB in size, with 1407 PDFs and ~ 650 HTML files. Word count is 560,282 unique, 120,705,955 total. In recent days I’ve found that scrolling the treeview has become sluggish and spikes the CPU up to 90% (on one core). I’m running DT Pro 2.3 on a 17" Macbook Pro, i7 with 8GB RAM so it’s not the machine.

After much fiddling, rebuilding database, running “repair”, etc. I finally copied the database using Finder. When I opened the copied database in DT the problem was done. So, I moved the old database into a temp folder and renamed the copied database to the same name as the original. When I opened the “copied-and-renamed-back-to-original-name” database the scrolling problem was back. Closing DT, renaming the database to any other filename and then reopening “fixes” the scrolling problem.

So, it seems that there is some other (cached?) data stored somewhere that is messed up that might be causing this problem.

Does anyone know a solution to this?

Also, how large can a DT database scale to? Am I reaching some sort of limitation?

Thanks,
Stuart.

2

Try Empty Cache in the DEVONthink menu. Then restart DEVONthink.

3

Seems to work great. I didn’t know about the “Empty Cache” menu item. I cleared the cache, quite DT and renamed database back to original filename. All seems to be well now – very fast.

Thanks a stack.

4

For what it’s worth, do you know what the limitations are in term of max number of documents that DT supports, max database size, etc?

5

Christian notes that the maximum size of a DEVONthink database is about 200,000 documents and about 300,000,000 total words.

But that doesn’t mean that I would attempt to run a database approaching that size (total word count is generally the most important measure) on my MB Air with 4 GB RAM.

My rule of thumb is that if I keep the total word count of the open databases at or below 40,000,000 total words I can keep free physical RAM available, so that I can hammer away at the databases without moving into use of Virtual Memory. Everything runs quickly.

But if free RAM gets used up, Apple’s Virtual Memory takes over, ‘extending’ the installed RAM by moving data back and forth between RAM and Virtual Memory swap files. That can mean a lot of disk access, and slowdowns resulting because read/write is much faster in RAM than on the disk (orders of magnitude faster than read/write on a conventional hard drive).

If your database slows down again, launch Activity Monitor.app (Applications > Utilities) and inspect the status of System Memory. I like to keep at least 700 MB of free RAM, zero pageouts and zero swap used.

I work with topical databases, each designed to meet a particular need or interest. My main database holds more than 25,000 references and more than 5,000 of my notes, with a total word count of just over 34,000,000 total words. I also have open with it several others for various purposes, with a total word count a bit under 40,000,000 total words.

Altogether I’ve got more than 250,000 documents among a number of databases. Needless to say, there’s not enough room on the 256 GB SSD in the Air to hold all of them. (But I can keep the entire collection available on an external drive for access when necessary. I’ve got them on a Pegasus Thunderbolt RAID drive.)

6

Thanks Bill. This is exactly the sort of info I was after.

7

Seems the problem is back. I added one new PDF to my “library” database and suddenly each time I scroll up/down in the list DT is sluggish and CPU spikes. Also, navigating the treeview using the cursor is sluggish … that is, each time I press an arrow key to move around in the treeview it takes a little while to process. For example, if I hold down the up arrow the selected row moves up slowly and CPU spikes to 90% on 1 core again.

Any ideas? I’ve tried clearing cache, restarting DT - to no effect. Also tried renaming the database but that had no effect as well.

I’ve tried rebuilding the database too but the problem persists. Thoughts?

8

Behaviour is inconsistent. If I open a new window, with treeview rooted at some child node from the main “Library” node and (in the new window) expand a few child nodes and scroll up/down then scrolling (with inertial scrolling on) is fast. Back in the “main” window it’s very sluggish and results in CPU spiking. Ironically, closing the “main” window and then navigating back up the treeview to the root in the “new window” (using Go->Enclosing Group) results in fast scrolling, even when I end up back at the root group. Something within DT seems to get “busy” and then scrolling gets sluggish.

Is there any chance that the mouse’s inertial scrolling (I have it turned on) can cause difficulties?

9

Another quick note… it seems that switching to “View -> As List” is what causes the sluggishness. Switching a view to use “View -> As Split” results in the sluggish scrolling going away. Switching view back to “As List” immediately triggers the sluggish scrolling again.

Memory usage is constant (and low) at around 200MB, with around 4GB RAM currently free (2.8GB free, 1.2GB inactive) and about 4GB used (1.1GB wired, 2.9GB active).

So it seems that a TreeView + Table (the Command+2 option — View -> As List) is the problem while in a view that’s only a Treeview (Cmd+4 or Cmd+5… View -> As Split or View -> As Three Panes) the treeview scrolls consistently and smoothly.

10

OS and DT versions?
Reboot recently?
Any Console messages that might be useful (please, though … don’t post the content of your Console … send it to DT Support, who can do more with it than we can on the Forum).

What is “treeview”?

Is this still true?

Is Adobe Reader installed? It can cause problems (though, not usually endemic, persistent problems such as those you write about).

So, if the problem went away when you created a new database (not a rebuilt instance of the bad database) why not just make the new database, put your data there, and have done with the bad old one? Sometimes its just not worth diagnosing what went wrong, if what matters is getting up and running.

11

OS X Lion, 10.7.1. DT Pro 2.3

Yes. Doesn’t make any difference.

How do I get “Console” messages…?

By “treeview” I mean the left-hand panel of the DT view. For example, when I select “View->as Split”, the left-hand part of the screen has a tree-like structure showing the hierarchy of folders while the right-hand part shows the contents of the currently selected document.

When in “View as Split” or “View as Three Panes” the left-hand panel shows a “tree view” that has only item (group or file) names. When switching to “View->as List”, the view that appears shows a combination of tree-view (hierarchy of groups/items) as well as a sort of table structure – with URL, Modified, Kind and Size columns to the right of the “treeview” and “Flags” to the left of the “treeview”.

No, as I mentioned copying the database, rebuilding it, renaming it, etc (I’ve tried many combinations) doesn’t fix the sluggish scrolling. It seemed to fix things when I first tried it, but I realise now that renaming the database makes DT lose its “memory” of views that were opened, etc (since the database filename has changed).

No… I detest Adobe Reader… that’s one of the reasons why I switched to Mac.

As I mentioned above, renaming and/or rebuilding hasn’t fixed the issue… I suspect my initial “It’s fixed!” reaction was due to simply viewing the renamed database using a different view (“as split”) instead of “as list”. Basically, it’s the “as list” view that becomes sluggish.

12

.

13

Stuart.

I have experienced an issue very similar to this. Though it didn’t completely eliminate the problem, disabling my antivirus program scanning greatly helped. If you are running such a program, you might give it a try.

Tom S.

14

Hi Tom,

I don’t have an on-access scanner - I use Clam AV which I run once a week manually (reminder in iCal :slight_smile: ). But thanks for the heads up.

15

The Console.app is in Applications > Utilities. It produces logs representing “what’s happening” as processes are underway. Each message is date/time stamped. Messages logged during a procedure may provide information useful to the developers in diagnosing a problem.

16

Ah cool. I launched Console.app and there are lots of different “log” files with all sorts of stuff in. Which file/s do I need to send to the DevonThink developers?