Helping to debug DEVONthink

DEVONthink Tips
1

Hi, guys.

I’ve had a problem with DT for some months now. I can send in a bug report but I’m pretty sure saying, “Sometimes I make a change and I get the spinning beach ball for 30 seconds.” will be pretty useless to you. I’d like to do something to provide you with more information to help track it down.

I can look at the system log and I can tell you there’s nothing remarkable to indicate a problem. If I sart DT from a terminal, there are no messages sent there (presumably the terminal is the stdout and stderr when I do that?).

Is there a way to start DT from the terminal with a “debug” option? Or perhaps there’s a “verbose” option that send more meaages to the terminal? Is there a log file somewhere I should be looking at with entries I can send that might give you a clue as to what’s going on?

Any suggestions would be helpful.

Tom S.

2

I’m somewhat surprised there was no reply to this. I suppose I could just report the bug but I thought maybe exposing suggestions to the forum would be more generally helpful.

Nothing to add?

Tom S.

3

Tom, the most frequent cause of the ‘spinning ball’ issue is running out of free RAM, and consequent heavy use of Virtual Memory swap files. Virtual Memory lets a procedure go to completion in the absence of free RAM, but as it involves swapping data back and forth between RAM and the HD - and read/write speeds are orders or magnitude slower on the HD - slowdowns happen.

Christian Grunenberg has noted on the forum that the ‘limits’ of a DEVONthink database are currently about 200,000 documents and about 300,000,000 total words. But I would find a database of that size unsatisfactory in performance. I’m spoiled; I want want most of my searches to produce results in 50 milliseconds or less, and See Also to pop up suggestions almost instantaneously.

DEVONthink’s File > Database Properties panel displays 3 measurements of database size, the file storage size, the number of documents and the total number of words in the database. The single most important measure is usually the total number or words.

In toto, I manage more than 250,000 documents among a number of databases. I do most of my work on laptops that have 4 GB installed RAM.

To satisfy my preference for quick performance I use topically-designed databases that satisfy a particular need or interest. If I use a rule of thumb that the aggregate number of words in my open databases is in the neighborhood of 40,000,000 total words or less, I get fast performance. With only DEVONthink Pro Office open with my set of open databases, I still have almost half the physical RAM free, so there are no page outs and no use of Virtual memory.

I treat my databases as information ‘Lego blocks’ that I can assemble for a given purpose.

If your database has grown very large and takes up most free RAM in operation, you might consider splitting it into two or more smaller databases.

But I usually have several other applications open, as well, and they use RAM resources also. Apple’s memory management is pretty good, but over time RAM will tend to get ‘clogged’ by no longer used data that’s reducing the amount of free RAM. Safari tends to be a memory hog, taking up more and more RAM as time goes on. If I’ve been doing a lot of browsing while collecting data into my databases, it will eventually fill up all my available free RAM. Quitting Safari often doesn’t release all the RAM it had been using. (By contrast, DEVONagent Pro doesn’t hog RAM.)

I’ve been using an OS X maintenance utility, C ocktail, for a long time. The current release includes a feature that I find very useful to keep my Macs running at full speed, a feature to purge inactive RAM and optimize free RAM. As the utility places a little icon in the menubar to indicate visually the amount of remaining free RAM, I can avoid page outs (and use of Virtual Memory swap files) by temporarily quitting Safari or another app that’s using lots of RAM, and purging inactive RAM. Presto! With my normal set of open databases in DT Pro Office, I’ve still got 1,500 to 2,000 MB free RAM, and I never see a spinnig ball.

My favorite laptop these days is a 13" MacBook Air with i7 CPU and 256 GB SSD. My DT Pro Office databases load very quickly from the solid state drive. True, the faster read/write speeds of the SSD would somewhat mitigate slowdowns when Virtual Memory swap files are being used, but that would also involve a lot of ‘hammering’ of the SSD when swap files are in heavy use, something I want to avoid.

4

Bill,

Thanks for the very lengthy reply. Very informative.

My database is less than 8000 documents and 50,000,000 total words (1,600,000 unique). I see the problem on all of my machines including the one I’m on now which has 8 GB of RAM. I also don’t see the same slow down in other programs and I don’t have a problem switching to other programs to work while waiting for the beach ball to finish. So I kind of doubt that’s the issue.

I think I’ll download the C ocktail utility just for grins and see what shakes out.

My gut feeling is that its the database. Perhaps there are some characters somewhere its having a hard time with. I’ve tried rebuilding with no apparent effect. The database is also on a USB thumb drive and I have considered the possibility that is causing the problem but I’ve been poking around this forum for more than a year and I haven’t seen anyone doing this where its an issue.

Again, it would be nice if there was a log file somewhere so I could see if there’s some command that’s timing out or some sort of error.

Thanks again for the reply.

Tom S.

5

Dear all,

I finally figured this out. I’m running Sophos Anti-Virus. Turning off the “on-access scanner” basically eliminated the “spinning beach ball problem”. There is an option to exclude specific files and directories from this scan. Turning the scanner back on and specifying the database as an excluded file/directory appears to help.

Its been so long since I ran Windows, I forgot what these anti-virus programs could do. I only recently installed Sophos as the malware problems appear to be finding their way slowly to the Mac. I expect this problem will begin to crop up more regularly as more people do so.

Anyway, for what its worth.

Tom S.

6

Personally, I don’t install antivirus software on my Macs.

I’ve told OS X not to automatically open any downloaded software. Instead, if I double-click an executable file, OS X will warn me that it was downloaded from the Internet, and offer to show me the Web site from which it came. Only if I agree to open it will it be launched.

When I run the C0cktail maintenance utility it scans the computer for any known malware. None has ever been found. Apple has provided security updates that are ‘aware’ of current malware apps as well.

Another line of protection is that I’m careful about the software that I install, and download it only from trusted sites.

Probably the biggest sources of malware are pirated software sites and porn sites. I’m not interested in either kind - indeed, there may be poetic justice if hackers have inserted some devious code into a pirated application.

As always, you are at greater risk of losing money to spammers than to spyware on a Mac. “Pfishing” spam, masquerading as a message from your bank, PayPal, Amazon, etc - sometimes very well crafted in appearance - tempts the viewer to click on a login link in the message. Don’t do that. You’ve just given some bad guys your login information.

7

This is what worries me. Like you, I can recognize 99.9% of these immediately. But I do get relevant email from students and a others who I don’t readily know by name. It is exceedingly rare but if I get an email that is phrased in just the right way, I could fall for it.

As an example, I got a very clever email which appeared to be from the IT department at my university. The email address was a university address and appeared to be correct. I clicked on the link and started to fill out a form. It was only when the form asked for my social security number that I stopped and got out of there.

I do generally trust myself not to download infected files and programs and the Mac App Store should help ensure that doesn’t happen even more. But I am worried that I’ll get an email from someone who is passing an infected file to me. Its never happened and right now its a rare event for everyone. But I can see it being a problem in the future.

Generally speaking, best for me to keep the extra layer of protection.

Thanks, again for the long response above regarding my problem, Bill. It was kind of you to take the time.

Tom S.

8

I find that paying attention to the tooltip that appears when hovering over links in Apple Mail frequently reveals that the link isn’t what it pretends to be. The tooltip shows the actually destination URL. If it’s supposed to be something-something @ “wachovia.com” but instead says “comwachovia.al” then I zap that message without going further.

Not a 100% guarantee of accuracy, but most legitimate sources don’t obfuscate their URLs.

As for attachments from notionally trusted sources - if I get an attachment and didn’t expect one, I ask before opening (“did you mean to send me the secret budget for the CIA”?).

9

That’s a good tip. Thanks.

Since many of the people I know aren’t running AV software, they might well purposely send me a file as an attachment that they don’t know is infected. More likely, someone might pass me an infected file via a USB drive.

Again, I think this is really rare on Macs right now. As in “practically unheard of”. But I’m assuming its just a matter of time.

Tom S.

10

Fortunately, if a Mac is instructed not to automatically launch new apps, the level of security is much, much better than has the case been on Windows computers. The kind of virus that’s been such a plague in the Windows world depends on executable code being executed without the user’s knowledge or permission.

The recent Defender malware tries to get a Mac user to launch it by promising a free goodie. It’s an old human engineering trick akin to the Nigerian scam. Common sense is the best defense against malware. “Beware Greeks bearing gifts.” Defender is a modern variant of the Trojan horse, a human engineering trick used to get concealed warriors inside the protective fortifications.