Suggestions for future release to help prevent/solve crashes

I have a suggestion with two elements: a more robust ability for the programme to survive unexpected disruptions; a comprehensive software uninstaller.

Devonthink seems to me to be very fragile and databases become inoperable in several different scenarios. If, for example, another application crashes and forces a hard restart of the computer; if the computer loses power unexpectedly; if the computer is turned off without Devonthink having been manually quitted.

Each of the above scenarios will, I think, prompt the ‘Database is currently in use’ message. I have had to deal with this problem on three separate occasions. On one occasion I was at a loss as to why this happened, while on another deleting each database’s lock file solved the issue.

On the other two occasions I have had to uninstall Devonthink. This is quite a long process, and if a file is missed it will potentially disrupt the fresh instal. The guidelines for uninstalling Devonthink provided in the manual are nowhere near comprehensive enough, which necessitates a spotlight search to identify all Devonthink-linked files for removal.

Devonthink is a programme that I like very much, but I find myself fretting every time I re-open it in case I encounter another problem. I hope these suggestions might prove useful.

The “in use” message will result if databases were not properly closed as a result of a System or application crash, a forced shutdown, a loss of power, etc. Next time the DEVONthink application is launched the message will be displayed. After pressing the Continue button to open a database, it’s prudent to run Tools > Verify & Repair to check for possible damage.

DEVONthink databases are normally very stable. My Main database has been evolving from my initial DEVONthink database started back in 2002. I haven’t seen an error message in it or any of my other databases for a long time, and haven’t needed to resort to a backup for years.

My current working Mac is a MacBook Pro (Retina) with i7 CPU, 500 GB SSD and 16 GB RAM. I moved all my working files and databases to it on 5 November, 2012. The CrashReporter folder at ~/Library/Logs/ is empty; there hasn’t been a crash of DEVONthink Pro Office or any other application since I started using this Mac.

Nevertheless, it’s wise to maintain a good backup strategy that contemplates the spectrum of things that could go wrong, from a crashed disk to a burglary or fire that causes loss of all one’s computer equipment. DEVONthink provides internal backups of the state of the database at a given time. The frequency of internal backups can be set in Preferences. There’s also Backup & Optimize under the Tools menu. After making significant changes to a database such as addition of large amounts of new content or reorganization changes, I make a habit of running Verify & Repair, followed (if no errors) by Backup & Optimize, to ensure a sound recent internal backup should something go wrong (including a dumb experiment in reorganization of groups). In that event, Restore Backup should recover to a sound working database.

But those internal backups would be of no use if the disk crashes. I also use Time Machine, free as a utility in OS X. My Time Machine backups are saved to a 3 TB external RAID drive. As I like working on a laptop, it’s not always connected to that external drive. When I return it to my desk, I Quit DEVONthink and immediately do a Time Machine backup. Then I do scheduled backups in Time Machine while the laptop is “docked”. (Although there are technical reasons to prefer backup of closed databases, I’ve tested the state of backups made with open databases, and have never encountered a problem with them.) So, should I encounter a damaged database, I would be able to go back in time in my Time Machine files to recover it in undamaged state.

But a burglary or fire might result in the loss of my computers and the external drive. As a final level of backup, I periodically update Database Archives (File > Export > Database Archive) of my most important databases to a portable drive that’s stored in a safety deposit box at my bank. I prefer this to cloud storage, as I have complete control of it, and could recover my large databases much more quickly than if they had been stored on the cloud.

I can’t recall any instance in my long use of DEVONthink that required a reinstallation of the application, although that is a possibility in the event of a corrupted application file, or the replacement of a corrupted preferences file, for example. That just hasn’t happened to me.

It’s important to keep the operating system “clean”. Every few weeks, I run C0cktail to check Preferences, rotate logs, empty caches, etc. (the free OnyX maintenance utility is also recommended). I’ll run Disk Utility to verify the disk directory. I avoid installing “hacks” to the operating system that could modify OS X in ways that could result in computer errors. Even with 16 GB RAM, I run C0cktail’s little utility to purge inactive RAM and optimize memory, so that I keep lots of free RAM available and don’t experience pageouts.

If database problems happen, we encourage users to send a message to Support, including any recent DEVONthink crash report and a description of what was going on when the problem occurred.

Why do problems happen? A quick summary of causes:

  1. A flaky computer. The operating system has become unstable, or memory errors have accumulated that result in strange behavior, sometimes causing database problems. Sometimes this is transient and can be corrected simply by a Restart. Sometimes this results from installation of utilities that improperly hack OS X and make the computer unstable (remedy is to remove the offending utility).

  2. Memory errors resulting from insufficient memory to support the size (number of documents, total word count) of open databases, or from a procedure that overloads available memory (such as moving a very large block of data). Although the current 64-bit operation of DEVONthink has reduced the impact of such errors, they are still possible.

  3. Insufficient free disk space (including use of a sparse disk image that has run out of free storage space). Although Macs have been getting larger disks over time, Apple engineers recommend keeping at least 15% of the storage space free, to allow room for temporary files required by the operating system and applications. Running out of free disk space can result in overwriting files by the operating system, resulting in lost data and possible database damage. Note that some procedures, such as OCR of a large PDF, can need a LOT of disk space for temporary files.

  4. A damaged disk directory, which can lose track of where and what files are stored, and possibly overwrite data. The remedy is to run Disk Utility > Repair Disk; but that won’t solve some existing problems resulting from data overwrites.

  5. Sharing & Permissions errors can prevent access to files, or prevent their modification.

  6. Hardware problems. Although not common, such problems including flaky RAM or a failing hard drive do happen.

Bill, thanks for taking the time to put together such a lengthy reply. I am, however, a little mystified as to why you felt it necessary to type up (or copy & paste) what is essentially a back-up and computer maintenance guide for dummies. To alleviate your concerns, I run a mid 2010 13" MBP, with upgraded RAM, an SSD in the main caddy and secondary HDD in the optibay. Neither RAM nor disk space are problems for me. My computer is not flaky, I have no permissions errors … etc. I take care of my computer, use TM to back up and also use CarbonCopyCloner to back-up my boot drive at two separate, off-site locations. I agree that fire and burglary are to be feared. You see the point I’m trying to make, I’m sure.

Now, you say that the “in use” message will appear if the databases were not properly closed. That’s exactly what I was trying to say. For instance, my niece shut down my mac last weekend without quitting DTPO. When I re-booted, I received the message. Your advice is to click continue and then run Verify and Repair - I would gladly have done so if my databases had opened, but they did not. Instead the left navigation pane was empty. I restored from TM back-up, but the problem persisted even after the lock file was deleted. I couldn’t solve the issue so uninstalled.

Now if I am using DTPO incorrectly and somehow have my settings set-up in such a way that, once the application thinks a database is already in use, it will not display it in the navigation pane (thus preventing me from verifying and repairing), then I’d be very grateful to be shown where I’m going wrong.

I think perhaps my choice of the word “robust” was poor. Yes, the databases are very stable - I have no problems at all on that score. The programme, on the other hand, can surely be set up to be better able to survive an unexpected closure.

My point re the uninstallation guidelines in the help manual being woefully inadequate still stands.