Rebuild Database Problem

I’m new to DevonThink Pro and am working my way through the manual and these forums but haven’t found the solution to my problem.

I did a lot of organizational work with my DevonThink database yesterday and thought it would be prudent to use the Rebuild Database command to put everything in order. However, when I did this I got a new database with more than twice the number of entries, many apparent duplicates, and a loss of some of my carefully crafted groups.

I was able to save the day by reverting to a previously saved database, but I’m not going to be using the Rebuild Database command until I understand what went wrong.

I don’t know if this is germane but it may be: I’ve set backups in preferences to hourly and keep 10 backups (hey, better safe than sorry I figured).

Also, I’ve been keeping my DevonThink Pro database file on a spacious thumb drive. I have the program itself installed on my home and office Macs but I run the database off the thumb drive. Could this be contributing to the problem in some way?

Many thanks in advance for any help with this issue and for maintaining such a useful forum!

Cordially,

Jonathan

Hi, Jonathan:

You didn’t say what “spacious” means. How big is your database (size of the file in the Finder), and how big is your thumb drive?

I would recommend cutting back the number of internal Backup folders to 3, as 10 is really overkill, especially because you have limited storage space on which the database resides.

Personally, I’m comfortable with 3 internal Backup folders because I habitually do two things: I use the DT Pro Scripts > Export > Backup Archive script to store the smallest possible, compressed and dated external backup of my database periodically, and especially before I try anything unusual. And I manually trigger Tools > Verify & Repair followed by Tools > Backup & Optimize after investing substantial time and effort in making changes to the database – usually at break time.

I don’t regard Tools > Rebuild Database as a normal or routine procedure. In over 4 years I’ve used Rebuild Database perhaps 3 times, and the last time was probably about 2 years ago. I would recommend a rebuild of an old DT PE database that was created with an early version of DT PE. But Rebuild isn’t required for routine preventive maintenance.

Your thumb drive should be several times the size of your database, to give your database room to ‘sprawl’.

To feel well backed up I would suggest that you periodically run the Backup Archive script and store the compressed backup on the computer you are using at the time. So: you’ve worked on the database at home. Before leaving for the office, run the script and save the archive to your home computer’s drive. If you modify the database at the office, repeat that routine on that computer. As you have a relatively small database (“spacious thumb drive” is a contradiction) this will run very quickly.

Thumb drives (I assume a USB flash drive) are very much less reliable than hard drives, especially if you use them like a hard drive. Which is to say that if you frequently write data to a flash drive, it will “wear out” much faster than a hard drive.

As your database grows, it will probably no longer fit (with needed “extra” space) on your thumb drive. At that point I would switch to carrying the compressed archive of the latest version of the database on the thumb drive. When you move, e.g. to the office computer, copy the archive to your office computer, uncompress it and double-click on it to open it under DT Pro. If you make changes, archive the new version of your database to the thumb drive. And so on.

Dear Bill,

Thanks for your detailed—and very prompt—reply. I’d obviously fallen prey to the old, “If a little is good, a lot is better” fallacy.

I’ll dial my backups back to three and rely on the scripted backups, a habit that it will be easy to acquire given the importance of my files to my work.

I’d always assumed that a USB thumb drive was a better form of storage, so I’m grateful for the information.

For the record, mine is a 2 GB thumb drive holding my 43 MB (yes, MB) file…I’m just getting started with DevonThink Pro and I didn’t know how much capacity I’d need.

Thanks again for the clarification. The more I explore, the richer the features I’m finding, many of which aren’t immediately obvious to me. That’s why this forum is so useful.

With gratitude,

Jonathan

Yikes! That sounds more severe than problems I’ve noticed with Rebuild Database, or after exporting all items and importing them into a new database (basically what Rebuild Database does).

Out of curiosity, did you have any replicated items (especially groups) in your original database?

I DID have some replicated items, but only out of ignorance. I wasn’t aware that I’d created them and they confounded me for quite a while—I kept trying to change the font color until I figured out that red and blue titles meant something, and then I spent 30 minutes figuring out what a replicant really was and how it worked.

At any rate, it really didn’t make any sense that the Rebuild so totally mashed up things. I’ll try it again on a duplicate of my database to see what happens.

If I can impose on you for one more question: are all of the Tools commands (verify, backup, rebuild) to be done with the database file closed?

With thanks,

Jonathan

That’s interesting, even if it’s only anecdotal information right now.

If you have replicated and/or duplicated items you might want to do a pre/post-rebuild comparison to make sure they’re restored in the rebuilt database. Unfortunately there’s no convenient way to do a “deep” comparison (yet). If nothing more, you can run File > Database Properties… (option-command-P) before/after and compare those results; saving save screenshots of that window is handy. For a long time I’ve noticed rebuilt databases often have different before/after numbers, seemingly related to dup/rep items and maybe other things. It’s tricky figuring out exactly which items are affected (and how) in large databases.

If the command line doesn’t scare you, Eric Fedel has written some useful scripts to interface with DT databases. The dt_contents.sh and dt_record_info.sh scripts can summarize database information that would be useful for comparison purposes although those are only building blocks rather than an easy end-user solution.

Sorry if that was overwhelming; it’s obviously a passionate topic of mine.

You’re not imposing. :slight_smile:

All those commands process whatever database is currently open and are inactive (i.e. greyed-out under the Tools menu) when the database is closed. Very few commands are active without an open database.

If you want to do some kind of database maintenance outside of DEVONthink (e.g. making a copy with Finder) then it’s highly recommended to close it first so there’s minimal risk of data corruption.

Hope that’s more helpful than confusing.