Rebuilding database doesn't restore replications

I ran verify and repair a couple of days ago and with horror discovered 97 checksum errors that wouldn’t go away after a couple of attempted repairs.

In the end there was nothing for it but to rebuild the database. Not something I was looking forward to since I have over 66 million words and some 127,000 documents or items in my database.

When I tried to rebuild, the files would export OK, but it has crashed every time trying to reimport them. In the end I had to do the task manually, which took forever.

Problem is, I had replicated many groups. In the rebuilt database, the CONTENTS of many groups are now replicated, but the groups themselves are not. Due to the way I am structuring my data, this gives me a massive problem, and I can’t see any way around it except to delete replicated data, and re-replicate the originals. This is not a small task, since these replicants must be located within 1,000 or more separate groups.

Firstly, should a re-import restore replications? If not why not? And can something be done about it, because this is not something I ever want to have to go through again.

Secondly, does anybody have any suggestions that can solve the problem for me.

Your responses are eagerly awaited.

Rollo

I hope these checksum errors don’t have anything to do with Leopard. I upgraded to Leopard a few days ago, and the checksum errors appeared soon after that. Coincidence? Or are the two connected?

Rollo

Hi, Rollo. Both from my training in science and the philosophy I tried to impart when I took on the job of agency quality assurance manager for the last two years before I retired, I push the approach of recognizing in advance things that might go wrong and taking actions to prevent them.

At the moment we are migrating from a very stable OS X 10.4.10 operating system and a very stable version of DEVONthink Pro/Office to the new environment under the initial release of OS X 10.5. Could one expect that there might be problems involved in that migration? Of course. Anyone who has followed developer comments during Apple’s coding of Leopard would be aware that there have been some bugs. And there’s the traditional IT manager position that one shouldn’t migrate mission-critical work to a new OS until after the first one or two updates.

Precaution 1: If in the midst of important work with a database, try to avoid the temptation to immediately switch to the new OS, or set up a fail-safe system. In my case I was in the midst of a project when Leopard was released. I finished the project before I installed Leopard. I would probably have waited still longer, except that I needed to participate in beta testing of fixes to database compatibility problems with Leopard.

Precaution 2: Protect your data in case anything does go wrong. At a minimum, make an external backup under the older stable OS and store it on an external device for possible use. DT Pro/Office provides a good system to do this using the Backup Archive script. In my case I have a second computer still running OS X 10.4.10 to which I can revert if necessary.

Precaution 3: Increase vigilance on error detection and make frequent backups. It’s a good idea to manually run Tools > Verify & Repair to check for database errors. It’s a very good idea not to depend on scheduled backups after making database changes, but to run the Backup Operation whenever significant changes have been made to the database, usually when one is taking a break for a while.

Precaution 4: If something does go wrong, try to analyze the situation before taking corrective action.

Let’s talk about Rollo’s experience. In the first place, I would not have done a database Rebuild in that situation. Instead, I would have gone to either a current internal or external backup.

Why? Because – from forum discussions and other sources – I know that a Rebuild operation is likely to encounter problems with some files, especially WebArchive files. That’s because there is some instability in the initial release of Leopard’s WebKit. And for all I know there might be stability problems affecting other files, perhaps PDFs. The Rebuild operation requires two stages of action on files, one to export them and another to import them. So the potential of errors is magnified.

Rollo also reports that there were changes in the organizational structure after the Rebuild. I have no idea why that happened, although Christian may be able to comment.

I don’t want to lose data or, for that matter, the work I’ve done in organizing material.

So my approach would be to work with a recent backup of the database, either by using Restore Backup or by launching a recent external archive produced by Backup Archive. I would then immediately run Tools > Verify & Repair to check for database integrity. If no errors, I would proceed with database use – and with frequent manual backups using Backup Archive.

Finally, if I were to encounter so many problems that it became difficult to get my work done, I would revert to my fail-safe system – running the database on my other computer that’s still running OS X 10.4.10.

My actual experience under Leopard: So far, so good. But I’ve got an advantage over most users. I didn’t install Leopard until there was a beta of DT Pro Office version 1.3.4 that had already corrected many of the compatibility problems with Leopard. Testing of the update is continuing. Plans are to release the update via an announcement on the forum on or before Friday of next week.

I’ve been checking database integrity every now and then, using Tools > Verify & Repair. No errors have been reported.

I have only rarely captured material as WebArchive, because that file format is Mac-only, and there are stability problems. Most of my captures are as rich text notes of selected text/images. That is working well under the beta I’m using. (But we caution against doing that with version 1.3.3.)

Tip: If you wish to capture all of the content of a Web page under Leopard, use the script to print and save to the database as PDF. Leopard’s PDF print routine now captures working hyperlinks, which I consider a major improvement. Unlike a WebArchive file, you can send a PDF to a Windows-using colleague without compatibility problems.

Warning to users of OS X 10.3.9: DT Pro/Office 1.3.3 will be the last version that’s compatible with 10.3.9. Version 1.3.4 will no longer be compatible with 10.3.9 – so don’t download it! The same warning applies to users of DEVONthink Personal. The current posted release will be the last one compatible with 10.3.9.

Comment on Leopard: Although there are teething problems, Apple’s latest OS is a magnificent achievement and will be the future of DEVONthink development. It’s really exciting.