Inconsistencies - Rebuild or die?

[Spailpin]

My MacBook suffered a catastrophic encounter with a cup of tea which means that I have now restored my system onto a new device from a Time Machine backup. When I opened my main Database (mostly imported files - 28 GB), it tells me the usual message about still being open. It says there are 2 major errors and I can Ignore or Rebuild.

Two errors is not too bad, but when I selected Rebuild, it tells me that there are 11,256,431 inconsistencies!

I ask because I don’t think there’s a way to check what inconsistencies are and I don’t think I have much choice but to Rebuild or go to an older Time Machine backup?

Any guidance welcome.

J

[BLUEFROG]

In DEVONthink, hold the Option key and select Help > Report Bug.

[comnio]

This is not a bug, and waiting for a new version of the software hoping it will fix your problem one day is not a practical solution.

11 million inconsistencies doesn’t mean your data is gone, but it does mean DEVONthink’s internal metadata layer is badly scrambled. This usually happens after a system restore, migration, disk-level corruption, or a Time Machine restore made while DEVONthink was open.

Inconsistencies refer to DEVONthink’s internal metadata tables, not your actual documents.

Your 17 missing files are the only files DEVONthink believes are gone.

The 11 million inconsistencies mean: broken internal references, bad UUID references, stale sync metadata, orphaned internal records, and folder structure mismatches from: Time Machine restore, macOS migration, or abrupt shutdown while DT was open.

When inconsistencies are counted in the millions, the normal repair tool cannot fix the metadata layer. It can only correct small issues.

You have two options:

File > Rebuild Database

This creates a new, clean database by re-importing all valid documents and regenerating metadata.

Pros

• Safest option

• Keeps all readable documents

• Strips out corruption

• Removes millions of ghost records

Cons

• You may lose: custom metadata, some annotations, last-opened positions, some groups with empty references. But you do not lose your actual documents.

Or, if you have a TM backup from before the corruption event and that DT database was fully closed at the time, restoring that version may give you an intact copy.

But if multiple backups were made after corruption began, rebuilding is safer.

DEVONthink does not provide a report of individual inconsistency items because they’re low-level internal references. What matters is whether the actual documents are present, and DEVONthink tells you that: only 17 missing files.

If this were my database, I’d do exactly this:

1. Duplicate the database file in Finder (for safety)

2. Open the duplicate in DEVONthink

3. Run File > Rebuild Database

4. Let the rebuild complete — it can take hours for large DBs

5. Review:

   • smart groups

   • tags

   • custom metadata

   • group structure

6. Restore anything missing manually from the original if needed

Almost always, users end up with a clean, healthy DB and no document loss.

Time Machine restores made while DEVONthink is open will always cause internal inconsistencies.

In the future, before restoring:

• Close DEVONthink

• Exclude the ~/Library/Application Support/DEVONthink 3 folder from TM (or whatever your version names it)

• Backup DT databases manually or with a tool like ChronoSync

[Siegbert_Schnoesel]

While it’s explicitly stated that sync stores are not backups, you could try to sync your database to your new Mac if you had a sync store created previous to your Mac failure (and didn’t sync the corrupt DB since then).

[Spailpin]

Just to thank everyone for their input here, and also to Jim at Devontechnologies Support.

I rebuilt the database and all appears to be well. There were 17 missing files and 2 empty files. These were all purged in the rebuild and otherwise the database is okay. I don’t use custom tags (tags are groups and groups are tags, so I just never saw the point) and my annotations were preserved.

@comnio ‘s discussion on the inconsistencies is enlightening. – I’m still not sure what actually the source of the problem was, as the Time Machine restore was a full-system migration. Although, I hadn’t backed up the Applications folder so I had to download Devonthink after the restore had completed. It’s also possible that the Time Machine backup was created while the Database was open. Either way, it won’t bother me now, as I have my Database back up and running.

I still had two orphaned files after the rebuild. One of these was fixed by verifying the database. The other is still there. It looks like this:

~/Databases/MyDatabase.dtBase2/Files.noindex/tsv/3/New Sheet 2.tsv

I’m not too sure what to do with it?

[BLUEFROG]

You’re welcome
Orphaned documents can be filed manually or removed if you don’t need them.