Help diagnosing why databases keep getting corrupted

DEVONthink
1

I’m doing everything I can think of to keep things working well, but errors keeps occuring. My biggest pain-point is currently missing files. I cannot link the phenomena to any action or event, but seemingly out of nowhere, I will have 500+ missing files in the database that need to be redownloaded from the cloud. Once I logged in to find there were 1000+ orphaned files, but upon further inspection, those files did still exist in their proper places within the database, so I’m not sure where they came from.

I’m sure it’s user error, any help would be greatly appreciated.

Here’s relevant data / what I’m doing:

  • DT is installed on the system volume of a new Macbook Pro.
  • The path to my databases is ~/Databases
  • All files are imported, none are indexed in place.
  • In DT preferences, I have two sync locations set up: dropbox and iCloud.
  • Time machine is not backing up the databases folder or the “Application Support/DevonThink 3” folder.
  • I am not exceeding the maximum database size I specified on database creation. Largest is 3.6GB (mostly PDF’s) of 10GB.
  • One of the databases are encrypted, the rest are not. The errors have appeared in multiple databases, but do seem to be more frequent in the encrypted database.

What am I missing?

2

Welcome to this alternative, pandemic-free reality that is this forum.

Do you mean that? Or do you mean in DT preferences? If the latter, why two sync locations? What are you syncing with/to?

No it isn’t :wink: Well, I don’t think it is, anyway - why do you think it is?

3

Whoops, I did indeed mean DT preferences. I originally set up sync to just icloud, but I started running out of space (I use icloud for more than this) so I set it up for dropbox as well. Syncing with dropbox was also much faster than icloud for some reason, so I just kept them both. Figured some extra redundancy would be a good thing - was I mistaken?

As for the file paths, this is the “Path” given in the inspector within DT. Upon further inspection, it appears you are correct and I was mistaken. When I click “Show in finder” my path is either /Volumes/ or ~/Databases depending on whether or not they are encrypted. So “path” is the index maybe? Thanks for pointing this error out. I’ll update the OP.

And one of my databases is encrypted, another error in my OP. My apologies everyone, it’s difficult enough to help a stranger troubleshoot through a discussion forum without them screwing up multiple details in their question.

4

To help @blanc (and I’ll pitch in) with others, a bit more clarification is needed.

: what makes you conclude the databases are “corrupted”? are you calling missing files “corruption”? What are symptoms you notice for “corruption”?

: can you clarify “upon further inspection, those files did still exist in their proper places within the databse, so I’m not sure where they came from”… You say this is past tense. So is this state no longer the case? What is the current state? Once imported DEVONthink doesn’t remember where files came from (far as I know, and if it does it’s irrelevant since I normally delete important files at their source), so you have to figure out if the “came from” location is valid, or the DEVONthink version is valid as the “current and good” version.

: You mention were DEVONthink (DT) is installed on “system” volume. The normal place for the application to be installed in “Macintosh HD/Applications”. Is it not there? If not there, a reason for why you installed differently? Why do you think this is relevant to the problem?

: What version of DEVONthink do you use? 3.6.1 is the current version.

: You say all files are imported, but not indexed in place. “Missing Files” sometimes happens with indexed files when the file does indeed go “missing”, e.g. it is not where DEVONthink last saw the file. DEVONthink doesn’t go hunting for missing files. What makes you believe the files ar “missing” and are you sure they are not at indexed locations?

: Are there any other computers involved? Where else do you have DEVONthink or DEVONthink ToGo (the IOS app) installed and running? and if so what versions. Do they also see “missing” corruption?

: If you are seeing file corruption (or is is just “missing files”), that could be occurring on either or all of your machines running DEVONthink, and could have nothing to do with DEVONthink. Have you used the OSX “Disk Utility” to do a repair on your hard disks(?) to make sure all is well? That should fix file/folder/disk corruption (but not the “missing file” problem, if that the problem).

: Since you didn’t mention other computers, my hunch is that maybe only one is involved? If this is true, then why have sync? Or maybe you do have other machines involved, and then sync is appropriate. But having two “cloud” sync locations doens’t help matters, especially when you say you were unhappy with Apple iCloud. (before suggesting changes, I’d for sure recommend you “clean” the iCloud sync location to stop using it on all devices.)

: Why is Time Machine NOT backing up “Application Support/DevonThink 3”? Did you exclude it in TimeMachine? If so, why? Is this folder actually “/Users/YOURUSERID/Library/Application Support”? Why do you think this backup exclusion is relevant? Is TimeMachine backing up the folder(s) where the databases are?

: I am confused about where your database files (actually “packages”) are located. Could you please give the full path names of each of your databases please (including file name and file extensions) DEVONthink menu: File/Database Properties …, top of the info window.

: I’m curious why you have encrypted the DEVONthink databases? That of course is your choice and depends on the sensitivity of the information. I get that. But to protect against what? Is you computer not sufficiently secure again someone else getting physical access to the machine with your ID? Are you using Apple’s FileVault on your computer(s) encrypt the data on your disk? If so, then is double encryption worth it to you? Just asking.

5

The Global Inbox is normally in the application support folder. Other databases are where you put them.

It is vital that the local databases are not themselves in a cloud folder. I assume not as their in the user home folder.

I’ve not used iCloud as a sync store as it is slow and unreliable for me generally. I routinely used Dropbox until mid-2020 when it started to cause sync problems with my databases.

When using Dropbox, I used the DTP encryption option. This worked fine for many years until the problems mentioned above.

Using encrypted volumes seems like an unnecessary complication and perhaps that is the source of the problems?

Could you not rely on the DTP encryption alone? Or have I misunderstood the set up?

6

I’m not having the issues reported. It’s @think. I don’t understand what is being said as symptoms.

I suspect the files are indeed missing, that they are “indexed” not “imported”, that sync is a red-herring (but the setup bad anyway), there are issues with backup and encryption, but might be red-herrings to the “corruption” problem–but certainly seems redundant.

8

Cutting to the chase:

: Find the missing files if you want them back into DEVONthink. They might still be in DEVONthink if not indexed, or if indexed and residing elsewhere, then they need to be found if you want them.

: read and understand section in manual “Repairing a Defective Database” esp the section on “Missing Files”. Page 178.

: once found, either “import”, or “index” if the source of the files is reliable and the file location won’t change.

: figure out why TimeMachine not backing up everything. Other backup problems lurking?

: Only sync if you need to. Don’t consider sync as a backup. It’s just not. If you must sync to a Cloud server, then pick one unless there is an overarching reason not to do that.

: If your devices for sync are all in the same local network, use Bonjour.

: only encrypt DEVONthink databases if you you really need to.

9

Apologies; hit the wrong reply button.