I ran Very and Repair on a database and it reported 3 missing files. I looked in the log to see what they were and established that the files were present in the correct place ( don’t understand why they were missing, but I opened them in their native app and they were fine the missing ones seemed to be dupes).
I read a post here that said rebuilding the database would clear the missing files, so I did a rebuild. After which there were NO files or folders in the Database in DT. The size of the database in ~/Databases was still 6 GB, so I figured the data had not gone.
I then tried Repair and Verify and ended up with 4000 orphaned files.
Then I restored the database from Time Machine from an hour previous, and am back up and running…phew!
As a new user of DT3 this has really scared me. What might I have done wrong?
This should definitely not happen and so far it’s the first report of its kind. Where is the database located, was there enough disk space available? Or is it an encrypted database?
Finally, rebuilding can’t repair missing files (only if there are also orphaned files having the same filename). But missing files are logged to Windows > Log and the contextual menu can be used to either trash or reveal these items.
Databases are in ~/Databases.
I have 70 GB free on the internal drive. I have six databases totalling 15GB. The affected one is 6GB.
Databases are not encrypted.
I understand that Rebuilding doesn’t fix missing files, but having established that they (or copies of them) were actually present, I went ahead to rebuild so that they would stop appearing as missing. I assume that rebuilding is like reindexing so if they are missing they won’t appear after a re-index.
Searching here for “Rebuild” does find some similar events eg this one, although there are differences.
At least this issue was already fixed in version 2. Are you able to reproduce the data loss? Of course back up your database first, e.g. try to rebuild a copy created in the Finder.
Yes. It is backed up to Time Machine and I have already successfully restored so all is good again, except that I would like to understand why it happened. I have also exported an archive.
I will have a break and then create a duplicate of the database in Finder, rename it ‘Test’ and try rebuilding it, as Christian suggests.
Does this sound sensible? If it works without problem, it won’t tell me why it went wrong before, but it will restore some confidence.
A copy of your logs would be great. After rebuilding the copied database please choose Help > Report Bug while pressing the Alt modifier key and send the result to cgrunenberg - at - devon-technologies.com - thanks!
Also, though I have never personally seen the issue you reported (but have had a handful of tickets over the years), my first reaction would be to hold the Option key and choose File > Restore Backup and roll back the metadata.
Hi I have done as you suggested and emailed a report plus logs after the Test Rebuild. I sent it to supplied address and yours. It has been acknowledged as Bug Report DEVONthink Pro 3.0.3#112867
Unfortunately I misspelled your name and it bounced so I have resent it with correct spelling.
When you say that your first reaction would be to “hold the Option key and choose File > Restore Backup and roll back the metadata”, do you mean that is what I should have done when I first hit the problem? …or if it happens again?..or now that I have restored from Time Machine?
I have done the test rebuild which did not repeat the problem, although the log showed a large number of other “no index”, “no text” and “unknown format” errors. These only apply to PDFs, which all appear to be fine when examined. It is all in BR #112867 which I hope you can see.
Jim, I realised that I still have the corrupt database. I saved it before restoring from Time Machine. I just opened it and it was still in that state of no folders, groups and 4000 orphaned.
I did as you suggest and opened it holding the option key and Restore Backup… There were three options and I chose the earliest which had 221MB and rejected the two with 4K.
Duplicating a Database and renaming it for test purposes may not be such a great idea. I renamed the copy of my MIKE database to TEST and opened it. The log reports that it has the same UUID as MIKE and the name in the list of open databases was MIKE. I renamed it there (via Database Properties) to TEST to try and keep it separate from the proper MIKE.
Then TEST started replacing MIKE on my other sync’d devices and a lot of uploading and downloading went on. I think have now deleted TEST and got everything back to MIKE on all devices. I should probably have disabled syncing before starting on the TEST experiment.
I am rather inclined to quit while I am ahead and stop investigating/experimenting.
I did as you suggest and opened it holding the option key and Restore Backup… There were three options and I chose the earliest which had 221MB and rejected the two with 4K.
Excellent!
Yeah, you don’t want to open the copied database while the other is open. And yes, you should have disabled syncing.
Perhaps we should disable sync when a duplicate database UUID is detected (though this feels like a discussion I’ve had ). @cgrunenberg would have to comment.
Can I make a suggestion completely off-topic with regard to the database situation? It does have to do with backups though.
@DrJJWMac mentioned “old school” CD/DVD’s as a backup medium a while ago. That made me reconsider them as an extra safety net to versioned on/off-site backups like Time Machine.
The most important reason to restart using them as an additional medium is the risk of falling prey to some kind of ransomware that seem to become more and more prevalent. Although optical media have it’s obvious downside they are the only media that are read only AFAIK.
To be clear, a good backup schemes should be able to withstand ransomware, but some apparently perform very nasty actions that are hard to protect yourself against. Hence the optical media.
I would not rely on them solely though, store them in the dark and test their content at least once yearly. Of course, there will always be some data loss, as you will probably not backup 15GB to DVDs or Blueray every hour. Whether you should use them is up to you of course.
Yep. But apparently they do still exist. The nice thing is that most drives have become quite cheap. Apple is actually still selling the Superdrive.
With ransomware issues appearing more frequently these days, I wouldn’t be surprised if there will be some kind of resurrection of these type of media. Or hard disks with a physical read only switch, but that requires human intervention.
People might also see the investment in a new drive as an investment in their bedroom, as it has the potential to improve one’s sleep
Thanks. Ransomware is indeed another good reason for offsite backups (along with fire and theft etc), since Ransomware can attack any attached backup, like Time Machine. I also backup exported DEVONthink Database archives to CrashPlan, which I find easier to keep up to date than CD/DVD/BDs. I do have a lot of photos and video which wouldn’t need updating as often, but with 4.3TB on CrashPlan, I would need more than few DVDs! Even BDs (which went up to 50Gb last time I used them) wouldn’t really cut it.
). 
