My largest database crashed recently and I tried to repair it. All of the documents in that database have now been disorganized from their folders, my ‘favorites’ show up as greyed out and unlinked in the sidebar.
I am not sure where to turn next. Is there any way to resolve this issue?
do you have a backup?
I do, but it is .7 GB smaller. Is there a reason that the repair process would have added any significant space to the documents? There is no way I have added that much data in the later half of this month.
And if I do restore, I am wondering if I should prepare my two iOS devices in any way (maybe by turning off iCloud sync?) … they are notoriously flaky with sync.
What exactly did crash?
The size of complete .dtBase2 packages are not really comparable and might change, e.g. depending on internal backups. How old is the backup?
Disabling the synchronization on all computers/devices until this is ironed out is definitely a good idea.
Do you know whether the database was already uploaded from the Mac to iCloud after repairing the database? Otherwise you could try to import the latest database from the sync store by first archiving the existing database, e.g. zip or copy it in the Finder, then deleting the database in DEVONthink and finally reimporting it via Preferences >Sync.
Or is the database on the iOS devices still fine and complete, e.g. doesn’t contain pending items?
This is a clear cut use case where a DB integrity check we discussed last week could be of use. As a user you’re never sure whether the dtBase2 package is similar to a historic version of that package on the primary drive, if the calculated size of the backup is different from the original.
One could argue whether it’s most logical that DT supports an integrity check, or the backup software itself. I just found out CCC actually does this with an md5 hash:
(see “Find and replace corrupted files”)
It appears Time Machine has a similar function, although it’s somewhat more difficult to perform I guess.
This is helpful thanks. To answer questions:
By crash, I mean to say that I opened the Mac version and all of my databases were closed. When I opened my largest one, the application closed itself and then I got the notification to rebuild when I reopened.
The backup is from 7/22, before the crash.
I have all three devices open now to see what is in sync. Seems like Mac did not sync back to iCloud now. The iPad and iPhone are fairly close in sync data. I temporarily turned off iCloud sync on the Mac.
iPad has 4 orphaned files inside a folder in the inbox, as does the iPhone. The iPad has a triangle on the cloud icon which say this when clicked:
“The file ‘master.plist’ couldn’t be opened/ NSCocoaErrorDomain 256. 7/12/20”
Do you still recommend I turn off iCloud on all devices and archive the most current Mac database? If so, which menu is that under?
Thanks so much for your help!
Ok, some things have changed since I last replied. Everything is 100 percent in sync across iOS. and the Mac on my Finances and Work databases. My giant Personal database is in sync between iPad and iPhone but the Mac version is the one that is out of sync.
In the iCloud Sync settings on Mac, I see two options under ‘Remote,’ both of which say Personal and one of them has the iOS icon next to it and another has the Mac icon. Right clicking allows me to Import, Import Encrypted, Verify, or Clean the database, but not Sync.
If I want to get the iOS database on my Mac from the cloud, and NOT wait weeks for it to resent everything when the files are already referenced on my Mac (just unorganized in the DT database), what can I do? And if I can’t do that, what is my next best option?
Longer term, I think I would like to try syncing this larger database in Dropbox. iCloud just never seems to have been up to this task.
A screenshot of Preferences > Sync would be great. However, after importing a database from a sync location no data has to be uploaded afterwards.
@cgrunenberg such “stories” make me a bit nervous.
The latest releases automatically verify the database(s) each time they were not closed properly (e.g. due to a crash). And of course you should never put your databases in cloud folders or folders synchronized or even encrypted or compressed by third-party apps. DEVONthink blocks this already in case of common folders like Dropbox or iCloud Drive.
It does already if such a major issue is detected until the database is either restored, repaired or rebuild.
Personally I would never rely only on automated backups or only on local backups, meaning that a good strategy includes local backups, remote/cloud backups and ideally manually archived & valid backups too as backup software can of course fail or might back up corrupt data, hardware can fail and cloud services might be unavailable or have their own issues.
And of course the more backups the better.
At least the database shouldn’t perform any automated tasks (e.g. smart rules, reminders, scripts or RSS refresh) during a backup, otherwise closing is definitely more reliable to ensure that all automated backups are valid.
Can you explain what you mean by ‘no data has to be uploaded afterwards’? My best guess is that the cloud copy of my Mac’s personal database has all of the same files, but they are organized into folders. It’s the local copy that is just all dispersed in a folder called ‘Orphaned Files.’
The database is about 200 GB. If I back up and then delete the Mac copy, and then import the cloud version of it, will I have to wait for all of those files to redownload back to my Mac? If it will work, I will do it, but it seems like a major inconvenience.
Here is a screenshot of my settings.
You were afraid that you would have to “wait weeks for it to resent everything when the files are already referenced on my Mac”, this is not the case after importing a database from a sync location as there’s no need to upload anything afterwards.
Did you create databases having the same name on multiple devices/computers? The name doesn’t matter for the synchronization, only the internal ID.
I didn’t create multiple databases with the same name on multiple computers, at least not intentionally… I created the Personal database on Mac and then synced to iOS. I have no idea why both are depicted as different versions as they used to be in sync with one another, although they would sync very slowly.
If I import from the cloud, I know that it wont have to upload, but it will have to download, right? That is the part I am worried about taking a long time.
Yes, the complete database would be downloaded. How large is the current database according to File > Database Properties?
It’s over 200 GB.
And how fast is your Internet connection?
Download speed is about 100 mbps. I am going for it. I actually already tried to download the Personal database with the Mac icon and it immediately appeared inside of DEVONthink with no data inside of it. Not sure how it got into the cloud, but I am trying the version that had the iOS icon now and it appears to be the correct size. In 15-20 minutes it has downloaded 1/12th of the data… not a bad sign.
Some weird things happened along the way, though. I was hesitant to delete the empty Personal database I attempted to download first, so I renamed it Personal - Mac Icon just to tell it apart. When I tried to import the correct version of the database, it prompted me to pick a local place to store it, as expected. When I attempted to name it Personal in the Finder, there was already a database in there called Personal in addition to the one I had just renamed Personal - Mac Icon.
I am not sure how this can be because I had already archived and relocated the local one with all the orphaned files. I decided to save the iCloud import as DEVONthink for now, and it seems to be going ok. I hope I made the right choices.
Please let me know if you think I have made any missteps and I will hopefully be able to report success on the iCloud import later today.
iCloud has been so slow with sync that I am wondering what it might look like to try syncing this database over Dropbox instead. Do you have an opinion on that? Perhaps asking the correct way to do that is subject for another thread.
As long as you archived the original database as suggested everything should be fine. By the way, which version of DEVONthink and of macOS do you use?
Dropbox isn’t very fast either but at least doesn’t depend on an asynchronous process controlled by the OS & Apple’s servers. However, one of the next releases might include a better option than iCloud or Dropbox.
People, we’ve got cliffhanger here… 
I can’t wait until the first DTDC (the DEVONthink developer conference) which probably starts off with presentations in the Bill DeVille Theater at Devon Park in Devon by @eboehnisch and @BLUEFROG.
As long as you archived the original database as suggested everything should be fine. By the way, which version of DEVONthink and of macOS do you use?
macOS 10.15.5
DT 3.5.2
However, one of the next releases might include a better option than iCloud or Dropbox.
Very interesting! In the meantime, sounds like Dropbox is worth a try.
So now things are getting weird… The databases is appearing in DEVONthink, with correct folder structure, but a lot of the folders are greyed out and many of them have little icons to the right that look like the Finder icon. Most folders are missing the documents in them and some are half filled. Top level organizing folders like Safari, Email, and Evernote from previous imports are missing.
The downloading from iCloud progress bar in the lower left corner of the app will disappear, making me think its done, but then clicking through some of the folders to look for documents, will trigger it to start again. Is it running this whole time? Is there a way to monitor it more consistently and watch documents as they download?
One final question… I had an archive of my macOS Documents and Desktop folder in this database previously. The top level folders Desktop and Documents are now missing and all of those folders are appearing as mixed in with the folders that were actually living in the database at the root level. What data from that archive would have been kept in a cloud copy of the database. I had assumed those archives were local to the device. If possible, may it would be easier for me to disable them and restart them from scratch once the documents unique to DEVONthink are downloaded?
Thanks again.
