My main iMac has been having iCloud sync issues for a few weeks. Nothing huge. In fact all small junk. Like it gets stuck downloading “zero bytes of 46kb” for days and days. So after a call to apple, 5 or 10 reboots, lots of experimenting, etc., I turned off iCloud Drive on that machine.
Now, however, I go into DT3 dbs and huge chunks of data are missing from some dbs. I open the same db on my laptop and it’s all INDEXED as it should be, but everything on the laptop says “file missing.:” I sync using CloudKit.
I am hesitant to rebuild cuz I don’t know what it’s actually going to rebuild. Rebuilding from an equally empty or damaged db won’t help me. I tried to “restore” and that did nothing.
There’s nothing listed to “select.” The group where the files or references are supposed to be is empty.
So to recap.
2 machines. iMac and MBP
The iMac is the one that’s been having iCloud issues, and on which I turned off iCloud Drive a couple of weeks ago due to ongoing sync issues. When I went back in to that iMac yesterday, and opened up DT3 AFTER having reconnected iCloud Drive, entire groups of a fairly large DB are just empty.
When I go over to the MBP to see if the same problem exists there, it does not. It’s a different problem. The db on THAT machine shows document references in the groups that are empty in the iMac, but they all say “file missing.”
The files are not missing. The iMac db referenced to a Dropbox folder. The dropbox folder is still there. Still populated.
I could just reindex the empty group, but that would just be a situational workaround. I don’t know what else is screwed up in what was about a 1.5 gb database with thousands of files in it.
Turning off iCloud Drive will have affected DEVONthink’s ability to sync; seeing as the iMac has its own local copy of the databases though, this should not lead to any loss of data. Reconnecting iCloud Drive should reenable sync, but should not lead to any data loss. Data loss could in theory occur when disconnection iCloud Drive if you were indexing from there, and started up DT at a point in time when iCloud Drive was not yet providing a local image of the files indexed. However, you state you are indexing files in Dropbox.
I think the following has happened:
on the iMac, the files which were being referenced from Dropbox are no longer available in the local location they previously were, and have therefore been removed from the database.
that change has not synced to the MBP (yet; presumably because iCloud Drive wasn’t available)
There can be a couple of reasons for the files no longer being available locally: One is a change to how Dropbox, OneDrive etc. work - see this thread. What is surprising to me, however, is the point in time at which this has happened - the pertinent updates to Dropbox have been around for a while. So I may be barking up the wrong tree - it would be worth taking a look at the thread though, and seeing whether you can relate anything to your situation.
Other reasons: when you turned off iCloud Drive, did you possibly opt to keep files on your Mac, possibly decreasing the amount of space available and triggering Dropbox to remove files from the local store? Is Dropbox set to keep the indexed files available locally at all times?
There is no “missing file.” There is a group that is empty. The computer with the “missing files” is a completely different machine. Do you want me to go to THAT machine and see what the path says?
That db is only accessing the sync db’s, so it would seem logical that if the iMac doesn’t have anything in the empty group, then the group on the MBP would also be empty. Or not. I dunno.
The empty groups in the screenshot are MCLE and Pets. They index Dropbox folders with a lot of material in them.
That is possibly where it looks as if it is, not where it is. It is possible (as per the thread I linked above) that the files actually reside in ~/Library/CloudStorage/Dropbox and that what you are seeing is only an alias. That is one of the reasons Jim has been asking you for the path of one of the missing files (and yes, obviously, one of the files marked missing rather that one of the ones that isn’t there at all; we are assuming that the problems on the two Macs are related - I explained a possible mechanism).
You didn’t respond to to my question:
This is an absolutely essential step when indexing (see Dropbox help, it’s called something like “make available offline”).
Whilst I understand that that is the one thing you did, there have been numerous under the hood changes made to cloud storage by the respective purveyors in recent times; you would not be the first person to report trouble after one of those changes.
So you’re right; the “missing” files on the laptop seem to SUPPOSED TO BE in ~users/[my name]/Library/Mobile Documents/ com~apple~CloudDocs/000 Administrative and Personal/… Drill down to subfolder names on Dropbox.
That is the name of the Dropbox folder from which this db was created.
Looking in the library, there is no folder called “com.***” anything in the Mobile Documents folder.
Re whether dropbox is set to keep files local at all times, some yes, some no.
At this point in time, unless you have some better idea that can be implemented with some efficiency, I’m thinking I should probably just recreate the db, and do it in a way that DT3 and iCloud don’t even communicate with each other. If I do that I will start over and skip the sync process entirely as it doesn’t really seem to be ready for primetime anyhow.
And so if that folder doesn’t exist, where are the documents? Most in this folder were indexed or obtained from Dropbox on the iMac. Some were scanned directly into DT3 from Scansoft, my Fuji scanning software after I started relying more heavily on DT3 instead of storing things on Dropbox. I don’t think I’ll make that mistake again.
The documents are in the folder which Dropbox has set up, I assume in ~/Library/CloudStorage/Dropbox (I haven’t got Dropbox installed, so cannot test). As I have pointed out, changes to macOS made it necessary for cloud providers to change their software and the associated paths.
That is a receipe for disaster when indexing; any indexed file must be available locally at all times (otherwise the following can happen: Mac A has indexed File X; File X is removed from local storage by Dropbox Software (DBS); DT on Mac A assumes File X has been deleted by the user and removes all reference to File X; DT Mac A syncs to DT Mac B, on which DBS has not yet finished syncing; DT Mac B is told that File X has been deleted; DT Mac B then deletes File X from its local location (so from the Dropbox folder). DBS notes the deletion and thus also deletes File X. What started with the file being offloaded has ended with it being deleted.
Something similar has happened to you, I guess: the local location for your files was moved; DT cannot be aware of that move, because it shifts the files out of an indexed location - it can only assume they were deleted. As such, they are missing from DT. What I cannot reliably explain is the differing behaviours of your two Macs; this could be different versions of the Dropbox Software on each, a failure of DT to sync between the two Macs (my current theory) or another factor which I haven’t thought of or can’t determine.
You weren’t asking me, and obviously waiting for Jim to respond is a good idea - he’s support and I’m just a random guy from the internet. For what it’s worth, though, I would recreate the DB, yes. I would first make sure that macOS and DBS are both up-to-date on both devices, and that DBS was using the same storage location (path) on both devices.
I’m not sure the blame game is terribly useful here. I first suggested in January that I thought there might be trouble ahead with the changes cloud providers were making. I’m sorry to hear that those changes seem to have caught up with you. There have been a small number of reports in the forum of ppl caught up with those changes, but it didn’t affect everybody (and I’m unsure why; DT spoke of some mitigations built in, but I have too little knowledge to understand why things didn’t work out in your case).
I understand. When I say some yes, some no, that is across 3.2 TB of Dropbox folders and files. A ton of material is archived in dropbox and does not need to be accessed, so it does NOT have a corresponding DB in DT3.
Anything indexed in DT3 is available locally at all times. I DO understand that is necessary.
Dropbox is just a folder on the HDD in ~user>[name]>Dropbox. That’s the default that Dropbox sets up. It is unaffected.
~/Library/CloudStorage/Dropbox doesn’t exist.
As for the “blame game,” that’s not really what’s up. I’m just very tired of screwing around with databases that can’t be moved, copied, cloned, backed-up, stored in the cloud, etc. It all seems to be a lot more work and risk than seems worth it. I’m thinking if this can’t be fixed with some efficiency, I’m just going to go back to my old system of storing things in folders. Things don’t disappear when I do that, and a robust search app like Houdahspot finds things pretty fast.
Thanks for that feedback which suggests different behaviours (of different versions?) in the wild. It still suggests that your files moved sometime along the way (as the pointer was to ~/Library/Mobile Documents/…)
Certainly best to do what you are happy with; it frustrates me all the same, seeing as DT just works smoothly for me. I’m sorry I’ve not been able to help you.
Thanks for your efforts. And the Dropbox folder isn’t moved, but replicated somehow. As for documents that are scanned directly into DT I have zero clue where those images go.
