Running DEVONthink 3 (3.7.2) on macOS Big Sur (11.4), I continue to be troubled by what seem to be erratic behaviors of DT3 and indexed files and folders: the indexed content is seldom reliably updated within DT3 when changes are made to indexed files in the Finder. Here’s a typical scenario:
My “Projects” DT3 database includes both folders (groups) and files stored only in DT3, and folders and files in external Finder directories, indexed into the “Projects” database. The indexed files in external Finder directories are in one big, top level directory that is synced with Dropbox. I don’t use Drobox to share the files with any other computers; I just use it as a potential route for accessing files – which I rarely do – in the event I should need to get to them from another computer, say, on a business trip.
Here’s where the trouble begins: I move some files around within the indexed external directories, rename them or – this seems most often to be the source of the problem (?) – consolidate some of the files into a subfolder and zip that into a single file for long-term archiving. (I do this for manuscript drafts, invoices, and the like, which are no longer needed and which I want to remove from Spotlight and DT3 searches, but which I don’t want to delete.) When I am making these changes, the “Projects” database is open. My assumption has been that such changes made in the Finder should be reflected nearly immediately in DT3 (?)
But: if at some later time I should “Verify and Repair” the DT3 database as part of my bi-weekly maintenance, I almost always get a warning that previously deleted files – that is, deleted in the Finder – are “Missing File(s)” – in other words, that DT3 didn’t pick up on their deletion. “Update Indexed Items” does NOT correct this problem. Closing and reopening the database doesn’t help. “Ignoring” the errors flagged by “Verify and Repair” doesn’t make the errors go away on subsequent verification. In other words, references to the so-called missing files persist in DT3, apparently no matter what I seem to do. Perhaps I should stress this point: it’s not the DT3 doesn’t immediately update on changes made in the Finder; it’s that hours or days later, DT3 still hasn’t updated on those changes, so I don’t see this as a Spotlight latency problem.
I have run into subtle variants of this problem in multiple DT3 databases that contain indexed files, from multiple indexed folders. Am I misunderstandiing how indexing should work? There are good reasons why I might sometimes work on indexed files in DT3 and sometimes in the Finder, but it I can’t get changes in the latter to reliably “take” in DT3, then the usefulness of indexing appears doubtful.
I have that problem as well… but it is not a real DT problem. It seems Finder does not communicate all changes in its file system watcher (or whatever it is called in macOS).
A very easy workaround si to go the parent folder inside DT and select “File → Update Indexed Files”. All is corrected inside original DT instance.
And there is another “issue” that really it is not. Inside DT, you have a “DT owner” of that file. Let’s assume two Mac with two DT installations, Mac1/DT1 and Mac2/DT2. If you add an indexed file in DT1, it is “owned” by DT1. If you remove/move it in DT1 all go right, but if you delete in Mac2 or DT2, the file remains “pending” until it is manually deleted from DT2 (or from DT1). This saves some complex situations and mistakes when, for example, in Mac2 your Dropbox installation takes some minutes to update your added file in DT1 (and I can confirm that so and then, Dropbox “forgets” to add some files added in other places until you manually go to that folder in web version (*) ).
Sometimes, the combination of both things deals in “strange” behaviour that it is really easily explained taking in consideration this two things.
(*) Yes, never happened to you, reader, you are lucky or do not have 3 million of files in your Dropbox.
Thanks, Bluefrong and rfog, for your replies. Yes, I have read the documentation Bluefrog flagged and I’m aware that indexing cloud-synced folders may require additional housekeeping. The problem I’m running into is that “Update Indexed Items” doesn’t correct the inconsistencies or end the error reports. Clicking on the “Repair” button after I’m notified that the files are missing (following a “Verify and Repair Database” command) doesn’t help; repairing fails every time. There seems to be no way to remove the phantom files and folders from DT3 once the application has determined that they should be there. When they’re aren’t there, not in the Finder and not displayed in DT3.
Have you emptied the database’s Trash? The Trash is still a location in a database. If you have indexed files in the Trash, that can certainly cause seemingly odd behavior.
However, I would encourage you to brush up on the section about deleting indexed files so you don’t end up with another surprise in the Finder when you empty the Trash.
Yes, the database’s Trash has been (had been) emptied. But your query got me to thinking about Trashes. So I tried emptying the macOS Trash, and… “Update Indexed Items” and “Verify and Repair” worked thereafter exactly as expected. It is my usual method to have the macOS Trash remove items from the Trash after 30 days; I have that preference set in Finder. Am I not going to be able to do that if I also sometimes delete files in the Finder that are indexed in DT3 and also synced in the cloud? Or was this more likely the product of some unusual glitch in Finder-DT3 communication resulting in DT3 not being alerted to the file deletions (i.e., were moved to the macOS Trash)?
Since the files are in a Dropbox folder, it’s possible there was no fsevent reported with the move to the Trash in the Finder. In fact, IIRC Dropbox has its own trash so it’s possible there was no event due to that. I’d have to look a bit deeper at a machine I have Dropbox installed on.
Also, are you using Dropbox’s Smart Sync feature where files are stored online but downloaded on demand or keeping the files stored locally?
That’s interesting to know for future reference, but I don’t think it applies in the situation I’ve been describing, as the phantom DT3 file references were definitely older than three days. I deleted those files at least two weeks ago…
