DEVONthink always reports indexed files moved in Finder or Obsidian as missing

[cgrunenberg]

@codegeek

DEVONthink detected that they were missing, but “Update Indexed Items” didn’t resolve the problem.

Did you just index the files on their own or their enclosing folders? In which folder in the filesystem were the items actually located?

[codegeek]

I see above that your files are in Apple’s iCloud. My hunch is that if DEVONthink app thinks the files are “missing” it might be because iCloud deleted them on your local machine to “help” you with reducing local disk space.

Thanks, but this is not the issue. These are files I actually deleted from my vault because they were unneeded (mostly duplicates created by a flaky plugin a few months ago). So, it’s not just how iCloud is managing files; these actually should be gone.

Also, do you really need the Obsidian files indexed into DEVONthink? Doesn’t Obsidian do a sufficient job of letting you find and manage the Obsidian files? Just wondering.

No, in the sense that I could survive without the indexing. And of course it’s easy to surface info from within Obsidian. But I want to leverage DEVONthink’s ability to index files to bring together my imported files and other information that doesn’t “live” in DEVONthink, like all of the thinking, connecting ideas, and brainstorming I do in Obsidian.

It all works extremely well together, except for this nuisance of the index not corresponding to what’s actually on my drive by “remembering” deleted files.

[rmschne]

thanks for the clarification.

[codegeek]

I indexed the actual folder for my Obsidian vault, which lives in iCloud. That is, /Users/my_user_name/Library/Mobile Documents/iCloud~md~obsidian/Documents/my_vault_folder.

I have the same problem with other indexed folders not in a cloud storage, though. For example, /Users/my_user_name/Documents/some_folder_I_want_to_index_in_DEVONthink. If I use the finder or some other app to delete a file there, DEVONthink will report that file as missing. This happens less often because those folders are pretty static compared to my Obsidian vault, which changes constantly. But it’s still a hassle to find out that Paper I Wrote 10 Years Ago copy NOT FINAL.docx is “missing” because I did some minor housekeeping.

[cgrunenberg]

@codegeek

For example, /Users/my_user_name/Documents/some_folder_I_want_to_index_in_DEVONthink.

And the Documents folder isn’t also synchronized via iCloud Drive? Does selecting the indexed group containing the missing items in the item list (e.g. list view) and then using File > Update Indexed Items fix the issue?

[codegeek]

And the Documents folder isn’t also synchronized via iCloud Drive?

It is not.

Does selecting the indexed group containing the missing items in the item list (e.g. list view) and then using File > Update Indexed Items fix the issue?

No. DEVONthink will report any deleted items as missing again the next time it tries to verify the database.

[cgrunenberg]

@codegeek

DEVONthink will report any deleted items as missing again the next time it tries to verify the database.

Even after trying to update the indexed group as suggested? Are the indexed folders again located on iCloud or local folders?

[BLUEFROG]

There are more variables at play here than you’re reporting.
Obsidian has three options on how to handle file deletions.

It’s unclear whether this is Obsidian not registering a filesystem event or DEVONthink not receiving one.

And yes @cgrunenberg, this is from an indexed Obsidian vault in iCloud Drive.

[codegeek]

Even after trying to update the indexed group as suggested?

Yes, unless I either (1) manually delete the missing files from DEVONthink or (2) remove the entire folder from the index and re-index it.

Are the indexed folders again located on iCloud or local folders?

I have both kinds, but both have the same issue. For clarity and full disclosure: I have some indexed folders in DEVONthink that are located in local folders, one (the Obsidian vault) in iCloud, and a couple in Dropbox. All of them display identical behavior, so far as I can tell.

[codegeek]

There are more variables at play here than you’re reporting.

My apologies. I’m not trying to obscure anything or be difficult, so I apologize if that’s how anything above came across.

Obsidian has three options on how to handle file deletions.

Mine is set to move files to system trash.

I just looked and do see the some of the files I deleted recently in the system trash. (Some were deleted by a shell script using rm and so are already gone forever, as expected.)

To try to figure this out and be as precise as possible, I just ran the following test:

1. Created three files in Obsidian (again, all in my main vault, which is in iCloud), each containing ~1 kB of dummy text:

• File to Delete from Finder.md
• File to Delete from within Obsidian.md
• File to Delete via rm.md

2. In DEVONthink, selected the group corresponding to my Obsidian vault, then File > Update Indexed Items.

3. Confirmed that DEVONthink has now indexed all three files.

4. Confirmed that all three files have synced with iCloud but are also still stored locally.

5. Deleted the new files, each via the indicated method (one from Finder, one from within Obsidian itself, and one via rm in the terminal). ← This is the point where I would expect DEVONthink to remove the files from the index automatically, just like they would no longer appear in Spotlight, Finder, etc.

6. Checked the status of all three files:

   • File to Delete from Finder.md appears in system trash, does not appear in the .trash folder in my Obsidian vault, shows as “missing” in DEVONthink.
   • File to Delete from within Obsidian.md also appears in system trash, also does not appear in the .trash folder in my Obsidian vault, also shows as “missing” in DEVONthink.
   • File to Delete via rm.md does not appear in system trash or in the .trash folder in my Obsidian vault and shows as “missing” in DEVONthink.

   All three files looked like this in DEVONthink at this point (with different file names, obviously):
   

   

7. In DEVONthink, selected the group corresponding to my Obsidian vault, then File > Verify & Repair Database, which reported 3 missing files:
   

8. Clicked “Repair,” which failed:
   
   

9. Clicked File > Update Indexed Items, which succeeded this time:
   

10. For completeness, I did a similar test by creating a file in a local folder that is indexed in DEVONthink but is not stored in any cloud (~/Documents/Some Folder/temp.txt). When I deleted it in Finder, it also showed as “missing” in DEVONthink. After File > Update Indexed Items, it did not, and subsequently clicking on File > Verify & Repair Database led to a successful verification.

So, on the upside, Update Indexed Items worked this time, though it has definitely failed several times in just the past few days after similar deletions. I’m still surprised that DEVONthink requires that, though, especially for the local-only indexed folders. Is this expected behavior?

[cgrunenberg]

@codegeek

So, on the upside, Update Indexed Items worked this time, though it has definitely failed several times in just the past few days after similar deletions. I’m still surprised that DEVONthink requires that, though, especially for the local-only indexed folders. Is this expected behavior?

Filesystem events in cloud folders are limited and files are not automatically removed from the indexed group and manually updating is indeed necessary.

The reason is that e.g. the item could have been moved into the database on a different Mac and the synchronization of the cloud folder & the filesystem event happen before the synchronization of DEVONthink’s database.

Unfortunately it’s impossible to ensure that filesystems events, DEVONthink’s synchronization and the cloud synchronization happen in the order required to handle deletions in indexed cloud folders automatically (and multiple devices and platforms like iPhone & iPad and multiple sync locations don’t simplify this).

[BLUEFROG]

My apologies. I’m not trying to obscure anything or be difficult, so I apologize if that’s how anything above came across.

No worries and you didn’t come across in any bad light. I was just stating there is more at play than simply, “I deleted a file in Obsidian…” as it has multiple options for deletion.

[codegeek]

Thank you for this explanation.

Because I do a lot of work in these indexed folders outside of DEVONthink, this kind of thing happens daily. I’m thinking of mitigating it by running Update Indexed Items automatically from time to time. In particular, I’d probably use Keyboard Maestro to run it a few minutes after my laptop boots or wakes, and I’d also assign a global shortcut key.

Do you see any cause for concern in that plan?

Thanks!

[codegeek]

Thanks, @BLUEFROG! I really appreciate all your help here.

[BLUEFROG]

You’re welcome
Do you need to have the Obsidian vault in iCloud, e.g., you’re using their mobile app as well?

[codegeek]

Do you need to have the Obsidian vault in iCloud, e.g., you’re using their mobile app as well?

I do, yes. I use the mobile app heavily, and I moved to iCloud after finding that Obsidian was a bit iffy on Dropbox, at least for me. It’s been a rock-solid setup except for this (the indexing in DEVONthink) piece of it.

[BLUEFROG]

I am curious how an alias would work in the situation, e.g., move the vault into its own folder in the root of your home directory, like a Vaults folder. Then alias that to the iCloud Drive folder.

Just a thought.

[Phileosophos]

Thanks for the discussion of this topic, y’all! I just started using Obsidian and was thinking of ways I might leverage it with DT3 and some other tools. This will be helpful.

[Harold]

I have read through this thread with some puzzlement. I have my Obsidian vault (in iCloud) indexed in DT3. Everything I do to notes in Obsidian is reflected immediately in DT3. I guess I don’t try to perform the tasks that create problems for others, although I don’t understand why I would want to edit, move, add or remove links, etc to my Obsidian notes outside of Obsidian. To me the two programs are complementary. I see thematic connections to non-Obsidian items when I look at Obsidian notes in DT3. But I don’t see Obsidian as a place to store documents or manage records as I do in DT; it’s just my place for creative brainstorming, the place where can easily link up fragments and scraps I write that may eventually develop into something more focused or formal.

[Maria]

This thread is fantastic. I also use Obsidian for some time now – 2 years with interruptions testing other software; now, I have settled with Obsidian for half a year. I use it for planning, gtd, and note-taking with Obsidian. It is beautiful, flexible, has an excellent editor, and serves some additional purposes like gtd.

I also used to index my Obsidian vault in DTP for precisely the same reasons mentioned by several in this thread. Still, I also had missing files and errors in sync – Obsidian synched via iCloud and DTP via iCloudKit. Maybe the problem is that I sometimes like to work in the Obsidian vault from Devonthink.

Anyway, I ended up with a strict separation of both; my vault is not indexed anymore. When a particular area is finished, I import that into Devonthink for storage and retrieval.

But this makes Devonthink increasingly an archive rather than a working place.

I accepted that with quite some regret.

My PDF library and PDF reading process, e.g., has fully migrated to Zotero, and note-taking is post-processed in Obsidian. Only the results are archived in Devonthink.

Instead of ending up as an archive (valuable though, since it also stores documents and helpful information), I would like Devonthink to take the next steps and (a) either get the flexible and comfortable aspects of Obsidian introduced to Devonthink or (b) find a way to integrate well with the software. – I am not informed but only have the impression that Obsidian is a clear winner in the note-taking and note-management world and can be taken as a serious partner.

This would return Devonthink’s use cases to everyday work.

So far, some ideas early in the morning; love Devonthink!

Maria