Updating Indexed Folder

[roged]

I have been tested the functionality of indexed folders, but the indexed folder is not updating if I add files outside of Devonthink. Update to the folder only works by using File-Update Indexed Items.

Details:
New Test database.
Local folder in Documents (not iCloud location) added as indexed.
Folder icon shows as indexed.
Trash is empty.
Creating new file in Devonthink folder creates file in Finder location.
Devonthink is not updating when moving an existing file (or creating a new file) into indexed folder.
Updating an existing file externally does update in Devonthink.

Is this the expected behavior when adding new content to an indexed folder outside of Devonthink? I can only update with File-Update Index Items.

[BLUEFROG]

Welcome @roged

• Why are you choosing to index?
  Did you read and understand the In & Out > Importing & Indexing section of the built-in Help and manual?

• Are you using macOS’ disk management causing your Desktop and Documents folders to be uploaded to iCloud Drive?

• What operating system are you running?

[roged]

Thanks. I got it working on 2 different Macs, one running Sonoma and the other Sequoia, with separate iCloud accounts. I’m looking to add a folder that requires access outside of Devonthink to both Macs, and testing functionality before implementing.

I looked at my 2 different iCloud settings. My documents folder was not being synced to iCloud on either Mac. I did turn off some app settings in iCloud; Automator and Scripts Editor on both and Devonthink on one. After that, the indexed folder updated for new files on both machines.

Many thanks for directing me in the right direction.

[BLUEFROG]

You’re welcome.

I’m looking to add a folder that requires access outside of Devonthink to both Macs

For what purpose or use case?

My documents folder was not being synced to iCloud on either Mac.

So did you enable it? We don’t advocate syncing these two folders to iCloud Drive, though we don’t mandate someone doesn’t.

[roged]

No, I did not enable iCloud Drive syncing, as I’m aware of the issues.

[toffy]

I am also struggling with getting DT to reliably update its database with any changes in an indexed folder (and syncing it with DT on my phone, but that’s a different issue).

I understand from the manual that:

When you rename a file in DEVONthink, the file’s name changes in the Finder. If you rename in the Finder, the change should be automatically reflected in the database. DEVONthink tries to keep the group in the database and the folder in the Finder in the same state.

I also understand from the manual that there are two exceptions, where this may not work:

1. Individually indexed files (I have note of those)
2. Files synced by other applications

In the latter case, the manual advises

you may need to use the File > Update Indexed Items command to manually update the indexed group.

Fair enough. The problem is: even when I do this, it still doesn’t update the database. And to make things worse, it doesn’t tell me anything about what it is doing or whether it is doing anything at all.

My test case for this post is a markdown file that has been renamed from greenhouse.md to my greenhouse.md and which resides in /Users/<username>/Library/Containers/co.noteplan.NotePlan-setapp/Data/Library/Application Support/co.noteplan.NotePlan-setapp/Notes (which is being synced with iCloud via Cloudkit).

I have clicked on File > Update Indexed Items which has led to the file greenhouse.md(which DT reported as missing) to immediately disappear from the search results.

My expectation was that a few seconds later (even a few minutes would have been “ok”) the file would reappear under its new name my greenhouse.md. But even 20 minutes later, it is nowhere to be found.

I can no longer click on File > Update Indexed Items (because it is greyed out). It also seems to have changed t’s name to Update Items - or was it called that all along? - in any case: DT is clearly not acting as it should and I have come to conclude that DT3 simply can’t handle indexed files reliably (if I need to start an investigation every other time I search for a file in DT, there is not much point in using DT for search in the first place) and I would like to ask whether this has been improved in DT4.

Just to be clear, I think there are two problems here: one is that DT3 doesn’t update indexed files, even when manually triggered to do so, and the other is that it doesn’t communicate about this failure (or some ongoing process, if that is what is causing the delay). DT3 is usually very verbose with its log window showing me what’s going on as well as progress messages in the bottom left corner, while it is syncing a database. But now there is nothing. The last entry in the log window is 2.5 hours old and states that 22 items are left to be uploaded to a database.

At the same time, DT on my phone is showing a tiny message at the bottom that it is “Synchronizing Items” with my Mac, but has been doing this for at least half an hour now. Earlier today, after I initialized that sync, I used other apps on my phone, assuming that DT would continue the sync in the background, but it didn’t (not sure if it itself wants to be in the foreground while syncing or whether iOS killed it) which is why I left DT in the foreground to make sure it finishes its sync. But it doesn’t. At least there is no indication of any progress, tapping on the cloud icon doesn’t do anything (this is the case most of the time, not just now. I have no idea how it is supposed to work, but the least I would expect is that it tells my why I can’t tap the icon when I’m tapping it. - Basic UI design principle: feedback on every user action.)

When I go to settings (in the iOS app), I see a spinner next to “Local network”, when I tap on “Local network”, I see a spinner next to my Mac. And when I tap on my Mac, I get an empty page with a spinner, which disappears after a few seconds.

When kill the iOS app and start it again, I can tap the cloud icon and ithe sync message will reappear and everything will be the same as above.

When I quite DT3 on my Mac, the iOS app (which claims to be syncing with that Mac) doesn’t even seem to notice. (I would expect it to complain about the host no longer being available or something, but no.)

When I restart DT on my Mac, everything is exactly like before I quite, except that the entry in the Log window has disappeared, which is probably expected.

What is going on here? Which of these behaviours are expected/ by design and which of these indicate a bug?

In case it matters: my phone is an iPhone 16 Pro with 256 GB of storage of whichDT uses 21.81 GB and 111.84 GB are free. Oh, and I just noticed that in the fast two hours or so, Devonthink has been eating 41% of my phones battery:

I also noticed that my phone was unusually warm this morning, after I left it to sync said database over night, and it is still very warm. I assume that this is related to the extreme battery usage of the app:

[jonmoore]

A few things to be aware of if you’re going down the route of indexing your assets vs importing them.

• Be careful you don’t do anything in DT that creates ancillary copies directly in the DT database. That can lead to problems updating the index.
• Another common gotcha is mistakingly adding content to your DT database when you believe you’re indexing the content. It’s an easy mistake, and one which most often occurs when drag/dropping content with the ⌥⌘ modifier keys held down. It’s all too easy to accidentally lift one of those modifiers just at the point of indexing in DT.
• On a personal note, I never index Library folders directly, neither System nor User, as this has caused me problems in the past. On paper, this may not be an issue these days but better safe than sorry.
• If indexing content from a Library location is essential to you (your NotePlan scenario),
  first copy/paste the folder outside the Library to e.g. your Document’s folder structure, and then create a symbolic link of that folder to place back in the original Library location (and deleting the original folder).
• Much as you can use a Cloud service as a Sync location between two separate macOS devices, I avoid this. Just search for “Sync Explained” in the DT in-app documentation, as that describes the logic behind this suggestion.

I’ve been using an “index content” DT workflow for nearly 20 years and have never encountered major issues. However, the Info Inspector is your best friend when checking if your content is actually indexed by DT or imported into DT. This Info Inspector is the first option in the Inspectors section on the right side of the UI.

Overall, don’t sweat it. DT has the tools that allow you to see exactly what’s going on with your data. You’re sure to make small errors along the way, but nothing critical, as long as you avoid using the likes of Drobox, iCloud, OneDrive etc. to sync DT across multiple devices directly. As it happens, I back up my DT databases to one of my cloud services, but that’s separate to using DT Sync.

A final note on using Documents as your main location for both your DT data and for other folders that you want to be indexed by DT. By following this convention, you’re guaranteed to have local file paths that start ~/Documents. The macOS file system will create identical local file/folder paths on both macOS device one, and macOS device two. And if you’re running out of space on your system drive, simply use symbolic links of those folders on secondary storage devices into your home Documents folder. This keeps your system drive, lean and clean.

[toffy]

Thank you for taking the time to respond. What i gather from your response is basically that DT is a fragile piece of software that wants to be treated in ways that may not be immediateky evident to the uninitiated. This confirms my impression after having used (or perhaps: tried to use) DT3 for a couple or years now.

I admire your willingness to comply with the needs of the software and I wish I could take a similar stance, but my basic attitude towards software is too different for that. I believe that software should comply with the needs of the user. And where this is not possible (it never is possible 100%), I expect it to be transparent and coherent with how it works. Based on the experience I described above, DT currently doesn’t seem to work that way, unfortunately. Which is why I inquired whether DT4 will be any different.

@jonmoore

Be careful you don’t do anything in DT that creates ancillary copies directly in the DT database. That can lead to problems updating the index.

What is an ancillary copy and what kind of action will create one? Shouldn’t DT itself be able to take care of this? In my mind, the user should not have to think about what actions inside DT might create “ancillary copies” and how this might affect DT’s performance.

@jonmoore

Another common gotcha is mistakingly adding content to your DT database when you believe you’re indexing the content. It’s an easy mistake, and one which most often occurs when drag/dropping content with the ⌥⌘ modifier keys held down.

I don’t see how this is related to the issues I’m describing, but, either way, I don’t do any dragging and dropping with modifier keys held down.

@jonmoore

never index Library folders directly, neither System nor User, as this has caused me problems in the past. On paper, this may not be an issue these days but better safe than sorry.

What kind of problems has this caused?

What I’m aware of DT may have problems with updating files in folders synced by third party apps (these folders may be under Library but not all folders under Library are synced, so I’m not sure what else might be special with those folders. I appreciate very much that the DT developers are transparent with this issue (and even explain in the manual that this has to do with sync-apps not necessarily creating system events that would allow DT to follow up on any updates). But, as I describe above, the manual update doesn’t seem to work either, and DT is completely silent about that.

@jonmoore

Much as you can use a Cloud service as a Sync location between two separate macOS devices, I avoid this.

Just to be clear: the syncprocess that I was referring to is not via a cloud service but directly between devices via the local network.

@jonmoore

DT has the tools that allow you to see exactly what’s going on with your data.

That’s what I thought, yes. But what I described above does not fit into that picture.

[BLUEFROG]

@toffy

In my mind, the user should not have to think about what actions inside DT might create “ancillary copies” and how this might affect DT’s performance.

If DEVONthink was a Finder replacement (and it’s not), perhaps you could take this stance. In DEVONthink, you must know and understand these things. This is why the In & Out > Importing & Indexing section of the help is extensive.

My test case for this post is a markdown file that has been renamed from greenhouse.md to my greenhouse.md

You renamed it where - on the same Mac you’re saying things aren’t updating or on another Mac with CloudKit syncing the Noteplan data?

---

Hold the Option key and choose Help > Report bug to start a support ticket and we can look at the logs.

[toffy]

@BLUEFROG

In DEVONthink, you must know and understand these things. This is why the In & Out > Importing & Indexing section of the help is extensive.

And that’s why I have read this section and keep coming back to it whenever something isn’t working as expected. Unfortunately, it rarely provides an explanation for what’s going on. (And it doesn’t explain what ancillary files are either).

What I am saying is not that I don’t want to learn how a software behaves. But I expect it to behave the way the manual says it does. And then there are certain things that I think the user should not be burdened with, e.g. learning a list of operations that should not be performed within the software. It makes sense to learn to avoid certain things outside the software in question, because the software doesn’t have control over these. But when the software has control, i.e. when I’m using it, it shall prevent me from doing things that it doesn’t like.

Anyway. this is just an aside. Let’s focus on the actual questions at hand: have any of the above mentioned issues been fixed/improved in DT4?

I have submitted the logs as instructed. Note that, during the past couple of hours, I have been trying to clean up the databases, hoping that I might find some clues as to what is going on, but unfortunately, I only came across more puzzling behaviour. For example, DT had a very old folder in its index, containing several thousand files/folders. That folder used to live on my Onedrive but I have long deleted it. Even a manual “Update items” couldn’t convince DT to remove those files and folders from its index. I even went into the terminal to make sure that the folder didn’t exist any longer, but DT just wouldn’t update. I ended up deleting the folder manually via the Trash and then emptying it and selecting remove everything.

As I was trying all these things, I noticed that the command “Update Items” apparently wond drill down into the folder hierarchy but only process the top level. Is this correct (again: DT gives no feedback on what it is doing)? If so, is it intentional? It makes that manual reindexing extremely laborious and basically pointless. In practice, it is not feasible to navigate through the entire folder hierarchy when updating the index.

Another weird behaviour I didn’t understand is that almost every time I deleted som file from an indexed folder (in the Finder), DT suddenly produced a status update (bottom left corner) that it was now uploading one or more (completely different) files to the cloud. While I’m always happy, when DT tells me what it does, I find it puzzling why deleting a file would trigger an upload of other files (all of which have been there for years).

@BLUEFROG

My test case for this post is a markdown file that has been renamed from greenhouse.md to my greenhouse.md

You renamed it where - on the same Mac you’re saying things aren’t updating or on another Mac with CloudKit syncing the Noteplan data?

The file was renamed (days or weeks ago) on the same Mac (I only use that one), the cloudkit sync is going to my iPhone. Other file names or content has been changed on the Noteplan Phone app, but not the greenhouse.md file.