With DEVONthink on the Mac, I use the indexed folder functionality to index a particular local folder on my disk. That folder’s contents are added to by another application (Zotero, to be specific). This approach been working more or less fine for many years. However, there are files in that folder that I really don’t need or want DEVONthink to index. So far, what I’ve been doing is to use smart groups to create views that basically hide those unwanted files in DEVONthink. This works, but is definitely not ideal because it requires active management and because it results in hundreds, maybe thousands, of files being indexed needlessly.
What would be better is to have some way to tell DEVONthink to exclude certain files from indexing, using some kind of criteria or rules. Requests to support exclusions have been made in this forum before (e.g., this posting in 2016, this posting in 2017, this posting in 2018, probably others). No built-in functionality is available currently in DEVONthink, so we’re down to using workarounds.
The workaround that’s been mentioned a couple of times is using the CLI command chflags hidden works, but in my experience, it only works if you set the flag when DEVONthink is not running. If DEVONthink is running, and has already indexed a given file, then doing chflags hidden on the file does not make it disappear from DEVONthink. Consequently, the only way I can see to use this approach is to remember to quit DEVONthink before I take any action in the other application that would add files to the folder in question, then run chflags hidden on the files I want to exclude, then restart DEVONthink. That’s just not a sustainable approach.
Are there other tricks for excluding a subset of files from being indexed?
If there’s a pattern (filename or extension) you could use chflags hidden as DEVONthink doesn’t index hidden files. But you have to make sure that the files are hidden before DEVONthink sees them - chflags hidden has no effect on DEVONthink if the files are already indexed (I guess this is a bug @cgrunenberg?)
If the solution were that easy, then I wouldn’t have to ask my question .
In the situation I’m facing, I’m already indexing the lowest-level folder that I can feasibly index. Being able to index the contents makes it possible for me to use DEVONthink’s many outstanding features, and is one of the reasons why I use DEVONthink. I wish I could change the other program’s behavior in this case – I really do, because it would make some other things easier too – but that’s not possible.
Thanks for your reply and practical suggestions, @pete31!
Regarding this:
It seems to be not a bug, but the expected behavior. In a posting 14 days ago, @BLUEFROG wrote the following in response to someone else’s inquiry about indexing hidden files:
So it seems to be a known behavior.
I’ve been thinking more about this, and I think there are arguments for and against how it currently behaves.
On the one hand, the current behavior seems (IMHO) inconsistent with how DEVONthink updates its indexed folder views in response to changes on the file system. When a file disappears due to be deleted or moved from an external, indexed folder, it disappears from the indexed folder view presented by DEVONthink almost immediately. Intuitively, it seems to me that marking a file as hidden means it is intended to disappear from the normal view in the Finder [1], so intuitively, I would expect it to disappear from the view presented by DEVONthink too. It’s clear from other people’s postings on this forum that they, too, have the same expectation about the behavior involving hidden files. So, I would argue that DEVONthink’s current behavior goes against user expectations with respect to indexed folders.
On the other hand, I wonder if changing DEVONthink to update its view in response to hidden files would introduce different unexpected behaviors. Two situations that might be problematic are: when the user trashes (in DEVONthink) folders containing hidden files or hidden subfolders, and when the user moves a folder from an indexed location into the database, resulting in the “consolidation” behavior (or is that deconsolidation? I never get the direction right).
The developers would certainly have a better sense for whether the situations in point #2 cause insurmountable problems.
[1] Unless one toggles the Finder setting to show hidden files.
chflags is definitely not something we’d openly advocate
Development would have to chime in here, but the issue you’re referring to isn’t very common. I’d say the vast majority of our users are just indexing a normal Finder folder with common applications, e.g., Word, Numbers, etc. or a cloud-synced folder they’re managing. So proliferating support files are not going to be problematic for them, so they wouldn’t need excluding or hiding things.
Just some wild associations: if DT relies on file system events top update indexed files… And if chflags hidden does not generate a filesystem event …
I have no idea if that’s what causes the inconsistency, though.
And I’d argue that to hide something from view is not the same as to move or delete it. Not at all (think of hiding money vs burning it). The hidden flag or a leading dot in a filename are there so that certain programs (ls , Finder) don’t show the file by default. Like a declutter mechanism, as such files are often not what the user is interested in. But it’s obviously easy to un-hide or rather display them. Which is not feasible with a removed file.
That’s a good question, actually! I searched for an answer, but could not find one, so finally I posted a question to Apple Stack Exchange.
I agree that they’re not really the same, functionally. Still, I think user expectations are, roughly, “if I can’t see it in the Finder, I don’t expect to see it when the location is indexed by DEVONthink”.
Sure, but setting aside my specific snowflake situation, the question of how to exclude/hide/not index selected content has been asked more than once in this forum. So, not to be argumentative, but it seems like the general problem (how to hide/exclude content from indexed folders dynamically) is something other people have run into, for other situations.
I did a little experiment. In the shell I did the following actions:
cd folder_that_is_being_monitored
# Create a test file
touch test.txt.
# The file test.txt is visible in DT as expected
mv test.txt renamed.txt
# No problems here. The rename is reflected in DT
chflags hidden renamed.txt
# No change in DT3. This was the observed behavior by others
mv renamed.txt test.txt
# Interestingly rename is picked up by DT3 and the file shows as test.txt
I would argue that the last line indicates a possible bug or opportunity for improvement on the DT3 side. Apparently when a file is renamed in DT3, there is system event that is being generated and picked up by DT3 but it does not inspect the properties of the file and therefore does not hide it consequently.
Did you use a cloud folder for testing? Then this is intentional and you have to use File > Update Indexed Items on your own (to avoid data loss due to race conditions caused by filesystem events, cloud sync and DEVONthink’s sync as their order is undefined).
It is a Dropbox folder. Just to be certain I repeated the test with a regular folder and indeed the behavior is different.
]It sill doesn’t pick up the hidden flag on renamed but does delete it when I remove the file.
I find this odd, because the Dropbox folder is (should) not (be) special. I have not converted to the new model of Dropbox which uses Cloud technology. Anyway, the main point being that the hidden flag is not picked during rename remains.
I am missing something I guess. What are you referring to with “see explanation above”. What I read it that it relies on system events.
What I can see is that both new files and renames are automatically picked up (both in Dropbox) and in a regular folder. The fact that this happens seems to proof that system events are generated in both situations. The difference is when I remove a file. This is not picked by DT when the file is inside Dropbox and is picked up in a regular folder. The same applies for moving files outside the indexed folder. This is picked up by DT in a regular folder and not inside Dropbox.
The latter can be used as a trick to apply the hidden status on an existing file.
# This only works in a regular indexed folder, i.e. not in Dropbox
mv file /tmp # The file is removed from DT
chflags hidden /tmp/file # The hidden status is applied
mv /tmp/file . # Move back. The file stays hidden in DT
.