Indexing files (academic papers) being renamed externally

Rant about Mendeley and a quest for a workaround -

Background: I’ve used Mendeley for years, and sure, the app is becoming obsolete but it still worked well enough for my needs (saving references & PDFs directly to my library from browser; having my nested library that I wasn’t keen on investing time to replicate elsewhere before I have to) however, I think it’s wrecking havoc on my DT experience.
Namely, every now and then my DT database has a bunch of missing files, but the actual PDFs are still in their finder location, DT just has some issue with them. I’m starting to think the issue is created by Mendeley’s renaming of files - very often I notice a PDF has a mistake in its metadata within Mendeley and hence in its name (Mendeley renames them automatically, no complaints there), so I fix the metadata → Mendeley renames the file again, but I think that results in the file being perceived as missing by DT.

Question: I’ve been thinking of switching to Zotero before Mendeley sinks entirely, has anyone experienced issues with Zotero’s renaming of files having a negative effect on their index in DT?

1 Like

You can see more about what happens when indexed files are changed outside of DEVONthink by reading again “Indexing and the Filesystem” on page 51 of the current 3.8 “DEVONthink Handbook”. (Deliberately not writing more here to avoid more confusion!).

I read that before posting. I don’t see how it answers my question - I get that having names of indexed files changed externally is not ideal, but am looking to hear from folks who rely on this feature in Zotero (or Bookends) whether it causes a lot of issues in DT or not.

OK. Well, anything (software or people) changing names/status of indexed files outside of DEVONthink will cause anomalies at best, file loss at worst. There are a lot of previous discussions on Zotero and Bookends here you might want to find.

1 Like

I did browse previous topics, but haven’t found substantial info on this particular issue.

To me that makes sense; but I would expect that if you manually triggered File > Update Indexed Items the problem should be solved. I guess, but don’t know, that at that point the “original” record would be removed from DT (as it is “missing”) and a “new” record would be created. The disadvantage is that any metadata you had added to the record in DT (tags etc.) would be lost.

Out of interest: do you keep DT open at all times? I kind of thought it would notice any changes made to an indexed folder whilst it was alive and kicking and that only changes made whilst it was in bed asleep would be a problem. But I don’t have any basis other than assumption for that claim.

I think this thread answers your question.

1 Like

Hi, OP of the Zotero thread chiming in here. As discusses in my post, Zotero causes similar issues for DEVONthink.

3 Likes

I know that this isn’t what you asked about, but I’ve been a happy Zotero/DT user for a long time. The big difference is that I store all my PDFs in DT (import not index), and only use Zotero to store the citation metadata and use its reference manager plug-ins with Word. It’s something I would recommend if you are considering a change up, and perhaps Mendeley offers a data export, or you could manually shift those PDFs to DT?

I’ve long thought of clever ways to integrate the two apps more fully (e.g. embed DT links to the Zotero record), but in practice while I use Zotero regularly, it’s just not worth the overhead since searching is great in both apps.

But I get that once you are invested in a system that works for you, it’s hard to change up and things may feel counter intuitive. Zotero’s PDF annotation option are sub-par to DTs for instance (I sync my academic PDF libraries with DTTG and I use that natively for all reading and annotation, another reason to keep all PDFs in DT).

Good luck!

4 Likes

so I fix the metadata → Mendeley renames the file again,

Can’t you disable that behavior?

#1 it sounds like a dreadful idea to me :slight_smile:
#2 You could test to see if it’s indeed the cause of the missing files issue.

It’s really handy when you download dozens of papers at a time, and want to have them named appropriately and logically, but don’t want to spend your life renaming them :wink:
I’ll let sleeping dogs lie atm while I finish what I’m working on, then reorganize my workflow entirely and get rid of Mendeley, and either let go of renaming of import rather than index my library and let DT do the hard work and Zotero pick up the referencing slack.

Have you found any workaround since your post or have you abandoned renaming?

We’d appreciate it if you could test #2 once you’ve finished your work. That is potentially useful information. Thanks.

Will do.
Just to clarify, we’re expecting the issue arises only when DT is open while files are first introduced to the indexed location and name changes are taking place (externally), not when DT is closed as this happens - it doesn’t do any sort of indexing in the background while app is closed?

2 Likes

Not really, unfortunately. I simply double-check the metadata (and manually edit when necessary) of newly added items in Zotero before indexing the attachment in DT.

That’s correct and thanks for your assistance on this!

Hi – I can’t speak to Mendeley, but I use Zotero daily. I index my Zotero storage location using DEVONthink, and can with certainty that when I do something that causes Zotero to rename files, I have not experienced problems in DEVONthink due to the renaming process. In fact, it’s kind of impressive to watch in real time: with a window open on my smart group for references (discussed in another posting), if a file has already been indexed in Zotero and it gets renamed, the DEVONthink smart group will show the updated name after a second or two.

For this to work, the entire Zotero storage folder needs to be indexed by DEVONthink. If you index individual files, you will probably run into the problem discussed in the thread linked to above by @Blanc.

I wrote some explanations about how I’m using Zotero with DEVONthink in the pages associated with a software tool that I wrote to help DEVONthink-Zotero integration.

4 Likes

I didn’t want to do this because the hierarchy of the storage folder is not human readable, for lack of a better term. PDFs are scattered across many us folders with gibberish folder names. Indexing files individually allows me to put each file in a folder I see fit.

Likewise. I’ve indexed the entire storage folder and have never had an issue.

I use a smart group so I don’t have to navigate the Zotero storage hierarchy. But I don’t organize them any more than that in DT.

In my case it’s the original Mendeley folder that I’ve indexed, and haven’t organized further within DT, other than tagging individual files then using smart rules to sort some of them.

Once I have the time I’ll test this out further, curious to see how Mendeley and Zotero influence this issue. Right now it compromised my entire database and it needs a rebuild, which will have to wait.

Absolutely true. The simple solution in DEVONthink, as @rpallred alluded to above (and also mentioned in another posting), is to use a smart folder over the storage folder to produce a flat list of files. I described my approach in another forum posting and recently wrote a slightly updated version of the same in a wiki page on GitHub.

3 Likes