I understand that, and I wasn’t maligning DT3. I was queried why I am indexing through Zotero rather than importing directly into DT3, so I am simply explaining my reason… As a newbie, I also gave the details in case there was something I didn’t know which would have solved this dilemma.
No offense taken at all. My clarification is for the entire class and future readers as well.
Some thoughts …
- I find the integration with PDFs to be more robust across the DT + Bookends pairing than across the DT + Zotero pairing. Especially when you will annotate on PDFs in Zotero, you will have extra effort to carry the annotations over compared to using Bookends.
OTOH …
- Zotero seems to handle storing Web pages as resources better than Bookends does.
I note here that I am biased for Bookends versus Zotero, primarily because I do not use Web pages as citation resources and because working with PDFs in Zotero is simply too much friction in with my other tools (DT, PDFExpert, LaTeX, iPad)
Otherwise, reading between the lines, you appear to want Zotero primarily to manage citations for a document and to want DT to manage the reading, assimilating, and processing on the collected content. If so, I second the recommendations that you work within DT to collect and process resources, exporting necessary content to Zotero primarily (if only) to compile citations.
Finally, consider testing a workflow to produce a brief (10 page) publication ready document on the theme “My Brief Review of Field XYZ”, using the apps that you have in mind (Word + DT + ???) with a mixed collection of the resources (Webpages, PDFs, …) that you already know well. Make this in a separate folder if not a separate volume on your hard drive. Keep notes as you go. Use it as your “Users Manual” once you start your processing on your real productions.
–
JJW
4 Likes
If you plan to move the files to Groups then you may be falling into one of the biggest potential gotchas with indexing.
2 Likes
My workflow:
- I put every file/folders into a file pool
- I build collections with smart groups based on metadata and tags
- I create note cards with context information and item link inside or wikilnks to connect my smart group collections.
Thanks, I would prefer this too, but I just can’t get DT to capture information accurately and easily. A lot of my sources are web-based pdfs, from sites like SSRN (Social Science Research Network), and reports produced by thinktanks, governments, and international bodies. The DT clipping options don’t seem to give an easy way to get them into DT in the original pdf form with a filename that reflects their origin and authors. I appreciate that this is not the purpose of DT, but it does mean that I end up going to Zotero for a quick one-click downloading and properly naming the file (and for my sources Zotero seems to do this better than Bookends does). If I could figure out a way to smoothly get this into DT in one step, I would gladly do it!
Thanks for the warning. From what I can see on this site, I’m not alone in using DT to index storage from a reference manager, so hopefully that means it can work. I’m trying to do this in a way that minimises risk: All the documents captured by zotero go into a single storage folder, which is used exclusively for that purpose. I’ll never directly interact with the storage folder, and will not touch any of the files in it directly. DT indexes this folder into a DT Group called ‘Zotero’. I will keep all the indexed items in this Group, to maintain a 1:1 item list with the original Zotero Folder.
When I organise items into other DT Groups, I will do so by creating replicants into the various Groups to which I categorise the items. That way I always have the indexed folder still with all items that are in Zotero.
I will only annotate and take notes in DT, not through Zotero. When I get to writing in Word, I can use the Zotero plugin and easily create all the citations and references.
Hopefully this process will ensure a clean and relatively simple way of working safely with an indexed folder.
For all other non-research database of documents I do plan to place the files directly in DT.
You can probably fix that with a smart rule that runs after import and sets the title to what you want. If that includes the authors (and if then the title becomes very long), depends on the availability of this information in the PDF’s metadata.
1 Like
OK. I understand better what you want. Unfortunately, as noted by @BLUEFROG, DT is not designed as a reference or citation manager in the way that you need. At the front end, a reference manager effectively handles retrieving and storing meta content for a resource document (reference). At the back end, a citation manager effectively handles linking the resource citation into a word processing app. The two jobs, reference and citation management, are handled by apps such as Zotero, Bookends, EndNote, or Papers3. Sometimes, these apps also overlap with things that are done well in DT, such as searching, cataloguing, and allowing annotation processing on the document resources. I might call these group of abilities as resource processing in reference to a note later below.
In my long experience with DT, and in review of postings to this effect, you should not expect DT to ever become adept at reference management to the level that you want or need. This goes double for the back end side of citation management. In this light, perhaps you might consider whether any of the reference/citation management apps does at least what you think you need from DT. More on this thought below.
At the risk of again showing my bias, consider posting on the Bookends forum to ask whether your issue is unique and/or has been resolved in some way. The user base includes a number of active folks in the social science + humanities fields. Ask also if only for the reason that Bookends ↔ DT are much better all around companion apps than Zotero ↔ DT. By reference also to this …
The attachments in Zotero are closed in their own internal folder system (which is likely one aspect that @BLUEFROG meant by saying that Zotero is a closed-loop system). The attachment folders in Bookends are collections of PDFs and other resources that are directly exposed in the Finder and thereby are easier to manage across the (Finder + DT + Bookends) set up.
Finally, to touch on the cornerstone of your twin needs: reference/citation management + resource processing. Aside from the issues I have with Zotero in how it handles PDFs, I could not truly grok how to use Zotero as a resource processing app. I therefore appreciate why you have the need to include DT when you stick with Zotero. I have in the meantime become very pleased with how Bookends is expanding its abilities in resource processing. Searching (smart groups), cataloguing, and annotation is quite good, even on the iPad (where I do a significant amount of my reading and annotating). To this extent, I don’t need DT to do resource processing because Bookends does what I need. But … to counter, the Bookends forum is replete with postings about how to manage the DT ↔ Bookends linking, and I do use DT as a resource manager for other jobs (e.g. assembling lecture, assignment, and example resources to manage course workflow).
As a sidebar, to offer other apps in the mix, Papers3 has become quite good at providing tools to search and catalogue its document resources (and for good measure, the Papers3 attachments are also exposed at the Finder level and could thereby more easily be indexed in DT vis-a-vis using Zotero). In some ways, Papers3 is also a more polished app than Bookends, albeit with fewer options to customize workflows. EndNote is also touted by some as a gold standard in reference/citation management. Perhaps your university has a site license and/or your library administration provides training for it.
I hope this helps gather greater insight to what you may need to tackle the job ahead of you.
–
JJW
3 Likes
Thank you, I very much appreciate the detailed and thoughtful response.
In Zotero I have the attachments moved automatically through the zotfile plugin to my workplace Onedrive folder (which is always fully downloaded on local computer), which is the one that gets indexed by DT. It means that the files are stored in a known and accessible folder if there ever were a problem.
I used endnotes in the past (university license as you say), but really disliked it. I hadn’t looked at Papers, I might check it out.
The reason I was drawn to DT was primarily around the combination of grouping with replicants, labelling, and in particular the search functions that find connections and help me pull out sources that I might not have remembered exist in my library. Being able to do that with a combination of pdfs, web sources, and notes, seemed unique and very useful. I hope I’ve not fallen into the sunk cost fallacy just because I already paid for DT… Fingers crossed that my faith in its superior organisation and searching turns out to be correct!
I download to the Global Inbox for this. The file will get whatever name the original site gave it, which may or may not be useful.
1 Like
I use Paperpile to capture and organize pdfs and their references (metadata). All files are stored in Google Drive. Then, I index the pdf files in DT. Everything else (reading, taking notes, etc) is done in DT. The system works well.
3 Likes
Just be aware that on Github it says that ZotFile is not actively developed or maintained.
2 Likes
I just finished a masters degree using DEVONthink and zotero together; both tools were indispensable to me. I find it’s best to use Zotero for capturing PDF documents from the web (with the Zotero browser extension), capturing metadata, and automatically renaming the PDF files based on the metadata. I’ve indexed the entire zotero storage folder into DT3, then created smart groups in DT3 to filter out all the randomly-named nested folders that Zotero uses internally. My smart group only captures PDF and ebook files, and it creates a flat folder hierarchy for easy browsing.
I do all my PDF annotations in Zotero because I find it more reliable than any other PDF annotation workflow. Annotations made on Zotero don’t actually touch or modify the PDF file at all - technically, the annotations are written to a separate (hidden) file, and then Zotero’s PDF viewer visualizes the annotations on the screen. This is a double-edged sword: annotations don’t show up in DT3 or any other PDF viewer, which has obvious downsides—but on the upside, you get a clean, annotation-free view of the document. I make do by using a multi-monitor setup, and I put Zotero in full screen on one monitor, and DEVONthink in full screen on another, and my writing software on another screen. I primarily use DT3 for running text searches for phrases and terms across my entire collection of text sources. It’s great because it lets you use boolean logic and widecards (which is much more advanced than the search features offered in Zotero. DT3 searches are faster and more seamless than Zotero, too).
The main reason I use Zotero for annotations instead of DT3 is because Zotero never makes changes to the actual PDF files. PDF is a terribly fragile file format.The PDF file specifications are a mess, and no PDF editing program is fully compliant (as far as I know). No two applications for editing PDFs work in exactly the same way, either. DT3 (desktop) and DTTG (mobile) are not fully consistent with each other in the way they make annotations—same goes for PDF Expert (desktop:mobile), Acrobat (desktop/mobile), etc… And in my experience, I’ve noticed that using a variety of different applications to make annotations in the same PDF tends to increase the likelihood of it becoming corrupt and breaking the usability of text search within the document. I frequently switch between desktop and mobile, and make annotations on both devices. In my experience, Zotero is the only PDF viewer that has never caused a PDF to become corrupt, no matter how many times I switch between desktop and mobile as I make annotations. To me, this reliability invaluable, because a PDF file with corrupt text doesn’t perform as well when I run text searches within the document. And since I primarily use DT3 for running text searches since I’ve outsourced most of this workflow to Zotero, I need the text searches to be reliable.
4 Likes
Oustanding plan @aaaaaaaaaaaaaaaa
Makes impeccable sense
That’s really interesting, thanks! I didn’t know about the annotation risks, so might consider doing it the way you suggested.
Given that I’m just about to build my workflow, I’m thinking of going straight to Zotero 7. I’ve seen there is a zotmoov plugin that can move files the way zotfile did. Have you tried it and do you know if it’s safe to install?
I’m also thinking of trying without any plugin and using a symbolic link to onedrive storage.
Acrobat would be the most compliant as Adobe is who created and developed the PDF format and specification. However, many other implementations have lead to a wide range of non-compliant and problematic PDFs since the format’s inception.
PS: Skim.app doesn’t modify the original PDF’s layers, instead writing the extended attributes of the file…
Whether this is a good approach or not depends on the situational needs but it’s certainly not portable across platforms and may even be stripped when put into a cloud-synced folder, e.g. , OneDrive. Testing would have to be done to determine such effects.
Fellow academic here! I have a simple setup using DT and Zotero. My trick? DT takes care of file storage, management, and searching etc. Zotero only contains the citation metadata that, as you’ve found out, DT is not well suited to collecting at the click of a button. DT is much, much more robust as a knowledge management system, but I love Zotero for the citation plugin with Word. For clarity – my Zotero database contains no PDFs at all.
This may seem cumbersome in that you end up downloading the citation + PDF to Zotero (often with the automated browser plugin), and then push the PDF to DT, but in truth that doesn’t take me much time. Nor do the odd times I need to manually add citation metadata.
The simplest setup, in my experience, is to avoid any roundtripping between the apps; avoid indexing of Zotero folders; etc. There is a little overhead, but nothing that’s getting in my way. Document management shared between DT3 and DTTG, and annotation on iPad with the Pencil, is out of this world – very robust, and very reliable for me. I sync via Dropbox.
3 Likes
Thanks. By “push” do you mean manually move each file into DT?
Is there a way to have DT watch a folder and auto import all files as they are added?
And does that not cause a problem in zotero if it thinks it has the files stored but you move them away? or will the citation word plugins work regardless
Yes - I do. I used to just open the PDF from the Zotero import file, then from Preview’s sharesheet push it to DT. Now it tend to download the PDF separately to Downloads and move it to DT from there. Smarter people can probably automate this process, e.g. via Zotfile and a Hazel rule, for instance. But I’m a simple person!
I actually find that this little bit of friction helps me think harder what PDFs I want in my DT and why. Sure I can easily download entire issues if not journals worth of stuff without bother via the Zotero plugin, but do I really need all of that stuff? Too much noise, too little signal. At least for my kind of work.
For much larger research jobs we are probably in the early days of new research possibilities anyway, running complex (AI) queries on large online repositories (e.g. https://www.semanticscholar.org; Elicit: The AI Research Assistant). So I prefer my DT workspaces to be smaller and more intentional – and my workflow helps with that.
Zotero to Word plugin doesn’t care if there are PDF or other attachments to its entries. As long as the metadata set is complete you are good to go. I’d run some test if I were you, mock up something to make sure it’s working for you.
But you are right, if you manually move a PDF that is added there by Zotero, you won’t be able to open that file anymore from within the app since it has lost track of the location.
But as you can see from my library, I just have no attachments (nothing marked under the paperclip symbol). Works just fine in Word!
1 Like
This is sage counsel and nicely reinforces our philosophical stance of thoughtful curation of a database. Thanks for sharing that sentiment! 
2 Likes