Question about syncing

DEVONthink
1

I am setting up DT to Index (NB not import) my Bookends attachments folder. However, I want to have the same setup on my Mac mini and on my Macbook. The Bookends attachments folders in the two machines are kept in sync using Synology Drive, both synced to the same server, and the two DT databases would be synced through a DT sync store on the same Synology DS using webdav.

Before I put the DT database on to the Macbook, I would like advice on whether problems are likely to arise with such a setup. When they are in sync, the two Bookends attachments folders should be identical, so it should not matter that I am syncing the same DTP database with both of them. (The locations are the same relative to my Home folder, though the usernames on the two machines are different.) However, syncing takes time, especially if files are large, so there will presumably be moments when they are out of sync or when the DTP databases are out of sync.

So my question is, Would I have to be meticulous in making sure that the two folders are fully synced, and that DT on one machine is closed before I open the database on the other, or can the sync protocols, about which I know nothing, cope with such situations? There is no problem with Bookends, because, without purchasing a second licence, it does not allow two instances to be active on the same network, so one has to be closed before the other is opened (and it does not matter if an attachment is temporarily unavailable).

2
  • Where is the Attachments folder for Bookends?
  • Why are you using the Synology Drive app to do the syncing of the Attachments folder?
  • Are you syncing the contents of indexed items to the WebDAV sync location?
3

Thank you for the quick response. I deduce from your questions that the answer is not a simple, “No problem - go ahead.” If it won’t work, perhaps because DTP relies on uuid’s not file paths/names, and those would be different on the two machines, then I can work out another method.

Answers to your questions:

  • BE attachments are in a folder inside the Documents folder in my Home folder; my aim is not to have any app other than BE change this folder, to avoid BE problems. I hoped DTP indexing and locking would avoid problems. (Well, no problem if the content of each PDF changes with annotations etc, but I do not want any app other than BE to touch the filenames or move the files.)

  • (Why Synology Drive?) To use BE on both my desktop (Mac mini) and laptop. The reasons for syncing using Synology Drive rather than having both BE store their attachments on the Synology are: (a) speed when using BE; (b) the laptop does not always have access to the Synology.

  • (Syncing indexed files to DT Sync store) I have synced the Mini to the Webdav location (the same Synology disk station) but I have not yet synced the MacBook to it. I was waiting to find out whether such a setup was safe, or whether I had to follow rules like never having two machines accessing it at the same time. Maybe it would not even work, if DTP does not use filenames to identify files. I don’t know because I have not tried. I have been using a sync store to sync DTP databases across machines because this seems incredibly quick and efficient. I forget why I chose webdav: I probably read something on this forum that made me think it would be a good choice.

My goal in indexing the BE attachments folder was to use DTP’s search capability to identify useful files, and then using wikilinks, go directly to another file in DTP that is the reference imported from BE (an RTF), to which I can add notes. I could also move these files into groups, create replicants and so on, using DTP to organise the notes into some structure.

But if this method is inadvisable or will not work, then I guess I could have two independent DTP databases (one on each machine) linking the attachments folders, and when I identify a useful PDF in DTP, I can then use an AppleScript (found on the forum) to switch to seeing it in Bookends and make my notes there. That would definitely work though I would lose the ability to use DTP to have notes wiki linked to the PDFs, and to organise notes in DTP.

I am still experimenting, trying to establish on a workflow that is better than the one in my last major project, where files in DTP were duplicated in BE, often with different filenames and not linked, so after identifying something useful in DTP, I then had to identify it “manually” in BE, which was not very efficient. The reason for this is that, back in 2011, or thereabouts, I just jumped into using DTP alongside a reference manager (initially Sente and then, from around 2016 or 2017, Bookends), without thinking through the best workflow. Once I had started, switching to a better workflow, such as in “DT for historians” would have meant re-doing a lot of the work I had already done. So I am trying to be more organised this time.

(I tried other scripts from this and the BE forums, but some (or mistakes I made when trying them out) forced me to restore files using my backups, so I dropped them.)

4

Indexed files are handled via their file paths.

my aim is not to have any app other than BE change this folder, to avoid BE problems.
…
(Well, no problem if the content of each PDF changes with annotations etc, but I do not want any app other than BE to touch the filenames or move the files.)

That’s already not the case if you’re using a sync mechanism, including Synology Drive.

  • (Why Synology Drive?) To use BE on both my desktop (Mac mini) and laptop. The reasons for syncing using Synology Drive rather than having both BE store their attachments on the Synology are: (a) speed when using BE; (b) the laptop does not always have access to the Synology.

This isn’t necessarily needed in my estimation unless you’re doing a bunch of work outside DEVONthink or one Macs where DEVONthink isn’t running. I’d also opt for fewer moving parts. You are talking about dual sync mechanisms as well as indexing files.

I’d…

  • Shut off Synology Drive for this folder on the Macs running DEVONthink.
  • The Attachments folder is indexed into DEVONthink on Mac 1.
  • Make sure you’re syncing the contents of indexed files in the sync location setup. This is also the best idea when adding DEVONthink To Go to the environment.
  • Sync Mac 1 to WebDAV.
  • Sync Mac 2 with the same sync location. Files will be added / updated in the local Attachments directory on this machine. This is done without needing to use a secondary sync process, like Synology Drive.

You can easily test this out by indexing a test folder in the Finder into a database then syncing it between Macs. The test folder will appear in the same relative location on the second Mac and will update between the Macs when synced.

PS: This is not new behavior. It has been in place since the 2.x days.

5

That is really helpful, though I will have to think about it because the reason for syncing via Synology Drive is so that the two instances of Bookends can both access the same attachments folder.

I think I was under a misunderstanding about what linking did. I assumed that if the linked file (outside DTP) was not present on Mac2, DTP on Mac2 would not know about its contents. It sounds as though DTP stores inside itself more information about the linked file than I realised.

Would your suggestion be OK if Synology Drive continued to sync the Mac1 files to Mac2, but placed them somewhere where DT would not look for them (ie not in the same location as on Mac1)? I could easily change the place where BE looks for attachments on Mac2. In other words, though two sync operations would be going on, they would be completely separate. The connection would be made only by DTP on Mac1.

I did not mean to imply that this could not have been done in 2011, merely that I jumped in naively without understanding the benefits of planning more carefully. I cannot even remember when I had the luxury of two Macs.

Thanks again for your help.

6

You’re welcome.

so that the two instances of Bookends can both access the same attachments folder.

Are you meaning the attachments directory is on the Synology Drive?

Would your suggestion be OK if Synology Drive continued to sync the Mac1 files to Mac2, but placed them somewhere where DT would not look for them (ie not in the same location as on Mac1)? I could easily change the place where BE looks for attachments on Mac2. In other words, though two sync operations would be going on, they would be completely separate. The connection would be made only by DTP on Mac1.

I don’t understand what the benefit would be here.

7

When I said the same folder, I meant folders with the same content because they were both synced with the same Diskstation. Sorry for my confusing shorthand. The reason for that is that BE syncs its library (via iCloud) but does not sync the attachments), hence the use of the diskstation as well.

Before responding any more, I shall set up a demo in the way you suggested, so I can understand better what happens when indexed files are synced via a sync store onto a computer that cannot access the original folder.

8

No worries!
Let me know how it works. I’ve been using this mechanism for a variety of things for years. It’s a super useful use of indexing IMHO.

9

Having played with a demo, I think I understand what is happening and it will do all I want. I now understand why using the DT sync store and indexed documents is an alternative to other sync methods. Thanks again. Indexing files is indeed a lot more powerful than I realised. Though I should have been able to work this out from the manual, I had not realised that it could be used to keep folders outside DTP databases synchronised across machines.

I am not clear what Marking a linked folder as locked does. Would this prevent me from accidentally moving or deleting a file when working in DTP, while still allowing DTP to sync the folders on two macs? If not, then I guess I just mark the folder in red - not to be touched - and create replicants of all the files in a different folder that can be moved around with impunity (ie without messing up Bookends).

10

Indeed! :slight_smile:

Locking a group does not inhibit moving or sending it to the trash. However, it will not let you empty the trash unless you explicity tell it to.

If not, then I guess I just mark the folder in red - not to be touched - and create replicants of all the files in a different folder that can be moved around with impunity (ie without messing up Bookends).

You should stop and (re)read the Importing and Indexing > Indexing and the filesystem section before yoou start replicating. These things are not done with impunity when it comes to indexing.

11

This time, I think the problem is that I was too loose in the way I described what I thought I could do, not that I had not understood the manual. What I am proposing to do is the following:

(1) Create a new DT database, containing two Groups, “DO NOT TOUCH” and “PDFs”.

(2) Inside the DO NOT TOUCH group I place an Indexed copy of my attachments folder (in the file system) and mark it as Locked. The DO NOT TOUCH group is just psychological, to reduce the chance of human error causing a silly accident.

(3) I then go into the attachments Group and Replicate all the individual linked PDFs, taking care NOT to replicate any Groups, into the PDFs folder. I assume the key is not to replicate the Indexed group itself or any sub-Groups of it.)

(4) Provided I never rename any of the replicants in the PDFs folder, it would then be possible to move them around, say to organise them into different groups, without doing anything that interferes with Bookends. I can then create links with other documents in DTP, including references imported from Bookends.

Does that organization make sense, or have I still not understood something? I am assuming that the memory cost of creating a replicant is small.

Thanks for your great patience over this.

12

You’re welcome!

I am assuming that the memory cost of creating a replicant is small.

Creating a replicant consumes no extra space.
Replicating indexed items also does not create any new files in the Finder.

Yes, you can index a Finder folder, then create replicants in another non-indexed group, and organize those replicants however you’d like.

However, I would agree that replicating indexed groups and reorganizing could lead to unexpected results in this scenario.