Sync generating lots of "copy" files with DT 3.5

DEVONthink
1

I’m seeing a lot of “copy” duplicate items appearing in DT as a result of sync.

Setup
– Two MacBook Pros with DT 3.5.1
– iPhone and iPad with DTTG 2.7.7
– Sync via iCloud
– Sync via Bonjour, with one MacBook Pro acting as host on local network.

Note that I have two sync path: one via iCloud, one via Bonjour (local network). When I enable only one of these paths only, sync behaves normally. When I enable both paths, any edited RTF note ends up generating lots of extra “copy” versions as a result of sync conflicts (pretty much one for every set of edits). However, I only ever edit the documents in question on one computer, so this is not a genuine conflict caused by overlapping edits on different computers. It seems to be a result of sync occurring via more than one sync path.

I’m not entirely surprised by this, to be honest, as getting multiple paths working can be tricky, but I can’t find anything in the UI, help, or anywhere else that says to avoid multiple sync stores. (If it’s a bad thing I would have expected to only allow enabling one sync store at a time).

(Incidentally, I’m doing this since iCloud sync is so clunky and slow. I would use Dropbox sync but for the problems with it silently stopping sync dead.)

2

Is the option to duplicate files (see Preferences > Sync) in case of conflicts enabled on any or even all computers/devices? As you edit documents only on one device, disabling this option should fix this.

The most likely cause is the asynchronous workflow of iCloud. Bonjour updates all devices synchronously and almost immediately whereas iCloud might cause delays and deliver old data first before delivering the latest data and this causes probably the conflicts.

However, we’re working already on an alternative.

3

Yes, the “duplicate files” option is on, and I realize that this is what is creating the “copy” files. And I also agree this is likely a result of the race condition between iCloud and Local Network. I left this option on (everywhere) so I could monitor for sync conflicts explicitly (after prior sync issues).

My concern is how to interpret the resulting “copy” files. Since I’m only editing on one device, I assume that the cause is more or less as follows:

(1) I edit a file “X” to make an “Xv1” version on computer A, and it immediately (more or less) syncs to computer B via Local Network. At the same time, Xv1 starts wandering through the iCloud path (slowly).

(2) Meantime, I edit “X” again to make “Xv2” version on A, and it again syncs via Local Network to deliver Xv2 to B, thus B now has “Xv2” version of “X” (and has discarded the old “Xv1”).

(3) Finally, iCloud delivers “Xv1” to computer B, but B sees that it has “Xv2”, which is newer, and so creates a renamed “X copy” file containing “Xv1”, while “X” is left as-is since it already has the (newer) “Xv2”.

(btw, I think a better way to name these would be “older”, not “copy”, as “copy” doesnt really hint about which the system thinks is newer vs older, though I realize in some cases its not clear which really is the older data.)

My concern here is really how much I can rely on the way computer B resolves the conflicts between the different versions when presented with stale versions via slower sync paths. If I disable “Duplicate Documents” as the Conflict option when syncing (which I would like to do), can I rely on the sync engine to always resolve the latest version correctly? Or should I continue to be paranoid and manually double-check the “copy” files before deleting them?

(The alternative, which I’m doing at present, is to carefully switch from Local Network to iCloud and back as necessary, taking care to allow enough time for data to sync as needed, which is far from ideal.)

I realize of course that in the worst case of editing the same file on different computers at the same time there will inevitably be a conflict, but that’s understood (and yes, as you might have guessed, I’m pretty familiar with sync and conflict resolutions, as that’s part of my day job!).

4

Yes, the latest version is always preferred in this case.

5

Thanks! However, I’ve been doing some more testing, and I’m still getting results I don’t understand.

I carefully adjusted my sync setup as follows:

– All devices (2x MacBooks, iPad, iPhone) use Local Network sync ONLY.
– One MacBook Pro designated as master (using Bonjour).
– This same MacBook Pro is also doing sync to iCloud (but it’s the only one, none of the others have anything but Local Network sync).
– Setup was done 2 weeks ago, all copy conflicts manually resolved and removed.

Now, my take on this topology is that there should never be conflicts if I edit on one computer, since there is only a single sync path (hub and spoke). HOWEVER, today I worked on editing an RTF document (rich text note) on the master MacBook Pro (and only that computer). The other MacBook Pro was syncing via local network.

After 2 hours of editing I now find I have 15 extra “copy” files of the document I was editing (on both the MacBook computers). Yes, I have “Duplicate Documents” enabled under Sync on both computers, but how can there BE a conflict when I’m editing on a single computer (and trust me, I am)??

This looks more like an issue with the sync engine to me. I edit on X, then sync to Y, and X sees a conflict with Y?

6

Is this an imported or indexed item? And what kind of item? A screenshot of the sync settings of both Macs would be useful.

7

It’s a “Rich Text” note created within DT and edited entirely within DT on my “main” Mac.

I’ve attached screenshots of both the “main” Mac (which has iCloud sync on, with Bonjour for local network), and the “second” Mac (which has local network sync only). As noted, I only edit on the “main” Mac. Also, the sync settings were established about 3 weeks ago and I took care to remove all extra sync copy files after I did a complete sync between the computers.

DT Sync Settings - Main Mac (with Bonjour+iCloud Sync)
DT Sync Settings - Main Mac (with Bonjour+iCloud Sync)1438×1276 225 KB
DT Sync Settings - Second Mac (with Bonjour Sync Only)
DT Sync Settings - Second Mac (with Bonjour Sync Only)1442×1278 223 KB

8

Note: Using punctuation like braces in a filename is not a wise idea. Spaces, hyphens, underscores, and periods are the best options if punctuation is to be used.

9

What’s the purpose of the iCloud sync on the first Mac as the second Mac doesn’t use this, at least according to the screenshots?

In case of multiple sync locations especially iCloud might cause such issues as it works asynchronously, meaning that other computers/devices might receive the latest changes first via a faster sync location and then receive older changes via iCloud before receiving the latest ones too. And this scenario might indeed cause duplicates due to conflicts.

One workaround is to use the latest document on all computers/devices, another one to use only one sync location or not to use iCloud. In addition, please ensure that all computers/devices uses the latest version of DEVONthink (To Go).

10

As I made clear, the computers are ONLY syncing via local network (Bonjour). The “fossil” iCloud sync on the one Mac is from when I was using iCloud only (a long story), but since it ONLY syncs from that one Mac to iCloud there are no multiple paths to sync with other devices. So I am using only one sync location, and I do have the latest versions of DT and DTTG.

This looks to me like a bug in the sync engine. I can repro it easily, just by doing edits on a document and watching as extra copies of files accumulate, even though all edits are on one computer, and the ONLY sync is to one other computer, and the only sync path is via local network.

11

And no iOS devices are synchronized via local network?

12

I’ve turned sync off on the iOS devices so I can isolate the issue. It really is the one sync to the second Mac. If I disable sync on that Mac … no “copy” files regardless of the edits I make. If I enable sync on the second Mac, and do NOTHING in DT except leave it running, then up pop lots of “copy” files whenever I edit a Rich Text item.

13

Thanks! I’ll try to reproduce this but actually I edit a lot of rich text files and duplicating is enabled on multiple computers but I have never experienced this so far. Only difference is that I prefer sync stores but at least theoretically this shouldn’t matter as the same core handles this. We’ll see…

14

It’s indeed an issue of the Bonjour sync, the next release will fix this.

15

Ah thank you for confirming this. Glad I was able to help find the issue.

1 Like
16

Not sure if I can talk about my issue here or in another thread.
My DT Pro 3 also creates duplicates of files (files created in DT, not indexed files) when I open my DT on a new terminal.
The thing is, I don’t sync through Bonjour or anything. Actually, I have my databases on a single external drive which I use between 3 terminals.
Sometimes I end up with 3 copies because I have opened the document/DT 3 on 3 different terminals.
What should I be looking for to solve this issue?
Do I need to close my databases on each terminal before closing DT and ejecting the drives, or is there something in the sync store or preferences that needs to be adjusted?

17

I have never used DEVONthink as you describe. I can only guess that it is important to close all DEVONthink databases, then properly eject the drive before moving the drive and re-opening the databases on another machine.

Perhaps coincidental, but you mention using three machines and you report you sometimes have three copies. A clue.

You also are using an out-of-date version of DEVONthink. Version 3.9.1 is now available.

18

I’m wondering if the duplicates/copies are generating because I have both a sync cloud as well as running DT3Pro on 3 systems through a single external SSD?
I created the DT sync file before I started using the single SSD as a common database source. I forgot to delete the sync file.
But, do I still need it? Will anything happen if I delete the sync file from the three systems? I am pretty happy using a single SSD to run DT on all 3 systems so I don’t see myself changing to a new syncing method any time soon.
Do any of you think deleting the sync file might stop the duplicates/copies showing up?
Only asking first to see if deleting the sync file is safe or that it wont affect my current method.

19

Actually, I have my databases on a single external drive which I use between 3 terminals.

Please clarify the setup here. Is the external drive attached to a file server on your network? Or are you ejecting and connecting it to each machine as you use it?

Sometimes I end up with 3 copies because I have opened the document/DT 3 on 3 different terminals.
What should I be looking for to solve this issue?

  • Do the files have copy appended to the name?
    • If so, that from a sync conflict.
20

I have the databases stored on a single SSD, form which I open the program on each device. There is no server or network. The app on each device is directed to the file path on the SSD.

Yes, the files have the word COPY appended.
At first I thought it was only happening to indexed files, eg from Obsidian, but then I noticed it occurring on non-indexed files as well.

So if it is a “sync” issue, what is the solution, as I am not syncing the application with others?
Should I delete the sync store from the preferences? I don’t actively “sync” the devices, so do you think that is the source of the problem?