Support: Frustration over Synchronization and Status of databases

Agreed, a rock-stable sync would be best :slight_smile:

As I wrote from time to time in this forum, the Sync is the heart of DT / DTTG and needs to work under any circumstances. Any issue with the Sync should start a BIG red blinking sign in the middle of the screen :slight_smile:

And any fix for a bad situation is still better than needing to resync all devices from a remote Sync location again.
The URL commands cannot fix it right now.

Your logs are showing issues with your WebDAV server - connections and other issues. That wouldnā€™t be a sync issue :slight_smile:

No, I explained that in my initial posting.

I migrated my WebDAV Sync location to a new provider and server, and those messages come that the time where I was configuring Apache and the SSL certs.

In that time, I did not add or delete items from the databases.

The WebDAV issues are solved since days.

The differences between the devices where definitely pre-existing to my server migration!

But even if WebDAV had a problem in the time when I added or deleted items, there should be a way to fix the sync state (items and item counts) for all devices.
I mean, this is what the support URL commands should do, right?

Or simply a new sync from each device and after that, another round of resync to get them into the same state.

But this does not work too.

Are you trying to sync on or off your network?

there should be a way to fix the sync state (items and item counts) for all devices.
I mean, this is what the support URL commands should do, right?

The sync reset command should initiate a full resync, not fix the sync state.

The Sync location is on a virtual server at a provider.

I cannot see a ā€œsync resetā€ command, did you mean ā€œresyncā€?
I used that on the iDevices!

Yes. And as I said, that wonā€™t fix the sync state.
If you have a connection or permissions issue on the WebDAV server, even forcing a full resync wouldnā€™t resolve the issue.

There can always be temporary problems, at any level.

The sync needs some way to fix any differences that may have introduced for whatever reason.

I tried to explain common techniques for any sync mechanism above.

Each device needs a way to force upload itā€™s items and index.
After that, the Sync location should have all items from all devices.
And then, just sync again to get all devices to that same state.

The Sync mechanism is similar to what high available cluster nodes need to solve, or file systems when writing blocks:

There can be any amount of difference between an index of things that have been done, are currently running and should be running next, and then the actual data can also differ from this index (esp. with multiple writers).

Those are common problems in many areas, but they are solvable and solved!

A cluster can bring nodes to resync forcefully, if some index number for commands was lost.

A file systems may use amount of additional checks, copy-on-write and checksums for blocks, there are many techniques used - see journaling filesystems, cluster filesystems or ZFS - or may finally require an ā€œfsckā€ run to bring index and items (blocks) up to the same state.

What I am trying to say here is, that there are happening errors, failures and mistakes at any level and in an unpredictable manner. Minor and major, constantly.

An unstable WebDAV server (and mine was and now is again as rock-stable as possible, after the migration - and I did not upload in that time) is just one example.

And any such sync mechanism needs to cope with this, being able to recover, resync, and finally offer a mechanism to recover totally. Without needing to recreate from scratch or backup.

Oddly Iā€™ve not had a sync problem ever. Have used Dropbox to sync my Macs and phone/iPad pretty much since they revamped DTTG. Has been rock solid since day one.

4 Likes

@tja We are currently investigating and the higher numbers on the iOS devices come from DEVONthink To Go not properly filtering records that are marked for deletion but have not yet been removed completely. As the full-text index is slow in deleting records (itā€™s based on SQLite FTS5), records are deleted in a two stage process. We are now testing an improved item counting that properly handles this.

4 Likes

so, to jump to a conclusion that might be erroneous but perhaps ameliorate the big concerns feared on this thread ā€¦ the issue is ā€œcountā€ not loss of files in the sync. true?

At least in my limited experience: yes, the data is there, but the count is wrong. Donā€™t know about the OP, though.

1 Like

This seems to be the case so far indeed.

1 Like

Iā€™ve also never had a big issue with sync, and I use the dreaded CloudKit. I occasionally drop a connection for a few mins due to Appleā€™s poor service, but DT/DTTG always wait it out and get back in sync when they can. The one time I messed up my sync, it was my fault (and unfortunately was exacerbated by how quick CloudKit was running that day :joy:).

For fun Iā€™ve just looked at my database total counts and I can see that the numbers are off on DTTG. But to confirm what others have said, the group counts inside the database all match across devices and Iā€™m not concerned that any files are actually missing - the total item count is just wrong.

I donā€™t know how WebDAV works so ignore this is if doesnā€™t work like that, but has the actual sync store been erased and then recreated from ā€œthe one source of truthā€ (which I assume is the Mac)? I ask because when I messed up my database, the sync issues werenā€™t fixed until I erased the whole ā€œcloudā€ version of the database, and created a new sync store that was correct. OP mentions deleting the DTTG app, but my understanding was that deleting the app doesnā€™t erase the sync store (at least in the case of CloudKit?).

2 Likes

I donā€™t see why that would be necessary at all. In my experience, WebDAV works reliably. And the numbers quoted here for the WebDAV sync store are totally unrelated to any database objects, so itā€™s not sensible to think about them.

1 Like

I donā€™t think thatā€™s what I meant, but actually Iā€™ve re-read OPā€™s comments and donā€™t think what I meant helps anyway.

My thinking was that when sync went wrong for me, it was the sync store that was ā€œbadā€, not DT (=my Mac). I fixed DT, but the changes didnā€™t push through to the sync store. So the version of the database that was getting pushed out to DTTG was the bad version from the sync store, not the correct version from DT. Because OP was reporting the same numbers on their iPads I wondered if they were downloading a bad version of the sync store. But actually Iā€™ve just realised the version on their iPhone is different, so itā€™s not because the sync store is wrong (if the sync store was wrong all 3 devices should have the same errors).

So my post was no help at all on this occasion!

Mind you - I still think itā€™s worth noting that deleting DTTG from your device does not delete a sync store, so perhaps doesnā€™t do much for sync issues.

1 Like

Update: The issue seems to be caused, as explained above, by incorrectly filtering already deleted items. This issue is not caused by sync and a mismatch does not necessarily indicate a sync issue.

We are now running an internal test on our fix and will provide our beta testers with a new build as soon as we find our changes reliable.

5 Likes

Oh great, that is the best news ever!

Many thanks already!

I was hesitating to use DT / DTTG at all in the last weekā€¦