Support: Frustration over Synchronization and Status of databases

[tja]

I write this in DTTG, even as I am not sure in how far this also applies to DT.

Currently, I am using mostly 3 databases and the Global Inbox, on my Mac and on 2 iPads. The iPhone does not download all content.

After some synchronization problems, partly because of a full encrypted container and / or becasue of SpotLight being required for some time, I set up all devices newly … and IIRC all showed the same status in regards to items per database.

I checked this again … and what to say:

ALL of them differ over ALL devices … again.

Out of frustration, I ran the “fix” and the"resync" URL commands from the help menu on both iPads.
This did not fix the situation.

I really cannot understand this.
Something in the sync process is just simply broken!

Each device should have an index about all items from each database.
If any devices differs from what is synced to my WebDAV location, it should either update it’s local index or the remote index.
If all devices do the same, each should have the same status.

This seems to be a simple thing to implement, you could even enhance this with an Id number for such an index, so that a devices notices that there was a jump in this Id number, it just downloads the remote index and Id to be in sync again.

And then, of course upload all remotely missing and download all lcoally missing files.

So, why does that not work? Constantly! Over years! Again and again!

I don’t even know what to do, as I have no idea which items may be missing where.

PLEASE fix this!

The synchronization is BY FAR the most important part of DT and DTTG!

P.S. It would be nice to see more details in the information page of DTTG, which does only show size and number of items, where DT offers more about number of items in different categories.

And sorry, but this is really frustrating … I already redid all databases on all devices two times! And it seems, I need to repeat - somehow - a third time.

[BLUEFROG]

This seems to be a simple thing to implement

Things often seem to be one way when they’re actually another.
This is actually compounded by the fact our sync engine supports more than one sync method.
Then it’s further compounded by factors beyond our control, e.g., poor network quality, slow or unresponsive remote servers, bandwidth throttling, firewalls, etc.

I already redid all databases on all devices two times!

What do you mean you “redid all databases” ?

• IIRC, you are indexing cloud-synced locations. Correct?
  • If so, are you manually updating the groups in the database?
• What is reported in DEVONthink’s Window > Log?
• Is there a triangle on the sync icon in DEVONthink To Go’s bottom toolbar? If so, tap it. What is reported?

[tja]

Hello Bluefrog,
with “redid” I meant that I deleted DTTG from my devices and resynced all datbases from the remote WebDAV Sync location.
At that moment, all instances of all databases for all devices where the same.
I did this once before I had DT and once after I had DT.

And of course I expect this to stay this way and not see different item count for all databases on all devices some months later

I don’t understand what you mean with “manually update the groups”.
I add items from various devices, of course.
And the databases should be synced fine to other devices, right?

The log window does not help, as this is did not just happen - I think that happened over some months. I never ever saw some related entry in the log, beside temporary sync messages when there are a network problem are when I migrated the Sync location.

No, there is no triangle, but yes when I had sync problems or was testing with the new Sync location, there was - which I then noticed and fixed.

Whatever the temporary problems where, the devices should still sync fine afterwards, right?

Or at least the “resync” URL command should have done so!

Many thanks for any help and esp. for a fix

[tja]

I give a concrete example:

One database shows 2695 items in DT, the same number on one iPad and 2725 on the other iPad.

So, either some index is broken and those 30 items simply do not exist - in this case, the device should be able to download the index from the Sync location and everything is fine.

Otherwise, when this device is right and has 30 more items than the other decices, it seems that those items would also be missing from the Sync location!
But in this case, the device should be able to upload the items and update the index, so that the other devices can get the current status too!

What else could be?

There could be 30 items more on one device and the Sync location also knows them - but in this case, a simple sync should update those devices.

So, what is happening here und what is missing?!?

[tja]

For another database, the item counts are 7406, 7446 and 5788 (DT!)

This database is indexed and the “find” counts shows this:

(base) tja@mini:/Volumes/CRYPTOMATOR_ONEDRIVE$ find . | wc -l
7036

So, about 7000+ items should be right.
I did not yet try to re-index DT on that folder - I will try in a moment …
But this still does not explain the differences for the iPads.

What else can I say … this is horrible, as you sure will understand.

[tja]

I re-indexed the indexed database in DT and now see 7406 items.
I am syncing to the iPads right now to see if that fixes them

[tja]

Let me add one thought:

For me, all devices should compare their items count with the sync location and if that is not the same after syncing, there should be some BIG warning, flashing in the middle of the screen in red.

Then, there should be a way to force uploading or downloading the index and the items based on some sync-Id that get’s constantly updated for any sync operation.

A device knows if it has more or different items or if there are missing items.

So, there needs to be way to re-create the index from content, both locally and also from the remote Sync location.
For this, each device should be able to force upload all local items that are missing remotely and update the index accordingly.
After doing that with all devices, a new sync should update any and all devices to the new state from the Sync location - and all should have the same index and the same items.

And the “resync” URL command is described as doing something like this, but it clearly does not work.

OK, I am silent now
Please forgive … I’m just frustrated.

[tja]

I just wrote a wrong text because I compared the wrong databases.
Here is the new situation:

One iPad finished and is now showing 9024 items

So, much more than DT and also much more than guessed from the indexed directory.

Very very strange!

Will edit this post for the second iPad too:

The second iPad finished with the sync and is now showing 9064 items for this database.

Close to the first iPad, but both different to the 7406 from DT.

[BLUEFROG]

No worries! You are stating the situation as well as your thoughts and opinions. You haven’t lashed out at anyone, not even Development, so you’re well within the boundaries of acceptable behavior

[tja]

I updated above for the second iPad.

Also, I counted files and directories in the WebDAV location … but that seems strange too:

root@localhost:/var/www/webdav/S4Y.dtCloud# for d in * ; do echo “$( find ${d} | wc -l ) ${d}”; done
1008 516147c0dc6347142627f2c366966fc854a29514ed175c67465c9cfe49378edc
815 73161ebfa2ea93a7045848335e75987104884dc4e3c82dd42ba59439339377d8
1681 87c62b1edca53f854feb4b1cb61d392637898af3f213d32644fc55fb3cc7395b
3625 f54963f536eaa37d410d20a8f1a3f21e97b32028835ddf7d6330ef478bae6c71
948 inbox

None of them shows over 7000 or even 9000 items!

I would have expect the numbers to be similar …
Or, we see part of the problems I have.

[chrillek]

These files on your WebDAV device are the blobs used for syncing (unless you’ve really stored your databases on the WebDAV server?). They have nothing (nothing at all, really) to do with your original databases. They’re just “something” that DT uses for its sync purposes. So looking at them might be entertaining (or not), but counting items there does not tell you anything about the sync between your devices nor about the number of items in your databases on them.

[tja]

Thanks.

But how can I fix the situation?

Right now, I have no idea which device has the right amount of items, or if the WebDAV Sync location is right. For each of the databases.

I’m totally stuck in uncertainy.

[cgrunenberg]

Screenshots showing the discrepancies would be useful.

[chrillek]

I can actually contribute a similar mismatch between the object counts on DT and DTTG.

If you sum up the elements of the groups below “Dokumente” and the five documents not in any group you get exactly 188 for the “Dokumente group”.

However, on DTTG (iPhone, current iOS and of course current DTTG), I get 186 for the “Dokumente” group:

Now, the individual document counts for the groups are exactly the same on DTTG as on DT, as is the number of documents not in any group. But still, DTTG shows another total count than DT.

Note I took overlapping screenshots so that the first entry on the second one is the same as the last on the first one and the third one begins with the last group seen in the second one. Sorry for that, but I couldn’t find a way to get a more compact display on the iPhone.

Not that this worries me, but it seems a bit weird – although iOS is not a fully blown operating system, arithmetic should work the same there as on macOS, shouldn’t it?

[cgrunenberg]

@chrillek

But still, DTTG shows another total count than DT.

Are there any replicated documents in the visible groups?

[chrillek]

Nope. I just made a smart group for this database and it shows no replicants at all. And there are no duplicates either.

And: This group “Dokumente” is the only one in the database that shows different object counts on both platforms.
The overall # of objects in the database is 717 on macOS and 716 on iOS – so here the difference is one whereas for the only differing group it is two.
Doing the addition myself on macOS, I also get 717. So that might be correct. Also, the number of tags for this database is identical for both platforms.

I wonder if that might be an off-by-one error somewhere (actually, two of them ;-))

Edit I checked all my databases now and found another one with discrepancies. In that case, there are items in two groups on iOS missing on macOS. Both installations sync just fine, though, using my own WebDAV server (i.e. I don’t see any errors). Verify and repair on both sides didn’t change anything, nor did a manual sync. The items missing from macOS are all bookmarks.

[cgrunenberg]

@chrillek

And: This group “Dokumente” is the only one in the database that shows different object counts on both platforms.

According to the screenshots the numbers on the Mac are right. In DEVONthink To Go the numbers of the subgroups of “Dokumente” are also right but not the total count of “Dokumente”.

Did you already try the fix or delete-caches service tasks, see UTILITY URL COMMANDS in documentation?

[chrillek]

@cgrunenberg

Did you already try the fix or delete-caches service tasks, see UTILITY URL COMMANDS in documentation?

fix fixed the count of elements in “Dokumente”, but there’s still a difference of one in the object count on both platforms(717 on macOS, 716 on iOS). For the other database, the number of objects on iOS is still bigger than on macOS (which is not really surprising).

delete-caches didn’t change anything further.

So there seem to be two issues:

• For database A: A different item count that cannot really be explained, since the objects in the groups are identical on both platforms and there are not objects outside of groups.
• For database B: Items on iOS that don’t get synchronized to macOS. That’s over WebDAV. Face palm: That is not an issue, since I excluded this database from sync on macOS. Well, most of the time the problem is in front of the screen.

[cgrunenberg]

@chrillek

fix fixed the count of elements in “Dokumente”, but there’s still a difference of one in the object count on both platforms(717 on macOS, 716 on iOS).

Screenshots would be great in this case too, thanks!

[chrillek]

Here you go: