Sync duplicates entire database

Howdy,

i have an imac and a macbook. I barely use the macbook.
I switched to cloudkit (from cloudme) on the imac and synced it to ios devices. has been working for a few weeks/months now (basically switched right when the feature was released).

set up the macbook a few days ago with the new cloudkit by deactivating cloudme and adding cloudkit. it synced. syccess, so i thought.

today i noticed that my database has a lot of files in it, a lot of them having been deleted or reshuffled ages ago. the insane thing? i did the re-organisation of the database last year in summer ON MY MACBOOK. somehow adding my macbook into the sync mix has completely messed up the sync.

i have no idea where they are coming from to begin with. even files i deleted in july are now back in the database. i have now an additional hundreds of groups and thousands of files.

how do i fix this? i am really not keen on renaming thousands of files again. and why would this happen anyways?

BR

Was the database on this computer actually up to date, especially as you rarely use it? As itā€™s a relatively new sync store, the only possible source is actually an outdated database.

The first time an existing database is synchronized with a sync store containing the database already, the local and the remote databases are merged.

the datase on the macbook was not up to date. but neither version had the groups or files in them that are now back again.

i had groups in the root of the db. on the macbook in july 2020 i created 3 new groups and removed all other root groups. this has been the state of the macbook. the macbook was synced in january 2021 last time. the imac was synced all the time and only had the 3 groups in it.

DT basically replayed 8 months old stale state ā€¦

@cgrunenberg question remains, how do i get rid of the stale content

There is no automatic way to remove the content.
Restoring from a current backup would likely be the quickest.

why was stale data replayed?

@BLUEFROG @cgrunenberg can you please write out a todo list on how to properly restore the database, esp. with sync in mind? i have a backup from 2 weeks ago but when i open it (have deleted the old one already) it says the db isnā€™t uploaded yet. so i guess sync is missing some information

Is it possible that DT got a chance to sync with the old sync store before you deactivated it? I think I would put back a backup, clean the sync store and start over with sync, myself.

Was the database on the MacBook up to date or outdated?

sync the old state to cloudme? probably but the files were gone on both imac and macbook (as in, not visible in the DB)

fun, reuploading 100gb and restoring it on 4 devices ā€¦

outdated

i have a backup from 2 weeks ago but when i open it (have deleted the old one already) it says the db isnā€™t uploaded yet.

If it says the database hasnā€™t been uploaded yet, the database hasnā€™t synced to that sync location. If it had, it would report or needs to merge.

please advise on the proper procedure

Then thatā€™s the cause. The local and the remote databases were merged during the first synchronization as the new sync store (CloudKit) was not used before on the MacBook and therefore there was no local sync history. Deleting the outdated database and importing the latest one from the sync store would have been a better option.

Did you delete/replace the database in the Finder or delete the database in DEVONthink? In the second case thereā€™s no more local sync history afterwards, merging it again with the existing sync store would cause the initial issue again.

Cleaning the sync store is unfortunately the only remaining option, afterwards import the database from the sync store to the other devices.

found a bug on osx: i deactivated the sync of the DB and DT keeps syncing.

merged what though? i really, really do not understand this.
there is no sane scenario as far as i can see that a deleted file from a macbook that has been deleted 10 months ago shows up in a merge. the file should not even be available to DT anymore in any capacity. it was outdated by 2 months, not 8 months. the file was deleted on the macbook. this makes no sense. saying ā€˜mergeā€™ and ā€˜outdatedā€™ does not explain (at all) why i am in this terrible situation now.

great, i will ā€˜clean the sync storeā€™ though i bet 100 euros i mess this up somehow too. i wish you guys would have implemented a good UX in DT3 around sync, this is genuinely unacceptable. i should get a free upgrade to DT4 or sth with how much of my time you are wasting on sync.

Yeah, thatā€™s kinda not how life works. The alternative approach would be to be grateful for the free support you are receiving from the publisher of the application. That you do not understand what has happened does not necessarily locate the problem at DEVONtechā€™s end - although Iā€™m happy to admit that Iā€™m not convinced I understand what has happened either.

My suggestion would be to focus on solving the problem and getting you back up and working. Criss has suggested cleaning the sync store; you previously said you had deleted the ā€œoldā€ database, but Iā€™m not sure from what you wrote on which device(s) you have deleted it. @cgrunenberg would it be advisable in this case to delete the database from every device before cleaning the sync store, putting back a backup and then re-syncing?
@bosie obviously you could copy changes which you know you have made since the backup to a new database before deleting every copy of the ā€œoldā€ database; that way you could drag those files back into the restored database once you have finished.

2 Likes

That was basically my suggestion.

Thanks; that was how I interpreted it, but was unsure whether that was what you meant.

It is what I use to do when messing with betasā€¦ Take in consideration that sync database is not a database but a journal. If one DT instance finds a file that is not in the journal, it uploads it and then all is synchronized.

For example, you have Mac A not synced time ago, you delete the sync DB from Mac B and perhaps rebuild a new local database. Then switches on Mac A and, voila!, you nave ā€œnewā€ files in Mac A that arenā€™t in sync database. I think is what has happened to @bosie. But that is not a DT/DTTG problem, it happens because not knowing how really DT/DTTG works.

There was no transactional data for a CloudKit sync on the MacBook. Therefore DEVONthink on the MacBook didnā€™t have any sync data from the iMacā€™s syncing to compare with the MacBook. The MacBook saw there was a copy of the database in the sync location and logically merged with it.