Sync database creates lots of duplicates with indexed folder on apple icloud

Hi, I try to find a solution to have a database synced on my Mac, MacBook, iPad, iPhone.
I only use an indexed folder to be able to access the files from without DT3.
This indexed folder is on my Apple iCloud.
The database is synced with iCloud(cloudKit) and the database is synced into each device.
I try by this to spare diskspace as the indexed folder is quite big (knowing that apple icloud is copying on each connected devices, this goal is probably utopic)
When I sync with DT3, I see that several subfolders are duplicated.
Then I cannot find which subfolder is the good one and if deleting one is not deleting files I want to keep.
Is it a question of apple iCloud not synchronising well between the cloud and the devices ?
I’m quite desperate and it is questioning my use of DT3 if I don’t find a reliable sync solution.
Thanks for your help

What’s the path of these indexed folders? Are these folders used by other apps too?

Why not simply import the files and use shallow sync on the mobile devices? I think that this would really save space (on the mobile devices).

Also, if you’re telling iCloud to “optimize” storage, it could remove files at will from the local device(s), which might cause problems.

iCloud is known to have problems with sync (I’m experiencing this every day with reminders that don’t get reliably removed from all devices when I remove them from one of them). There are other options like Dropbox that seem to be better. Or use Bonjour if you don’t need data at the same time locally and remotely.

2 Likes

I experience this too. Even worse is that sometimes reminders are missing but shouldn’t or scheduled reminders get the wrong date after marking them as done. Maybe I’m using too many devices/computers and boot volumes.

2 Likes

it is really awful: no trust in a system doing this.
I am trying now with my NAS

  1. without any DT3 sync (one DB on macbook indexing folder on nas, another db on mac indexing same folder on nas)
    I really need to access these data from my macbook AND my mac and I still hope I can use DT3 for this… but seemingly not possible… still searching…

Try Bonjour method. Fastest and most reliable. Instructions to setup in the outstanding “DEVONthink Manual”

Fyi in addition to Bonjour i use a WebDav server on my Synology NAS. Works. More complicated to set up but works (slower than Bonjour, though)

Perhaps get in touch with Apple about this distrust issue.

2 Likes

here another screencopy but of the finder

  1. you see that
    • folder “ProcĂ©dure” has been duplicated (with a “-1” for the duplicate)
    • there is a cloud icon indicating not synced locally (yet)
    • “ProcĂ©dure” folder contains all the files
    • “ProcĂ©dure-1” is empty

  1. I renamed folder “PROJETS” into "PROJETS-PARA
    • both folder are still in the finder
    • the newly renamed “PROJETS-PARA” has a cloud icon
    • “PROJETS-PARA” contains the file
    • “PROJETS” is empty, except for a folder “ProcĂ©dure-1” that is empty and it was the original place of “Procedure” that I moved to the parent folder

How is that related to DT? And now I’m at a complete loss understanding if we’re talking about folders “in iCloud” (where exactly - Cloud Drive) or somewhere else like on a NAS (which you mentioned in your previous post).

Please take some time to describe the situation in detail and clearly.

What kind of sync locations do you actually use? Only CloudKit? In addition, what’s the path of the folders and is it used by other apps?

did you read the post? it is about syncing an indexed file with DT3 with indexed folder on apple icloud.
This generates duplicate through interaction of icloud syncing and DT3 syncing (cloudkit)

yes only Cloudkit (thought it was the best way)

I have a folder on icloud; it is indexed in a DT3 database
This database is synced with DT3(Cloudkit)
This generates duplicate folders and files totally uncontrolable

Well, that neither explains the role of WebDAV. Nor why you don’t use Bonjour for syncing. Nor did you answer my earlier question if you told iCloud to “optimize” the storage. Or why you even index the items in the first place given that they have to be locally available anyway (ie there’s no space to be saved by indexing or importing in DT).

So yes, I did read your posts. But they omit details.

3 Likes

@ChrisVH, perhaps I can clarify something here. There is some time I don’t see that behavior in iCloud Drive because I don’t use it too much, but I’ve suffered the “-1” duplication issue and it is not related to DT.

When iCloud Drive synchronizes some files in a slow network, let’s say file.txt, it downloads the file and names it file-1.txt. Once it has both files in filesystem, deleted original file.txt and renames file-1.txt as file.txt. Same with folder names.

What happens if someone is accessing file-1.txt that suddenly appeared (like DT, but not only DT)? Well, then macOS gets confused, or there is a not yet solved bug, the mix of both files is not done, and you end with two same file/folder. Then if DT sees those files, it indexes them.

Why do they do that so botched way? I don’t know, but previous way to do it was even worse, because it added a number, like file1.txt, and if file name ended in number, like contacts_2022.txt, you will end having two identical files with names contacts_2022.txt and contacts_2023.txt, and contacts_2024.txt and contacts_2025.txt… demonstrating one more time that Apple is paying bananas and so getting monkeys as developers. I still wonder how they could end mathematically adding the number in the file name.

But there is even worse. Sometimes it happens “the three (or more) body problem”. It seems Dropbox has resolved it (not completely sure), but surely iCloud Drive has not. Basically, if a file in cloud is accessed by three or more programs at same time, the behavior is undefined, and you get at least duplicated files or a corrupted file.

And there is no way to resolve it because it must be solved by Apple.

You could minimize this problem if you disable storage optimization (as @chrillek surely will recommend) and download all files on disk.

3 Likes

Do you use the same versions of macOS and DEVONthink 3 on all computers? Might a third-party app affect these indexed folder?

That’s quite simple in some programming languages (eg JavaScript) that try to be „smart“ and also reuse the + operator. When they „think“ that 2022 is a string, + 1 results in 20221 (concatenation). When they „think“ that 2022 is a number, + 1 results in 2023.

So the programmer needs to be smarter than the language …

2 Likes

At least DEVONthink doesn’t think that way :slight_smile:

2 Likes

Probably because it’s not so fond of JavaScript.

And of course, to do a cloud sync engine, the best language is JS. :joy: And instead to integrate in kernel the file update going under userland to avoid blocks etc, change it to do it in a Finder complement. MS has a driver model that work in userland but performs tasks at kernel level.

Well, Apple certainly does not have the best reputation concerning software development. Nor concerning bug fixes and their handing of security fixes. But hey, who’s going to complain as long as they get out shiny new icons and emojis every year.

3 Likes