For reference, I unchecked all databases except one. I limited connections. Above 4 connections and I get errors and apparently no transfers. At 4 connections, I transfer bursts of dozens of files. Below 4 connections, I get the same duration bursts, but fewer files transfer. 4 seems to be the sweet spot.
Before I reduced connections and databases, as best I could tell, DevonThink wasn’t actually transferring any files at all and was producing a steady stream of errors. (attached below)
With one database and 4 connections, I see the files transfer in the spurts. The transfers stop coinciding with 1 or 2 additional error in the log. The spurts are roughly 2 minutes apart.
To add some further observations: After having synced 4 DBs (2 small, 2 around 20 GBs) with CloudKit I started to sync them one-by-one on an iOS device. The 2 small DBs synched without problems. The first large one stalled after some minutes, showing no progress in the sync progress bar for about an hour. After force quitting and restarting the iOS app the sync continued until it stalled again. I had to do this three times for a complete sync. The DB seems to be complete on iOS now.
During my analysis I found a potential bug, though I can not say whether this is new behavior: on macOS I have a group “Annotations” in the root of the DB. I can’t remember creating it. It holds a couple of RTF files with, well, annotations that I entered in the annotations section in the info panel (hopefully this is more or less the correct wording). That’s probably as expected although I haven’t noticed it until now. On iOS these RTF files don’t live in the Annotations folder. This folder is empty and the RTF files are in the root of the DB (which is a bug I presume?).
I am little bit nervous now that the CloudKit sync mixes up file locations and hope to read more reports of user experiences before I delete the old sync location.
I had to unselect all of the databases, then select them one at a time and leave it that way until the sync was complete. Now they are all synchronized, but I still get the rate limited errors even though I’ve not made any real changes to the database contents.
Last Updated for Clarity: February 10, 2021 at 2:07 PM PST
Not sure if anyone else can corroborate this experience but it seems like doing the following will work:
Set Max Connections to 1
Set Schedule to Automatic
Exit DEVONthink and DEVONThink To Go on all macOS machines and iOS devices except one. Meaning, you can only upload from one macOS computer or one iOS device at a time.
I am not sure how this is going to translate into a smooth sync experience. For now, I guess the first task is to get the databases/data into iCloud (CloudKit). Maybe after that, it will work by setting different sync frequencies on all machines/devices. Or, maybe setting them all to Automatic will work. I am not there yet as I am still trying in the initial upload stage.
When I had an error I tried to restart DT and this usually helped, also I think it depends how fast your network is, when I used faster wifi I hit an error much faster and more often
503 Service Unavailable. The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header.
Considering this definition, it’s likely something with Apple’s servers throttling not the sync mechanism itself.
This also is bearing our anecdotally from the comments here and the support ticket I had earlier where I clearly saw the retries and then the finish of the sync with no intervention.
Update: I just gave up. This is too much time and effort just to try the latest and greatest. As @BLUEFROG said, this could be because of issues on Apple’s end but I think I am going to have to wait for other braver folks to do this successfully. Thank you @BLUEFROG and @echej for your help.
FWIW I am experiencing intermittent CKErrorDomain 6 errors by taking my Max connections down to 1. Earlier, I would see CKErrorDomain 6 and 7 errors, and those errors would completely stop the sync process. Now with max connections down to 1 and it being 21:30 local time, the sync now restarts quickly and continues to process. I hope this is a blip on Apple’s end. CloudKit sync is as fast as advertised.
Frankly… can’t tell for sure. I had the beta / Testflight of DTTG 3 installed, downloaded the official release from the App Store and during the onboarding process there was no “free” version until June (btw, minor issue: the text in the onboarding process gives a different date than the actual date of the in-app purchase); at least pressing “restore purchases” didn’t do anything. So I figured out that I needed to install DTTG 2 first. So I downloaded DTTG 2 from the app store. DTTG 2 then found (former?) DBs (synched with iCloud legacy). I deactivated the iCloud legacy sync location in DTTG 2 but the DBs are listed/populated in the DTTG 2 app. Then I started the onboarding process in DTTG 3 again and this time, DTTG 3 offered me a free license until summer. in DTTG 3 I only used the CloudKit sync store, but I can’t say if/which local information from DTTG 2 was used.
I hope this bunch of words is comprehensible and helps. In summary, I don’t know if/what information DTTG 3 has taken over from DTTG 2.
One more observation, this time a positive one: last evening I started a sync of a > 20GB DB from DT on macOS to CloudKit (expecting a lot of “rate limit” log messages). In the morning: DB was/is synched, no log messages. I just synced this DB to an iOS device and again, no issues. Can’t tell any difference in setup/infrastructure (same place, network, devices).
So for those who finished the initial sync - do the problems cease at that point? I have a lot of data to get up there but I don’t habitually upload truckloads, so as long as it’s stable once you’re up and running I’m optimistic still.
)