How to synchronize DTTG such that a large database can synchronize with metadata and search data

and the download of a file can happen on demand.

as of now, the Mac DT3 Pro is synced to iCloud and also open to Bonjour connection. On the DTTG side, I think I setup the database to not download the file each time, but it still seems to take a long time to sync. And the sync message on iOS says - “Downloading items from iCloud (CloudKit) 130 of 4150 items and it is taking forever”.

the net effect is that my iOS database is almost never synced. :frowning:

not having iCloud is also not an option for me. there are many times I want to access one or two files when on the move. further, the bonjour connection seems to flake out as it runs into the error - Mac db not available etc.

I am sure I am doing something wrong. And am open to deleting the database on iOS and starting over if I can speed up the sync.

Thank you!

Read the hundreds of other posts here about Apple iCloud not working well for some. Not everyone experiences issues, but there are issues none the less. Guess in the meantime you’ll have to wait for Apple to fix it, or get a bigger portable device and sync more/all files, or try another internet sync service.

1 Like

got it.

and I have read various posts in the past. I am ok to stop iCloud.

I had moved to having the data stored in WebDAV store on an “always” on Mac at home but moved away from that because the Mac was not reliably on.

Before I give up on iCloud, key question is whether I can sync with iCloud such that the entire file does not have to sync. so long as I can sync such that the data is searchable on the iOS app but entire files don’t get downloaded - not because there is lack of space, but because it does not seem to synchronize then.

This is not a flaked out error. It’s stating you are trying to sync a database via Bonjour and that database isn’t open on the device acting as the Bonjour server.

And the sync message on iOS says - “Downloading items from iCloud (CloudKit) 130 of 4150 items and it is taking forever”.

As stated many times, we have no control over the speed and responsiveness of any remote server. CloudKit has been especially unreliable for quite a few people in the pasrt 9 months or so.

1 Like

For my Bonjour sync error, I posted a screenshot of my Mac log below
The iPad shows DTLow@…macmini.hitronhub.home.local: Connection tt:tt to server timed out.
I usually get to 15000 of 30159 items
image

I think I’m one of the few people in these forums who doesn’t gripe about CloudKit at every opportunity :slight_smile: . I’m successfully using it for multiple devices (discussed in a different forum posting). To give you an idea of how much data I’m sync’ing, if I look at the Settings panel on my iPhone and look into iCloud Storage usage by different apps, DEVONthink is using a total of 90 GB. So, from the perspective of someone who uses it successfully, here are a few ideas about the problems you’re experiencing:

  1. One thing that may make a big difference is the speed and nature of your internet connection. Is it possible that your connection to the wider internet (specifically to reach Apple’s servers) is relatively slow, or saturated from traffic from other devices, or unreliable? You would probably know this from your personal experiences in general, and from clues such as whether the performance of other network-heavy applications seems slow (e.g., whether you can play videos smoothly or whether there are a lot of pauses). You could also test your network speed using various sites on the net that offer free speed tests (e.g., https://www.speedtest.net – this is not an endorsement of that site, just an example).

  2. Have you tried the step of cleaning the DEVONthink sync destination, deleting the database(s) from your iOS device, and letting your iOS device re-download the database from a clean starting point? (Note this is different from recreating the database.)

  3. Not knowing anything about the state of your devices, is it possible your mac and/or iOS device are old and slow (kind of like me)? The mac is probably fine unless you have a very old machine, but I remember when I was using my iPhone 6S a couple of years ago before finally getting a new one, it struggled – and in fact, I couldn’t sync all my databases because the phone didn’t have enough storage space. I think DEVONthink will run on probably all macs and phones and ipads; however, performance problems may contribute to the symptoms you’re seeing.

Some of the things mentioned above may be outside your control, but IMHO it’s still important to consider some of these variables when trying to identify the root cause of a problem.

2 Likes

I also get the same “connection to server timed out” error. synchronization starts and then for no apparent reason this error pops up. :frowning:

thank you @BLUEFROG for the quick reply.

Thank you for your comment.

#2 sounds like a good idea to try. I did delete the iOS database recently and started over, but I did not cleanup the sync destination from inside Devonthink on the Mac. My devices are both new and connection speed is quite good too.

One more question for you: is your iCloud setup on the iOS device set to Download all files or is it set to Download on Demand. Mine set to Download “On Demand” Screenshot 2023-04-20 at 1.08.47 PM

I do have both iCloud and Local network setup. I had hoped that local network will help faster sync and there will be little left for iCloud to accomplish. But I don’t think that is being achieved.

For DTTG on iOS, I use the “always” setting. Admittedly this makes DTTG take much longer to perform the initial sync, but as I’m sure you know, it means the files are available on your device even when you have no network connection. It also makes accessing the files much faster, because they never have to be downloaded. (Of course, that assumes you periodically open the DTTG app on your iOS device so that DTTG has a chance to keep itself up-to-date after the initial full sync.)

With respect to cleaning and re-synchronizing a database on an iOS device, here is the procedure I follow. This is kind of an extreme protocol, not based on recommendations from DEVONtechnologies (so don’t blame them) but based on thinking through what is happening on the different devices and trying to avoid DTTG starting to sync a database before it is fully present on the sync store destination.

A. Preparatory steps on the IOS device:

  1. get out of the DTTG app on the iOS device and close it – this is not the same as merely getting out of the app

  2. turn off networking on the iOS device by putting it in airplane mode and turning off wifi (because wifi is a separate toggle from just putting the device in airplane mode) – basically, prevent it from doing anything over the network

  3. re-open the DTTG app on the iOS device

  4. delete the database(s) you want to resync (left-swipe on the database name in the list of databases at the top level)

  5. let DTTG finish deleting the database, and check it’s gone

  6. close the DTTG app again and do not open it again until part C below

  7. you can reenable networking on the iOS device

B. Clean & recreate the database on the sync store

  1. in DT on the Mac, go to Preferences ▹ Sync, select the sync store destination you are using, right-click on the appropriate database name in the list, and select “Clean Database…”. Here’s an example if I wanted to clean my “projects” database:

  2. let it finish the cleaning step; when it’s done, the checkbox next to the database in the list will no longer be checked

  3. close the preferences panel

  4. select the database in a normal window in DT (any normal window where you have the left sidebar showing the list of databases you have)

  5. run the command File ▹ Verify and Repair Database in DT; if there are any problems, resolve them

  6. run the command File ▹ Check File Integrity; if there are any problems, resolve them

  7. go back to Preferences ▹ Sync and once again select the sync store destination you are using

  8. check the checkbox for the database to make DT start sync’ing the database

  9. let DT finish sync’ing completely – for my ~30GB projects database, on an M2 MacBook and relatively fast network service in Southern California, this took around 6-8 hours

C. Back on the iOS after DEVONthink has finished uploading the database to the sync store

  1. open the DTTG app

  2. go to the sync locations and select the correct sync location

  3. wait a while for the database to show up in the list of databases – this can take a couple of minutes, so if it’s not there at first, give it a little time

  4. flip the toggle next to the database’s name

  5. close the preferences panel and go back to the front of the DTTG app; it should start showing messages about downloading the database

  6. let it finish completely this initial download – since DTTG is not downloading the entire database (yet), this sync shouldn’t take as long as the upload from the Mac, but for me it took maybe ~2-3 hours

  7. let DTTG run until it finishes, but don’t actually use it – don’t even bother looking inside the database being sync’ed because it will not look correct until the synchronization is completely finished

Optionally, if you want to use the “always” downloaded setting on iOS:

  1. select the database in the list of databases

  2. tap the circled “i” button

  3. flip the “On demand” to “Always” for that database; this will make it start downloading the full content and this will take many more hours to finish.

Note: instead of doing “On demand” for the whole database, you could also opt to make specific folders be “On demand”, if you don’t have space on the iOS device to store the entire database.

I’m using only one sync mechanism, and watch out that the procedure above was written with only one sync mechanism in mind …

Thank you for such a thorough response. I will exactly the same steps. They make sense to me. And if they work you, then chances are they should work.

I am curious as to what happens if there are two sync stores - I am guessing that if that is the case, then the iCloud sync should happen faster if all downloaded locally itself. I think I read something along the lines here some time ago. But I hear you that two sync stores is not part of your working model.

Thank you again @mhucka!

1 Like

:person_shrugging: Honestly, I have no idea. Hopefully other people will chime in about that scenario …