CloudKit: Missing Manifest & Broken Database Structure

[jwarthman]

I’ve been syncing my primary database and Global Inbox between my iMac, MacBook Pro, iPad, and iPhone using CloudKit with good results, until recently when I’ve been getting iCloud (CloudKit) manifest missing and database structure issues.

My setup:
The iMac is running DEVONthink 3 Pro version 3.8.5 and macOS Monterey 12.5.
The MacBook Pro is running DEVONthink 3 Pro 3.8.5 and macOS Catalina 10.15.7.

The iPad and iPhone are running DEVONthink To Go 3 version 3.5.2 (17229) on iPadOS / iOS 15.6

I am not using Bonjour syncing. On both Macs, “Enable incoming connections” is off and Bonjour syncing is “off:” on both iOS devices.

Both Macs and the iPad and iPhone are set to sync the Global Inbox and my primary database, and the iPad and iPhone have CloudKit set to download files “Always”. So effectively the two databases are mirrored on all devices.

What I’ve done
A wile back I started getting “manifest missing” log messages on all my devices. Initially I believe many, many files were being listed in in the log, but after doing a “Clean” on the iMac the logs on each device list a single file name as having the missing manifest, and it’s the same on all devices.

I researched this in DEVONthink Help and in this community and tried various steps.Not sure I can remember everything I tried, but I did “Verify Location Quickly” on the iMac, and eventually I resorted to “Clean Location”.

I still get the manifest missing messages, but more ominous is that now, when I “Verify Location Quickly” I receive the following message in the log:

Broken or not yet complete data structure of database “my database name”.

I get this same message on both Macs, which appears to be related to the CloudKit sync store, not the locally stored database(s).

I’ve been troubleshooting this unsuccessfully on and off for weeks. Perhaps this warrants a trouble ticket / bug report but I thought I wad ask here first.

Regards,

Jim

[rmschne]

Apple’s Cloudkit misbehaving again. Search this forum for “missing manifest”.

If you need sync, then setup and try another sync service. I use Bonjour, Dropbox, and supplemented with WebDav with a local Synology NAS.

[jwarthman]

Thanks, rmschne.

Yes, before I posted I read probably a dozen threads concerning “missing manifest” and “broken database structure”, and I followed a number of suggestions from those threads.

I have use Dropbox in the past as a sync store but was under the impression that CloudKit is more reliable these days. Unless I’m mistaken, and Dropbox is the preferred alternative, I’d prefer to stick witih CloudKit.

Regards,

Jim

[rmschne]

@jwarthman

I’d prefer to stick witih CloudKit.

That’s your choice. I prefer to focus on my DEVONthink content instead of debugging Apple’s stuff.

Haven’t done the statistics, but based on numerous reports here, Apple’s CloudKit is the one third party service reported most often as being unreliable for some. Sometimes clears on its own – sometimes not.

At minumum check you are not using Apple’s “optimise disk” with your iCloud service. That will break sync as DEVONthink expects the files to be in place on the local disk.

[BLUEFROG]

We don’t have a cloud service we advocate. However, CloudKit has been very unreliable in the past several months.
And if you don’t need to use a remote sync option, we suggest you use a sync on your local network.

[jwarthman]

Thanks, BLUEFROG, that’s pretty definitive. I’m willing to transition off CloudKit in favor of reliability!

I’ll search the community of course, but can you say anything about Dropbox reliability “lately”?

Last, I need to learn more about local sync options, so I’ll read the documentation. If anyone has pointers to useful discussions, I’m interested!

rmschne, I see you use a combination of dropbox “supplemented by” a local sync option. I’m interested to hear more! Are you syncing the same databases to both Dropbox and local sync stores?

Regards,

Jim

[rmschne]

@jwarthman

but can you say anything about Dropbox reliability “lately”?

Absence of reports does not prove absence of reliability issues. We see so many reports about Apple’s services failing yet I cannot remember any about Dropbox that dealt with unreliability vs. user error in setting up.

Dropbox runs on computers, and is surely not perfect, of course.

Using Dropbox to sync to phone and iPad, I have only my “Works in Progress” database that holds stuff I want to read and stuff that I’m working on (writing, editing, etc.), or stuff that I might need for travelling (bookings, certificates, etc.).

I “forget about it”. For me Dropbox “just works.” I do not attempt to sync everything that exists on the real computers for “just in case wanted”.

These few databases also set to sync with Bonjour which only kicks-in when the device is inside the local network.

@jwarthman

rmschne, I see you use a combination of dropbox “supplemented by” a local sync option. I’m interested to hear more! Are you syncing the same databases to both Dropbox and local sync stores?

You’ll find I’ve written here so many other times on what and how I sync. Search here.

I don’t use Local Sync Store.

As mentioned in my reply above, I use Bonjour and WebDav on the local network. Both sync everything between an iMac and a Macbook.

See page 60 of the current and outstanding “DEVONthink Handbook” for more info on synching.

Remember -

• sync only if you need to
• sync only what you need to.

[BLUEFROG]

Dropbox had a crash with their API last release but that appears to be addressed now. Otherwise, there haven’t been many recent reports of issues with it.

Note: You do not have to use the Dropbox application on your machine to use Dropbox as a sync location. You don’t even need the Dropbox app installed! Our sync engine deals directly with Dropbox’s servers.

[awcb]

Hate to be dumb, but what’s the difference between remote synching and synching on a local network? In my case, the only thing I need to sync are items from Devonthink To Go’s Inbox (iphone and ipad) to Devonthink on my laptop.

thanks,
Amy

[chrillek]

Local network: Bonjour (or perhaps WebDAV, if you have that on a NAS, for example)
Remote: CloudKit, Dropbox and the like (or perhaps WebDAV, if you have booked that somewhere).

Bonjour is reputed to be the easiest method (if you follow the rules, that is). You might want to read up on it in the fine manual.

[rmschne]

Remote syncing is when you use a “remote” server as in Dropbox, iCloud, etc.

Local means “local network”. While local could mean using a WebDav server on your local network, the best to use first would be Bonjour.

The “DEVONthink Handbook” explains more and explains how to set up and use Bonjour. Fast and reliable.

[rmschne]

See

Sync Types Explained

First Questions Sync is simple but you need to choose the type of sync method to use. Regarding syncing, the first question you need to ask yourself is, “Do I need a remote sync option?”. Consider these questions… Do you need to sync between machines – especially desktop Macs – in different geographic locations? Do you have a colleague, assistant, significant other, etc. that needs frequent updates to synced data? Are you using a shallow sync, i.e., Download Files: On demand in DEVONthink To …

and

Bonjour Simplified

To help clarify the conditions under which Bonjour can be used, as well as show the simple steps to setting up such a sync, here are notes and steps to follow. Requirements Bonjour can be used to Sync between devices, IF: The devices are on the same network, noting it will not work over VPN. The network is private or one that allows Bonjour connections or non-standard ports. (Public and corporate WiFi sometimes disallow these.) Your firewall or an application like Little Snitch is Off or has …