Dropbox or CloudKit?

Thanks to all. It does seem prudent then to wait until we are past version 1.0 (as always).

David

I put files on dropbox and indexed them to Devothink.
I set dropbox with ‘synchronize content of indexed items’ and synced database to my DTG3 in my iPhone. That let me access the files seamlessly on DTG3. I also set iCloud w aithout ‘synchronize content of indexed items’ and synced the asme database to my another Mac mini. In my Mac mini, the files was on dropbox sync folder. That reduce the storage of iCloud and avoid duplicated by sync to the location ‘dropbox’ with ‘synchronize content of indexed items’.

Wondering if anyone else has migrated from Dropbox to CloudKit and can comment if there is any meaningful improvement … better background sync, faster on-demand sync, etc.

Been with Dropbox for years. No real complaints, but if there is a faster, more reliable method, I’m game to switch.

But if they’re comparable, then no need to pull the trigger.

Thanks.

I switched to CloudKit from Dropbox. It works fast and no issues yet. When I say fast, it seems faster to me. I don’t have an objective measure on that.

My reason for switching:
An odd error would creep in from DB stopping sync. According to Jim in DT support there is a difficult error in DB affecting some DB & DT users. DT is continuing to attempt workarounds but has not been successful yet.

My DT databases would not reliably sync on DB. They might work for a season, then stop. I would then clean the store or create a new sync store then start over. Things would be fine for a season, then stop. I even tried multiple sync stores spreading the sync load and again, it would work for a season then stop.

When CloudKit became available, I tried it since the most recent updates to DTTG and DT stopped my DB sync again.

My problem was somewhat complicated by the Ghost files DTTG 3.x created upon conversion. These also stopped my sync, even with CloudKit.

Once free of Ghosts, CloudKit has worked very well. It is important you sync only one DB at a time as suggested in the forum (Note from Jim). CloudKit will throttle your efforts if you try all your DBs at once.

I have one DB that would not sync with CloudKit. I have a 72 GB database with 852 scanned books. A few years ago I scanned all of the books in my library. They are now all searchable in DT. This process is a subject for another story. While CloudKit would stall syncing this to my iPad, I was able sync on DB (go figure). I give this database its own sync store. We will see if this sync will stall like the rest but for now it works. It is fun having my library available on my iPad.

One additional note. All my DBs are indexed. The files live on my iMac but the folders are also synced to my DB account. I had several reasons for indexing. One of the reasons for my shift to indexing was to remedy the issues I was having with the DB sync. Like all my other efforts, sync worked for a season, then stopped.

I hope my story helps you in your evaluation of CloudKit. I had great help from Jim with DT support in understanding the issue with DB. He also gave me advice on sync with CloudKit.

2 Likes

I would say, “If it ain’t broke, don’t fix it.” :wink:

However, we have been hearing very good reports that conform to our experience in testing. And as @GuernBlanston helpfully pointed out, being patient with the initial push is seeming to help avoid some of the throttling we’ve had reports of.

1 Like

Thank you! This is very helpful. I think perhaps I will test CloudKit sync on a small DB and if I find it works without problems, move the main ones over.

How do you assure that you are syncing only one DB at a time? If 2 are open, both will sync, won’t they? So does “make sure you sync one at a time” = “only have one database open at a time”?

David

We believe in opt-in processes as much as possible. This means just creating or opening a database will not sync unless you explictly enable it. (That’s a decision for the user to make, not the software.)

Just check the checkbox for one database in the sync location in DEVONthink’s Preferences > Sync. Wait for it’s activity to finish, then enable the next, if needed.

@Dberreby Follow Bluefrog’s recommendation, this is exactly what I did.

DB allows for multiple sync stores where Cloud Kit only allows one. Bring up everything slowly, one database at a time. I made a mistake once and thought a database had uploaded and checked off the next box. Sure enough, CloudKit throttled the upload. I noticed my CloudKit databases uploaded quickly when taking this measured approach. I have pretty fast internet service so it may look a little different for you.

2 Likes

Got it!

I gather this this a one-time thing, when one is first syncing big databases? If I had to do this every time I opened databases, I would certainly lose track (forgetting to check and uncheck the boxes). But if it is something to do once when first starting the process, I can live with it.

Thanks, as always, for the quick and useful answer!

David

You’re welcome and yes, if you enable syncing a database, it stays enabled unless you disable it or clean the sync data.

1 Like

Just a postscript to say that after much hesitation I finally switched over to CloudKit. And, as others have mentioned, it is faster and smoother (far fewer “timed out” messages or other hiccups). It’s a real improvement! Thanks for encouraging me to make the jump.

1 Like

Glad to hear it and you’re welcome! :slight_smile:

Hi Jim! As of today my desktop and my laptops are synchronized using local networks.
I now need to move two of the laptops to a new location and want to allow DEVONthink To Go to easily sync metadata without being on the local network. So I need to switch to an online sync method.

I have enough storage on both iCloud and Dropbox. Initially I considered using Dropbox because I never had a reliability problem with it and because of all the bad experiences with iCloud, the iTunes-related data losses I experienced years ago, the lost notes and the Safari tabs that never correctly sync up to this day.

Then I read this thread that highlights how iCloud (CloudKit) is the fastest option, so I considered using it. I did a bit of research and found a very different opinion on iCloud / CloudKit from developers:
https://www.lifewire.com/icloud-sync-may-not-be-as-dependable-as-you-d-like-5217076
https://www.macrumors.com/2022/01/24/developers-icloud-unreliability-bug/
https://reverttosaved.com/2022/01/24/icloud-sync-is-randomly-breaking/

I want to take the absolute safest choice because, not only do I not want to lose random files during sync, but considering the very big size of my databases (for instance one with 44k elements in 146GB and one with 47k elements in 236GB) I would never realize soon enough if a dozen files started missing somewhere.

What is your take on this, your own experience and the feedback you got from support request? Should I not use Dropbox and play it safe, even at the cost of some performance?

Glad to hear you’re up and synced via Bonjour!

Regarding a remote sync, we don’t advocate one service over another. iCloud seems to behave better in some instances (or just on some days) and for others, not so much so. That being said, you are correct that iCloud / CloudKit is a sore topic for many developers (and users) so if I were to be asked over a coffee or an adult beverage, I’d opt for Dropbox - again, with no affinity for the service.

But as noted here, any remote sync option comes with it’s inherent weaknesses…

One side note: Many people feel Dropbox is too intrusive on your system. (I will reserve my comments for the second coffee or adult beverage - LOL). That being said, Dropbox’s application is not required to sync DEVONthink with it. So if you’re only going to use it for non-syncing use, I wouldn’t even install their application.

1 Like

Thanks for the answers Jim!
I realize that I sounded like I just started using Bonjour sync while I really meant I was currently using it and have been happily using it for years with no single issue in the recent years.

Now how can I materialize these two adult drinks? Is there an online tip jar? I’d be happy to use it!

I share your concerns about public statements over an undisclosed fruit company that generally discourages free expression.
I already have the Dropbox app installed on all computers so even though I hate their reminders to force me to move my Documents folder to their cloud, I suppose I am already used to it. Unless they do more intrusive things that I’m not aware or. And yes I’m aware that Condoleeza Rice is a member of their board and that NSA definitely has their own private key access to all data, the same way the infamous Windows leaked NSAKEY showed many years ago, and iCloud is probably no different because of US laws. I suppose a probably-non-NSA solution is through a GDPR WebDAV provider such as CloudMe, but with a big difference in price, and a limitation on files over 2GB (maybe due to the WebDAV protocol). I wish there was an option to use the non-WebDAV Proton Drive, that has a proprietary security system.

Has there ever been a verified instance of data loss using Dropbox? As you can see I’m struggling to find a balance between the comfort of interacting with my databases from anywhere with my iPhone, and the safety of data.

Haha! Thanks for the offer but I’ll patiently wait until we all can easily share an aperitif in person :wink:

Has there ever been a verified instance of data loss using Dropbox? As you can see I’m struggling to find a balance between the comfort of interacting with my databases from anywhere with my iPhone, and the safety of data.

Due to just using Dropbox as a sync location? Not that I recall.
There are certainly cases where people have indexed data and accidenally or unknowingly deleted things. We do have one ticket open on some unknown deletions. However, it is unclear where that issue is coming from.

I have no indexed data so I should be fine.
I made a test database to try things out. I have noticed that the upload speed was 2MB/s on average, way less than the 50MB/s my optical fiber network allows, and the download speed on the second computer was 7.5MB/s, still way less than the 100MB/s allowed by my network. I’m not sure what the reason is. I see no CPU bottleneck that would be explained by the encryption. I also made sure to remove the upload bandwidth limit in Dropbox preferences (no download limit by default). By the way, the encryption layer offered by DTPO is a great option—take that, Condi!

For a better sync speed between my laptop and my desktop in case they’re on the same network, and for a wider reach when they’re apart, I’d like to keep both the direct sync method activated on my laptop (my desktop being the server) and the Dropbox method activated on both.

In the small test setup that I made with an empty database, when I add a very big file on the desktop computer, it triggers the slow Dropbox sync and, during that long time, the laptop regularly tries to do a direct sync (automatic setting) but says that the remote (desktop) database is busy.

Then, after the upload of the big file has been made to the Dropbox sync store by the desktop, the laptop starts to slowly download the big file from Dropbox instead of using direct sync.

So, in my experience, it seems that DTPO starts to sync from the top of the list of sync methods, and I cannot find a way to reorder how syncs are attempted (the same way you can reorder networks in macOS Preferences).

It works but it’s not convenient to disable automatic syncs (and uncheck the “on exit / deactivation” checkbox) in order to manually trigger the right sync method because when on the local network, I would have to manually trigger the direct sync for better speed, wait for it to complete (indeed quite quickly), then trigger the Dropbox sync so that I don’t have to remember to do a direct sync on my iPhone or iPad before leaving. Also making it manual would lose the automatic backup to a local sync store that I setup on the desktop, and overall be prone to mistakes and complicated.

Setting a different sync period for each method would mostly solve the issue (every minute for direct sync on the laptop, every 15 minutes for Dropbox on laptop and desktop, automatic for local database backup on desktop), unfortunately it’s not possible.

There must be a better way for a simple use case as mine. Am I missing something?

the laptop starts to slowly download the big file from Dropbox instead of using direct sync.

You should not have local sync data on your machine.
Please read the In & Out > Sync > Dropbox section of the help about this.

Also, a ”very large file” will be chunked and transmitted in pieces. This is the norm and how sync has worked for a long time.

I read three times the help page on sync. I have excluded the DEVONthink Packet Sync from the synchronized folders on both computers. I meant that the laptop took its time to download from the remote Dropbox server through the Dropbox DT connector (not the Dropbox app).
I saw that the big file was chunked, I just did so that I would have the time to mesure the average speed and to give me enough time to see the chronology of the sync methods.