Syncing a Very Large Database

I have a very large database I am syncing to my iPhone (450,000 items, 500 Gb).

The phone has enough storage space. Once it syncs it works well.

The problem is that when it resyncs it needs to go through over 400,000 items - and if that syncing is interrupted then it has to start all over again.

Thus a file of this size is impractical on the iPhone because inevitably I use another app or I travel out of cell range etc adn the syncing is incomplete.

I don’t mind if it takes a number of days to complete the syncinc - as long as it is possible.

Is there some technical/software reason why the sync cannot automatically resume where it left off?

  1. You have not mentioned the sync method. Details matter.

  2. 450,000 items far exceeds the comfortable limits in a database. Can it be exceeded? Sure, but it’s quite unlikely it will be performant on mobile. Have you considered splitting it?

  3. Why are you trying to put such a massive database on your phone?

I can’t speak for the DT folks, but I wouldn’t even attempt to sync such a large database via the public internet. It just isn’t fast or reliable enough. I’d run a wire from my computer to the phone.

1 - I have tried both Bonjour and also Webdav over my Intranet in my home office to my Synology NAS. Similar issues with both.

2 - I did get it to sync at one point and its performance was perfect - no problems at all. Performance on the Mac is great too - which is why I have not split it.

3 - I am putting it on my phone because it is really convenient to have documents accessible when a client calls.

I could split the database by time period or other ways. But when I once tried that I noted some sync problems. Let’s say I moved some files on my main desktop computer from one database to another. And let’s say I didn’t turn my laptop on for a couple weeks because I did not do any traveling. I found that the sync did not always work and I wound up in some cases where a given file stayed in the original database on my laptop but was moved to the other database on the desktop. That obviously was less than desirable.

I am trying over Bonjour (local Bluetooth) or by Intranet (Webdav on a Synology NAS connected to the router in my house - and syncing while the phone is connected to the same Wifi network, using the Intranet IP rather than a public IP).

What does “resync” mean in this case? Only the first import of a remote database (or merging with a local database) requires all items.

For reasons unclear to me sometimes the app initiates an enormous sync operation which appears to be as large as or similar to the original sync.

You tell me - why might that happen?

Are all documents already downloaded or some still pending? And are documents locally removed only on demand or automatically?

Big databases are a pain. Patience req’d as it takes the time it takes to sync due to volume of data.

I’m taking that you are having issues both with laptop and iPhone syncing.

Recommend:

: On desktop split the databases into one that does not change often, and another that is likely to change. Or more might be in order. I have a dozen or so databases on my real computers, and five on the iOS devices.

: Verify both databases and resolve any issues,

: Get good syncs both databases between desktop and laptop. Use Bonjour for quickest, Synology NAS will work. Ensure only one device (desktop best) has Bonjour set to “accept incoming requests” if using Bonjour.

: routinely turn on laptop even when not travelling and sync. Don’t leave the syncs go to long as your databases are BIG. I think more and smaller syncs might be useful.

: Once you have the two databases on desktop and laptop, sync both to the iOS device (iPhone). iOS controls the frequency and speed of the syncs, not DEVONthink so much. Keep the iPhone “alive” and synching. Watch it. I don’t know how much more space beyond size of database is required on iPhone to sync. but I’m pretty sure more free space is needed. Check that you have a healthy amount of space.

; verify all devices (computers and iPhone) after the initial new syncs, and then again do it routinely (weekly?). With the huge databases, I’m thinking likely not getting syncs completed and I hope verify picks up on that.

1 Like

Everything is downloaded. Documents are removed automatically.

I think I pinned down the behavior a bit more.

I turned off WebDav and left the phone uninterrupted for about 10 hours; the synced finished completely under Bonjour and all seemed well.

Then I turned on WebDav via Intranet to be sure that works; my plan is to switch to WebvDav via Internet when I leave the house later in the day. As soon as I switched to WebDav via Intranet it started the countdown again of over 400,000 items - even though those items are definitely already synced via Bonjour.

Is there some reason why DTTG would not see the WebDav sync and the Bonjour sync to be the same database? Why is it repeating the sync for the files already present?

Thank you - all helpful.

But laptop sync is not an issue - that works well either by Bonjour or WebDav. Though generally I sync the laptop only by WebDav since I can only have one Bonjour connection to my main computer and it seems preferable to leave that to the iPhone. Sync to the Laptop never takes much time.

Sync locations are completely independent, therefore each one has to be fully synchronized at least once.

OK that makes sense.

Is there some way for it to resume the WebDav sync where it left off rather than to keep resetting it back to the start after it is interrupted?

Also - if the sync source is a Synology NAS that can be reached either by Intranet (when in the house) or by Internet (when outside the house) - is that considered to be one source, or do the two IP addresses mean they are different sources even though they both lead to the same WebDav server?

Correction… the Webdav sync went pretty fast after I first synced by Bonjour - so it apparently verified all 400,000+ files but did not try to download them all.

So I guess the lessons learned are (1) Don’t let it get too far out of sync; and (2) Whenever possible use Bonjour on large syncs rather than Webdav.

2 Likes

Bonjour sync directly between devices, I think. WebDAV uses a local “copy” on your NAS (yeah, I know, that is not really a copy of the database).

Whatever is on the NAS is not database. So, DTTG can’t really “see” it as such. Sync is just a process, not a thing. And as I said before, Bonjour does not create the same thing as WebDAV does.

2 Likes

Each sync is actually resumed (but the visible progress indication always begins with 1 as it indicates only the remaining steps).

Multiple sync locations (e.g. WebDAV location and local sync location or multiple WebDAV locations using different URLs) to access the same sync store are no problem at all.

1 Like

@cgrunenberg

After doing more experimentation it is working very well now syncing the large database both to iPhone and to iPad

What I have concluded is that over Intranet Bonjour works best and over Internet iCloud sync works best. Either one of these is efficient in syncing the database when changes are made.

But for some reason my Webdav, which works well to sync DT3 on the Mac, does not work well on DTTG to sync a large database on iPhone/iPad. It seems to take an unusually long time to sync even when just a few small files are added.

Maybe it is some quirk of my database or my networking setup if others have not noticed this. As long as I have a solution that works well that’s fine with me.