I’m trying to set up sync of a database via WebDAV to an OS X Mavericks Server. I want to use WebDAV compared to something else because I have the infrastructure, it should be fast when I’m at home on the network, and it should continue to work when I’m not at home since I have it exposed.
However, it seems really slow. I’ve got a 1GB database, syncing over gigabit ethernet, and it’s only done about 60MB in twenty minutes. Why is that? It’s definitely not saturating the network.
Some googling suggests that Mavericks WebDAV is really slow for some reason. I’d love other people’s experience with WebDAV.
WebDAV in general isn’t very good IMO. Lots of promises but a lot less delivery.
I’m having the same problem syncing with Dropbox.
I have a 4.5GB database filled (mainly) with email. Everything that’s not an email is indexed. I’ve literally let Dropbox sync go for 12-15+ hours straight (overnight plus some) and it had only synced around half the database (according to the progress bar, that is).
If any one has any thoughts it’d be much appreciated.
A 4.5GB database is not small. I don’t find your timeframe to be unusual considering the size of your database and the lack of consistent network throughput. Of course, this should only be on the first Sync since the whole database isn’t Synced every time.
Also, be aware that indexed files Sync too.
From watching my server logs, I can see two things that could be killing WebDAV speed:
- DTPO seems to establish a new TCP connection for each request instead of using keepalives, so every operation (“is this file here?” “show me its properties.” “replace it with this content.”) requires an entire TCP handshake. This gets a lot worse if SSL is thrown into the mix because SSL negotiations are “expensive” - they take a relatively large amount of time and computing resources.
- DTPO tries a lot of requests without sending authentication headers, getting a 401 Forbidden error from the server, and then retrying with authentication. This current logfile has 469,000 requests from “DEVONthink Sync Plugin/2.5”. Of those, 112,000 were unauthenticated and denied, then immediately successfully retried with authentication.
The net effect of those are that DTPO makes many more connections than it has to, and sends many more requests than it should. I’ll go out on a limb and guesstimate that using persistent connections and always sending authentication would make WebDAV syncing at least 30% faster without changing the underlying sync protocol at all.
I’m not sure how others are doing this, but here is how I do syncing between my office and home mac.
I use a program called NeoRouter, which is a free VPN solution (neorouter.com/). I host my own private server on my work computer so it is very secure.
I then can see all the macs on my VPN, connect directly to the shared folder, and can sync a 25 GB database with a direct connection in 20-30 min. My upload on both ends is around 2 - 4 MB/sec. I found this to work better than trying to use direct over BackToMyMac or any other WebDav/Dropbox type solution.
However, I also have a 128 GB USB Drive with a local sync store, and honestly I prefer that over the network sync because it is much faster. The USB Drive is encrypted, and I have physical control over it, so I have assurance my data isn’t floating around out in the cloud. I keep the VPN up for screen sharing and to sync if I forget the USB Drive at work
Hopefully this gives some ideas and insights.