Some thoughts on sync

Hi all,

I wanted to share some of my thoughts and experiences around how DEVONthink does sync and the approach that finally works for me.

To give you some background, I’ve been developing software since the 1980s, and leading large software teams for the past decade. My current team is about 1000 software developers. I consider myself, rightly or wrongly, a software expert, and I have done deep work on various distributed systems in telco and eCommerce environments. I’m opinionated, and I’m frustrated with DEVONthink.

I have been using DEVONthink for almost exactly three years, and it is a foundational part of, well, my life. I’ve gone completely paperless, all my main gets scanned when it comes in the door (I got a Brother MFC-9342CDW duplex printer/duplex scanner to support this), and I’m running DT3 on three different MacBooks, two iPhones, and 1 iPad. I have built up an extensive professional library. Almost everything I read gets converted to PDF and put into DT3. I’ve become more disciplined about tagging and smarter about how to organize things, and it provides enormous value in both my personal and my work life. DT3 is an essential tool.

Unfortunately, the sync is hugely unreliable. I realized that I am spending >40 hours/year, minimum, making sure that sync is working, and fixing problems when it doesn’t.

My conclusion is this: Dropbox sync doesn’t work, iCloud sync requires a second local copy of the data, and MyCloud doesn’t seem like a very trustworthy vendor.

Christian (CTO) has been super helpful working with me to try to diagnose Dropbox, but it seems clear to me that Dropbox is just not up to the task for the scale at which I’m using DT. Furthermore, it appears that DT does not use the full Dropbox feature set (which is rather git-like, and, for example, able to compare hashes to avoid sending files over the wire). Whatever it’s doing (treating it like WebDav, I guess), Dropbox doesn’t support well?

So, what is the committed DT user to do? I think I finally found something that works. It’s early days, but it already feels more reliable and also faster than Dropbox (which has been my primary sync tool from the beginning). What I did was the following:

  • Got a Synology DS718+ NAS with 8TB
  • Enabled WebDav
  • Used the Synology DDNS feature so I can have an I can access from outside the house
  • Added it as a sync
  • Converted everything to it

What can I say? So far (2 days later), it is already working much better than Dropbox. My ambient stress levels are down. When I’m at home, it about 10x faster than syncing externally, and, more importantly, it syncs without timeouts. So far, the various Dropbox failure modes (like, files can’t be downloaded, or files added to DTTG don’t end up in Dropbox, or DT3 gets “stuck” syncing) have stopped happening.

Now I have a new problem, which is that the NAS is currently not backed up (though it’s RAID 10 and external backup is possible with Synology).

Anyhow – bottom line, I wish someone had spelled this out for me. If you are frustrated with the sync options, it seems like WebDav is a lot better than the cloud options. Curious if this is well known, and I’d love to hear people’s thoughts on this.

Christian - I appreciate all the support you gave me. I’d be happy to try again with Dropbox if there is a breakthrough! And like I said, I am a committed and enthusiastic customer. But for all the positives of DT3, the sync has got me down.

  • Eric

Thank you for sharing your experience! A NAS is indeed a very good alternative (especially in local networks it’s very fast, almost as fast as a Bonjour sync).

Instead of backing up sync stores I would definitely recommend to backup the local databases.

2 Likes

Eric,

I’m just curious what sort of failure modes you were experiencing with Dropbox Sync. I’ve been using Dropbox Sync with a 4.7 GB database (I sync an iMac, a laptop, an iPad and an iPhone) for quite some time (going back to DT2, now DT3.5.1) without any problems. I have 100 Mb symmetric service from my provider (Verizon FiOS), if that makes any difference.

Thanks,
Andy

I’m curious about this too. I converted from iCloud sync which was completely broken for me (not just for DEVONthink, but for everything) to Dropbox and my 10 databases, totaling 30,000 notes / 15 GB sync almost instantly without issue.

Granted I’ve only used DEVONthink for a few weeks, so I wonder if the sync performance can somehow degrade over time?

Since I don’t like being dependent on Dropbox I will probably end up creating a setup similar to yours. Good to hear that it’s performing well.

@ebowman Thanks for sharing your experience. I too have been frustrated with the Dropbox sync reliability. It works fine most of the time, but occasionally (about once a month) it will just stop working. It won’t restart until I completely clean the Dropbox sync store, and re-download my entire database on all my machines.

I’ve never looked at WebDAV, but at your suggestion, I set up a server locally (using the free WebDAVNav Server app), and a DDNS. Total setup time was < 1 hour. It’s been a couple of weeks and no issues. So far it’s faster when I’m at home, and hasn’t failed yet – though the true test will come over time to compare its reliability to Dropbox.

I’m using a Synology router so I used their DDNS. It’s a router and not a NAS, so I’m running the WebDAV server on a standalone Mac.

-Hankk

For me, DT3 hangs consistently trying to sync and i have to force quit the app couple times a day. This is completely non-transparent in DT and I would end up with ios devices out of sync for days. Sometimes it recovers but if you need to rely on dropbox (at least from my side) you would have to run a cronjob and kill DT hourly.
now i am on icloud, which is hilariously unreliable, but at least i don’t need to auto-kill DT all the time
Additionally with dropbox, it would stop working about once a month/every two months and I had to wipe out the location

Regarding webdav, how do you guys make sure it syncs reliably when you aren’t home? i guess you need to open up ports on the firewall to allow ios devices to connect to the NAS?

What DDNS are you using, if I may ask?

I found that sync would “hang”: it would not make progress or give up. 3.5.1 had some diagnostic code and a partial improvement, where it would timeout after 30 seconds (and you can see that in Console.log). It seems that the way DT3 uses the Dropbox API is not a way that Dropbox really like. It could be there is throttling going on. It’s a pity that DT3 doesn’t use the git-like API calls with Dropbox, which would allow it to sync much faster and use less bandwidth.

The other issue I have had is after using a Dropbox sync for some time, files added in DTTG would simply never make it into Dropbox. The fix was every few months to migrate to a new sync in Dropbox, but this was quite frustrating. I have no clue what was going on there, and I’m pretty sure I lost a couple of files due to it.

I’m using synology.me, which is free for Synology owners, apparently.

https://www.synology.com/en-global/knowledgebase/DSM/tutorial/Network/Frequently_asked_questions_about_DDNS

Or if you want a cloud-based solution, use CloudMe, which is supported by DT3 and has worked flawlessly for me for months now since I had similar Dropbox issues.

1 Like

I have similar issues.

We have been using DT3 for a couple of years. 3 different sites use it to Sync all our docs etc via Drop Box. We didn’t have a problem until a month ago when I went to look at a document that one of my team dropped in, and it was not there. After much digging, it was simply not syncing between ANY of the 3 sites.

So the guys at DT suggested we re-replicate the DB to DropBox. So we did, and all seemed well.

Today - Same thing… Nothing is syncing across sites. No errors - Nada - It says it sycned, but nothing gets uploaded. :frowning:

Synology DDNS too. Their routers include it as a freebie, and although there may be some good free DDNS providers, this works well and is quick to set up.

Yeah, this is along-standing and terrifying bug. It seems to not happen when I sync via WebDAV, except in some cases for databases that index a directory. It was fairly common when syncing to Dropbox.

It’s an absolute pain and we are about to dump Devonthink because of it.

It’s now done the same thing 3 times since I posted my last message.

Such a shame, because it’s a good product, but such a bug makes it unusable for us, and unfortunately we are already heavily hooked into drop box to have to pay for another service.