syncing DTPO over the net Webdav

DEVONthink Tips
1

Hi all
task: to keep several databases on my office iMac and my home Macbook in sync.
I was really excited that DTPO offered syncing via Webdav as well as a dedicated sync store.

I had high hopes that I could make use of my server and finally retire the USB hard drive that needed plugging in, syncing after arriving at a workplace and before leaving again. Not to mention the constant worry about forgetting it.
What I was envisioning was putting the Devonthink syncstore on my syncstore and pushing the Sync button before leaving for home and arrive home finding a fully synced DTPO database. Forgetting to push the button would be no problem as I could always VPN to the other computer to do so should it be needed.

The server runs Ubuntu and I have tried:
owncloud
Seafile
git-annex
Dropbox
and experimented with either syncing to a single syncstore over Webdav or syncing to a DTPO syncstore locally that was located in a synced folder. The disadvantage of the latter solution obviously is that the local syncstore needs space locally which is a problem on the Macbook, on the other hand syncing locally happens almost instantly which makes leaving spontaneously a real option. Syncing over Webdav takes a lot longer obviously, which wouldn’t be a problem if it was foolproof and successful every time.
But sadly it isn’t, so leaving before knowing that the sync hasn’t crapped out for some reason turned out to be impossible.

Dropbox was the best of the bunch but I wanted more control over my data so I haven’t tested extensively.

git-annex has some really clever features but is not really up to the task. The problem is not so much the size but the number of files inside the database or syncstore. It is simply too slow for my databases which are between 1 and 3 GB each.

owncloud and Seafile are very similar in what they achieve regarding the sync but differ in how they achieve it. The short story is that owncloud is a full-featured, includes a calendar solution and lots of other options beyond simple file syncing a la Dropbox. I wished that it tried to do less but that well enough but have to say that if you are really interested in syncing mostly it is clearly inferior to Seafile. owncloud scales poorly, is a lot slower and less reliable.

Unfortunately, Seafile is better with regard to syncing but like owncloud it will only sync successfully sometimes but not always. Without scientific rigour I would guess that 70% of the sync attempts were successful. Combined with Devonthink’s habit of failing silently it was all too easy to think that everything has gone smoothly when it hadn’t. When more than one or two changes had happened in the database, Devonthink would fail stating that some document had not been found when inspecting the remote database via VPN showed the document clearly.

I contacted the Devontec support but although invariably the response time was good it was impossible to troubleshoot to the level of detail needed to get it to work eventually. Keep in mind that Webdav is a complex thing so many things can go wrong that might be beyond the Devontec control that I cannot really blame them. That said, it looks to me that Devonthink had its share in creating the problems.

The practical upshot nevertheless is that out of the box syncing over the net does not work via Webdav or self-hosted file syncing solutions, at least not for databases above 1GB that are changing fairly dynamically. Dropbox may be okay but was out for other reasons for me personally so I am back to using an external SSD drive connected by USB.

If anybody can contribute his or her experiences, especially more positive ones, I’d love to hear about it.

Best regards
Prion

2

Please explain what that means. I’m using Dropbox (not for DT syncing) and I’m not sure what I’m missing in terms of “control”.

I’m not really in a position to comment sensibly here, because I’m living with a “one laptop with a 1 TB SSD goes wherever I go” scenario, so I only sync to DTTG. But nevertheless, I observe that if Dropbox can work, it’s strange to dismiss is so easily.

Edit: I think I get it, you mean control as in security. Fair enough.

3

Yes.

While remote Sync solutions are always going to be more error prone due to network vagaries, API limitations, etc., Dropbox has always been the most problematic. I have successfuly Synced to WebDAV from the beginning: ownCloud, Box.com, and now CloudMe. The issue with WebDAV is that, while it’s a “standard”, it’s a loose standard. However, the three I mentioned operate quite nicely. (I actually would personally advocate Box’s WebDAV over Dropbox any day.)

Note: Other than inherent weaknesses in remote solutions, the major failing of Sync is not being diligent in checking the health of databases. Sync is NOT going to transmit incoherent databases. (And yes, a check is done at Sync time to see if the database or the Sync Location is “ready" for the transmit.)

4

We sync 9 Macs with a 13GB database over either WebDAV when away from the office, or Direct Sync when in the office.

It works wonderfully well, and each time I sync I am slightly gobsmacked by how awesome this is. It means my colleagues always have the latest information available to them, wherever they are.

We have a central Mac Mini which is the master, and it is set up to automatically sync at 4am to the WebDAV sync store also hosted on that Mac Mini.

All the other Macs sync to that Mini.

In the early years we did have sync problems, but Nathan provided excellent support, far beyond what I could have expected, to help get it running.

When the new DTTG finally breaks cover, it is likely to also have this same fantastic sync capability.

In my opinion what Devontech have created with this sync capability is incredible.

5

I am among the few, I guess, unlucky ones who’ve always grappled with DevonThink Sync. Troubleshooting with support, while speedy, did not track down the root cause of the issue. Something was clearly wrong with my database, but we never quite found out exactly what. I tried both Dropbox and local stores via an external hard drive—but never WebDAV. Admittedly, my database is complicated: half of my data is imported, the other half is indexed; and I move between an old machine running 10.8 (therefore with no native Finder tag support) and a newer one running 10.11.

To make matters worse, I never quite understood why manually ordered items would sync in the wrong order… until the DevonTech crew, responding to my question on this forum, clarified that this not by accident but by design. An non-negligible omission, in my opinion, and in any case one that users should be warned about prior to their first sync operation.

My third reason for giving up on DevonThink Sync is the disproportionately sized sync stores, which duplicate all your indexed data (i.e. all files that reside outside the dtBase package). In my case, this involved a sync store 10x the size of my database (19GB sync store for a 2.2GB of data). Not a viable requirement on my end.

This is strictly my own experience, of course, and I realize that the majority of users are happy with DevonThink Sync.

It’s now been a week since I started using BitTorrent Sync for P2P syncing between my machines, with no “cloud server” involved. Try this at your own risk. Several users on this forum, including the DevonThink crew, will advise you to never sync your DevonThink database directly, without the mediation of a sync store. But BitTorrent Sync has been working like a charm here, and I’ve been really stress-testing it lately. Two very important caveats:

  1. Keep the DevonThink database open only on the machine that you’re actively working on. It should be closed on any machine that you are syncing to.

  2. Make sure to set up BitTorrent Sync so that it does not ignore any extended attributes. You can do that by inserting the following lines at the top of .sync/StreamsList:

*
com.apple.*
org.openmetainfo.*
com.dropbox.*
kMDI*

The first line is probably enough but better safe than sorry.

Also be sure to check .sync/IgnoreList out and ensure that BitTorrent is not excluding from syncing any file names that are used in your database.

6

So does Dropbox appear to… until it jacks up your database. I will reiterate: TRY THIS AT YOUR OWN RISK. It is not supported behavior.

7

Yes, that’s what I meant.

8

Whilst this is certainly meant to sound positive, my experience is not like this at all. Sync will go haywire unpredictably and it will at times corrupt the remote syncstore as demonstrated by failed subsequent syncs with duplicated databases preserving a state that had not been problematic.

And even if these problems could be identified before the sync (which I don’t think they can, see above), I would like to see a more informative error message. All I can do as a user is verify and repair the database from time to time and this has never indicated a problem beforehand.

9

peter999

this is precisely the kind of post I had been hoping for. Does the syncstore reside inside the OS X file system? I have never tried that, perhaps I should…

If I may ask some earnest questions:
Can you be a bit more specific what the original problems were and what Nathan recommended you did to solve them?

How long has this been going well? How much of the content changes between the syncs? Do you only add new stuff or modify and delete existing content? And how long does a sync via Webdav take?

Thanks very much for this information, it may mean that what I was dreaming up is possible after all.

Prion

PS:
It occurred to me that I also tried some more services
Strato HiDrive (offers Webdav, but was unsuccessful, too)
Telekom MediaCenter (same story)
Copy.com (I don’t remember what the problem was here to be honest)

10

Point well taken, Jim. Allow me to say, though, that I tried the DevonThink sync mechanism also at my own risk— and it didn’t work as effectively (on my system) as BitTorrent Sync, which so far has been flawless. As Woody Allen said, “whatever works”. :smiley: Also, there are strong reasons to believe that BitTorrent Sync might be a robust solution. Its design principles are completely different from those of Dropbox (there is no “cloud” in-between, it’s functionally similar to rsync, only much faster). Anyway, just sharing an experience with what indeed is a risk-prone experiment. I look forward to trying the built-in DevonThink Sync mechanism again as soon as DTTG is released.

11

See replies above below your comments.

One very important point to note:

Always do a Validate and Backup routine, then export a Backup.zip of the database BEFORE you sync, and do the same on all other machine on a regular basis.

I hope this helps.

12

”Always” is a strong word, but I won’t argue if someone wants to be extra-careful.

13

Indeed, I’ve noticed that sync times tend to be considerably longer after “Validate & Backup” routines, even if the my data hasn’t changed significantly. This is just an informal observation of course, and I could be totally wrong. But it’s the reason why I try to curb my urge to Validate all the time.

14

Just to be clear… are we talking about two different things here?
Tools > Verify & Repair should be routinely run. Period.
Tools > Backup & Optimize can be run as frequently as the User wants.

15

I stand corrected, Jim, thanks. (This is what can happen when you write scholarship at 4am, thankfully as DevonThink keeps churning out tons of relevant information with no signs of fatigue).

I was referring to Backup & Optimize.

16

:smiley:

17

Sorry, I did indeed mean always Verify & Repair, and Backup & Optimise.

I do also export a Database.zip backup before undertaking a major (+50MB or so) sync.

It may be overkill, but I sleep easier… :smiley:

18

Never underestimate the beauty of resting well. :smiley: