WebDAV sync store & nginx

I have a WebDAV server set up, with its root folder /SomeVolume/webdav/contents and address 192.168.1.2:12345. User and password are set and the server can be accessed from a browser.

I am directing a domain name to the same machine using nginx, like this:

server {
    server_name my.domain.me;
    location / {
        proxy_pass http://192.168.1.2:12345;
    }
    # plus the usual Certbot SSL stuff

This was working perfectly well as a DT sync store, with HTTPS authentication and everything. The .dtCloud file was saved in the contents folder and both DT and DTTG were syncing happily.

I wanted to make this a bit more tidy and only changed couple of things:

  • WebDav server root to /SomeVolume/webdav (instead of /SomeVolume/webdav/contents), restarted the server.
  • proxy_pass http://192.168.1.2:12345 changed to proxy_pass http://192.168.1.2:12345/contents. Restarted ngninx.

Nothing else was modified.

I can still login through the browser, but DT has stopped working because it gets authentication errors (401) from the server (seen in the Log window).

What am I not understanding here? Is it some caching problem with DT or have I misunderstood how location & proxy_pass work?

Thanks.

Did you update the settings (especially the URL) of the sync location? Are you able to access the WebDAV server in the Finder using the same URL?

That’s the point, the url in DT is just set to dt.mydomain.com and I’d like to keep it that way. The domain still points to my nginx server (otherwise I would get nothing instead of an authentication error), and I was hoping to just reroute the default / location in nginx to the contents folder, instead of having the contents folder be the root of the WebDAV server.

I can login through the Finder with no problems, but this stops working if I use the /contents suffix in the proxy_pass directive. However, I can login perfectly well with 192.168.1.2:12345/contents/. I just don’t understand why 401 is getting thrown at me, but perhaps this is not a DT issue.

If the finder can’t connect, it might not be a DT issue.

Did you read this: Nginx: Everything about proxy_pass - DEV Community

But frankly, you lost me with dt suffix in the proxy_pass. Before it was contents, wasn’t it? Anyway, if you can connect with finder to the WebDAV location, DT can too. In my experience. Just set up DT with the same data as in your finder connection.

In Apache, I’d define an Alias and that’s all. But I don’t know about nginx.

Sorry for the mess up, I’ve fixed the address in my previous post. I’ll see if I can it sorted out, apparently it’s not a DT issue. Thanks a lot.

Still trying to work this out, I was wondering if the regular 401 and 404 errors I get in my access log are normal or are there a sign of something wrong. From a user perspective I see no issue, everything is properly syncing.

Thanks.

192.168.1.254 - - [08/Feb/2023:09:35:26 +0100] "PROPFIND /DT.dtCloud/ HTTP/1.1" 401 0 "-" "DEVONcloudy 1.22.2" 
192.168.1.254 - - [08/Feb/2023:09:35:26 +0100] "PROPFIND /DT.dtCloud/ HTTP/1.1" 207 271 "-" "DEVONcloudy 1.22.2" 
192.168.1.254 - - [08/Feb/2023:09:35:26 +0100] "GET /DT.dtCloud/7b8b717fa62fddda7fe843879f442edee1fba942ff339ba131e4f01d752316f5/master.plist HTTP/1.1" 200 224 "-" "DEVONcloudy 1.22.2" 
192.168.1.254 - - [08/Feb/2023:09:35:26 +0100] "PROPFIND /DT.dtCloud/7b8b717fa62fddda7fe843879f442edee1fba942ff339ba131e4f01d752316f5/ HTTP/1.1" 207 2003 "-" "DEVONcloudy 1.22.2" 
192.168.1.254 - - [08/Feb/2023:09:35:26 +0100] "PROPFIND /DT.dtCloud/7b8b717fa62fddda7fe843879f442edee1fba942ff339ba131e4f01d752316f5/697477836.365-b6fbffca9417cd58b0e52c9974ca275fe3c7a539-4212375748.manifest HTTP/1.1" 404 0 "-" "DEVONcloudy 1.22.2" 
192.168.1.254 - - [08/Feb/2023:09:35:26 +0100] "GET /DT.dtCloud/7b8b717fa62fddda7fe843879f442edee1fba942ff339ba131e4f01d752316f5/697495454.582-0d588f8c340cfb490f4ee47f35944dbd1e173d80-1675104251.manifest HTTP/1.1" 200 5968 "-" "DEVONcloudy 1.22.2" 
192.168.1.254 - - [08/Feb/2023:09:35:35 +0100] "PROPFIND /DT.dtCloud/ HTTP/1.1" 401 0 "-" "DEVONcloudy 3.6.3" 
192.168.1.254 - - [08/Feb/2023:09:35:35 +0100] "PROPFIND /DT.dtCloud/ HTTP/1.1" 207 271 "-" "DEVONcloudy 3.6.3" 
192.168.1.254 - - [08/Feb/2023:09:35:35 +0100] "GET /DT.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/master.plist HTTP/1.1" 200 224 "-" "DEVONcloudy 3.6.3" 
192.168.1.254 - - [08/Feb/2023:09:35:35 +0100] "PROPFIND /DT.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/ HTTP/1.1" 207 2003 "-" "DEVONcloudy 3.6.3" 
192.168.1.254 - - [08/Feb/2023:09:35:35 +0100] "GET /DT.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/697538122.636-b6fbffca9417cd58b0e52c9974ca275fe3c7a539-3506847300.manifest HTTP/1.1" 200 828304 "-" "DEVONcloudy 3.6.3" 
192.168.1.254 - - [08/Feb/2023:09:35:36 +0100] "GET /DT.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/receipts/1986517562622b7493ad1a2df36eb96e48e12084035a00c0e7eb03f5a2041d06.receipt/559a0ffcd43320f48667deb4c21e47cb954c0967b29d7a11359e395e09f43766.item HTTP/1.1" 200 9712 "-" "DEVONcloudy 3.6.3" 

Usually there should be no errors and as there weren’t any issues before changing the setup, something seems to be still broken.

What baffles me is that DT seems to be synchronizing perfectly well with no errors reported elsewhere.

Maybe these errors are temporary server issues? DEVONthink performs automatic retries in case of certain HTTP errors.

I don’t know. What’s weird is that (1) I get the following log error even when no clients are trying to access the server (i.e. no DT desktop open and iOS clients quit) and (2) I can see 401 and 404 errors immediately followed by success (207).

What is triggering these PROPFIND requests when no client is running? It seems to happen every 15 minutes or so.

192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND / HTTP/1.1" 401 0 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND /dt/ HTTP/1.1" 404 0 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND /dt/ HTTP/1.1" 404 0 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND / HTTP/1.1" 207 381 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND / HTTP/1.1" 401 0 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND / HTTP/1.1" 207 381 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND / HTTP/1.1" 207 381 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)" 
192.168.1.254 - - [2023-02-09T10:01:20+0100] "PROPFIND / HTTP/1.1" 207 381 "-" "WebDAVFS/3.0.0 (03008000) Darwin/21.6.0 (x86_64)"

The numerous "PROPFIND /DT.dtCloud/ HTTP/1.1" in my previous post seem to have been greatly reduced after I added this in nginx:

proxy_set_header Host              $host;
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host  $host;
proxy_set_header X-Forwarded-Port  $server_port;

However I still get a few errors when the server is accessed, e.g. this is what I see when I do a simple sync with no changes from iOS:

37.169.78.93 - - [2023-02-09T10:13:28+0100] "GET /DT.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/master.plist HTTP/1.1" 200 224 "-" "DEVONcloudy 3.6.3"
37.169.78.93 - - [2023-02-09T10:13:28+0100] "PROPFIND /DT-NAS4.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/ HTTP/1.1" 207 2002 "-" "DEVONcloudy 3.6.3"
37.169.78.93 - - [2023-02-09T10:13:28+0100] "PROPFIND /DT-NAS4.dtCloud/53cbd346392b8673a03715b8c62f03559f470c38c6c79289f273814a7023841f/697624789.297-1c352d65c707ea74c13c48f0189659367565151b-2122896057.manifest HTTP/1.1" 404 0 "-" "DEVONcloudy 3.6.3"

If it’s an internal way of doing things in DT, I don’t mind, I could just stop logging these errors. But I’d like to understand where the issues lies between DT and my server.

Most likely either not all clients were terminated and/or automatically launched again on iOS in the background.

They aren’t, I’ve double-checked. Could it be the global Inbox shortcut installed in the Finder?

Could it be the global Inbox shortcut installed in the Finder?

That’s merely an alias to the original directory, nothing more.

OK. I’m still struggling a bit with this, the reason being my old NAS WebDAV server died and I need to set up another one manually.

I’ve managed to create the sync store and waited for my DT databases to upload everything. I then downloaded everything to my laptop DT and tested that changes in files were correctly in sync with some edits, group creations and deletions.

However:

  • When starting sync, my desktop DT log keeps saying “Couldn’t upload 6 pending files, 6 files left to be uploaded”, but I can’t understand which ones and why (there are over 15000 files in my databases). It always mentions exactly 6 files. I’ve verified the database and the location successfully multiple times.

  • My old sync store size was 19 gigabytes. The new one has 16. The database hasn’t changed so much. Can there be some left over receipts or something else that amount to so much data?

  • I’m still getting seemingly random “Unauthorized” and “Failed creating collection” in the WebDAV log, but nothing in DT log (except the above). Can this be related to the Max Connections setting of 16 (which worked fine on the old server)?

  • For some reason the Sync icon in the toolbar is always on, even with only one DT instance open, sync finished and no messages in the DT log.

The sync store functionality of DT has never failed me in years of use, so I’m pretty confident in its errors and verifications, but this is the first time I’m handling the innards of a WebDAV server and I need to be confident that I’m not losing data in the process.

Thanks a lot for your feedback on this.

A toolbar search for item:pending should find them.

The sizes aren’t comparable, only the ones shown in File > Database Properties are.

item:pending only shows one item, although 6 are still mentioned in the log:

And the single pending item can’t be downloaded when clicking on download, and I don’t understand why it was removed from the DT desktop database (this particular group hasn’t been modified in years).

Does File > Verify & Repair Database report any issues? Or do your databases contain indexed items which might be located on not mounted volumes?

Both have been successfully verified. And no, I don’t indexed items at all. I’ve double-checked with the following smart group.

This seems to be a smart group searching only one database, a toolbar search for item:indexed in all databases would be useful.

I had duplicated that smart group in my three databases when I checked, but here’s a toolbar search: